getniz opened a new issue #2101:
URL: https://github.com/apache/hudi/issues/2101
Issue details:
With in a nested JSON data schema with below format is there a way to
consume the child object alone ignoring the parent field.
It is not recognizing the Child fields of data field for any recordkey
rather it is taking as 1 field(data) and its child fields are taken as a single
column value while loading in Hudi table.
If something I'm doing wrong please guide otherwise what is the practise to
integrate with Apache Hudi for the example scenario.
Example scenario
Current JSON Format from Kafka topic
{"data":
{"NUMBER": "2223",
"NAME": "XXXXXXXXXXXXXX",
"CHANGED_BY": "YYYYYYY"}}
{"data":
{"NUMBER": "2224",
"NAME": "XXXXXXXXXXXXXX",
"CHANGED_BY": "YYYYYYY"}}
test_schema.avsc
{ "name": "data",
"type": {
"name": "data",
"type": "record",
"fields": [
{
"name": "NUMBER",
"type": "string"
},
{
"name": "NAME",
"type": "string"
},
{
"name": "CHANGED_BY",
"type": "string"
}
]
}
}
How it is loaded in Hudi S3 table (ignored the hoodie commit fields):
RecordKey | data
1 | "NUMBER": "2223", "NAME": "XXXXXXXXXXXXXX", "CHANGED_BY":
"2019-08-01"
1 | "NUMBER": "2224", "NAME": "XXXXXXXXXXXXXX", "CHANGED_BY":
"2020-08-01"
How it is expected in Hudi S3 table
RecordKeyNUMBER)| NUMBER | Name |
Changed_by
2223 | 2223 | XXXXXXXXXXXXXX |
2019-08-01
2223 | 2223 | XXXXXXXXXXXXXX |
2020-08-01
Here the Hudi batch command:
===========================
spark-submit --jars
"/mnt/hudi/packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.6.1-SNAPSHOT.jar"
\
--deploy-mode "client" \
--class "org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer" \
/mnt/hudi/packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.11-0.6.1-SNAPSHOT.jar
\
--props file:///mnt/hudi/docker/demo/config/kafka-source.properties \
--table-type MERGE_ON_READ \
--source-class org.apache.hudi.utilities.sources.JSONKafkaSource \
--source-ordering-field Changed_by \
--target-base-path s3://landing/test/ \
--target-table test-table \
--schemaprovider-class
org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
--enable-hive-sync --hoodie-conf
hoodie.datasource.hive_sync.jdbcurl=jdbc:hive2://localhost:10000 \
--hoodie-conf hoodie.datasource.hive_sync.username=hive \
--hoodie-conf hoodie.datasource.hive_sync.database=default \
--hoodie-conf hoodie.datasource.hive_sync.table=test \
--hoodie-conf
hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.NonpartitionedKeyGenerator
\
--hoodie-conf
hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.NonPartitionedExtractor
\ --hoodie-conf hoodie.compact.inline=true
Kafka-source.properties
=======================
include=base.properties
# Key fields, for kafka example
hoodie.datasource.write.recordkey.field=DATE_CREATED
hoodie.datasource.write.partitionpath.field=date
# Schema provider props (change to absolute path based on your installation)
hoodie.deltastreamer.schemaprovider.source.schema.file=file:///mnt/hudi/docker/demo/config/test_schema.avsc
hoodie.deltastreamer.schemaprovider.target.schema.file=file:///mnt/hudi/docker/demo/config/test_schema.avsc
# Kafka Source topic
hoodie.deltastreamer.source.kafka.topic=test
#Kafka props
bootstrap.servers=kafka.com:9092
auto.offset.reset=earliest
**Environment Description**
Everything is installed manually in aws emr
* Hudi version : 0.6.1
* Spark version : 2.4.6
* Hive version : Hadoop 2.8.5
* Hadoop version : Hive 2.3.6
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
