kirillklimenko opened a new issue, #11348: URL: https://github.com/apache/hudi/issues/11348
**Describe the problem you faced** I'm using [Hudi Streamer](https://hudi.apache.org/docs/hoodie_streaming_ingestion/) to ingest data from Kafka to Hudi MOR table. If I produce the `{"sha512": "hash", "column1": "before", "column2": "before", "update_dt": "2024-01-01T00:00:00.000000"}` to the Kafka topic, I will get the following result in the Hudi table (after compaction): | sha512 | column1 | column2 | update_dt | |:------:|:-------:|:-------:|:---------:| | hash | before | before | 2024-01-01T00:00:00.000000 | My goal is to update `column2`, without affecting `column1` by producing `{"sha512": "hash", "column2": "after", "update_dt": "2024-02-02T00:00:00.000000"}` to the Kafka topic (after compaction). What I expect as a result (after compaction): | sha512 | column1 | column2 | update_dt | |:------:|:-------:|:-------:|:---------:| | hash | before | after | 2024-02-02T00:00:00.000000 | What I have as a result (after compaction): | sha512 | column1 | column2 | update_dt | |:------:|:-------:|:-------:|:---------:| | hash | null | after | 2024-02-02T00:00:00.000000 | Since I have `PartialUpdateAvroPayload` in my settings below, I expect a partial update, but I suspect it doesn't work because of `"default": null` keyword in my schema definition. Could you suggest how can I update `column2`, without affecting `column1` with the current setup using Hudi Streamer? My `spark-submit` command: ```bash spark-submit \ --jars /usr/lib/hudi/hudi-aws-bundle.jar \ --class org.apache.hudi.utilities.streamer.HoodieStreamer \ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \ --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog \ --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension \ --conf spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar \ /usr/lib/hudi/hudi-utilities-bundle.jar \ --hoodie-conf bootstrap.servers=$KAFKA_BROKERS \ --op UPSERT \ --props /app/config/hudi.properties \ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \ --source-ordering-field updated_dt \ --table-type MERGE_ON_READ \ --target-base-path s3://$S3_BUCKET/bronze \ --target-table bronze \ --transformer-class org.apache.hudi.utilities.transform.SqlFileBasedTransformer ``` My `hudi.properties` file: ``` auto.offset.reset=earliest hoodie.index.type=GLOBAL_BLOOM hoodie.bloom.index.update.partition.path=false hoodie.datasource.write.recordkey.field=sha512 hoodie.datasource.write.partitionpath.field=year,month,day hoodie.datasource.write.precombine.field=updated_dt hoodie.datasource.write.hive_style_partitioning=true hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.ComplexKeyGenerator hoodie.datasource.write.payload.class=org.apache.hudi.common.model.PartialUpdateAvroPayload hoodie.streamer.source.kafka.topic=hudi-stream hoodie.streamer.transformer.sql.file=/app/sql/transformer.sql hoodie.streamer.schemaprovider.source.schema.file=/ulh-data-streamer/app/schema/source.avsc hoodie.streamer.schemaprovider.target.schema.file=/ulh-data-streamer/app/schema/target.avsc ``` My `transformer.sql` file: ```sql SELECT t.*, DATE_FORMAT(t.updated_dt, 'yyyy') AS year, DATE_FORMAT(t.updated_dt, 'MM') AS month, DATE_FORMAT(t.updated_dt, 'dd') AS day FROM <SRC> t; ``` My `source.avsc` file: ``` { "type": "record", "name": "source_schema", "fields": [ {"name": "sha512", "type": "string"}, {"name": "column1", "type": ["null", "string"], "default": null}, {"name": "column2", "type": ["null", "string"], "default": null}, {"name": "updated_dt", "type": "string"} ] } ``` My `target.avsc` file: ``` { "type": "record", "name": "target_schema", "fields": [ {"name": "sha512", "type": "string"}, {"name": "column1", "type": ["null", "string"], "default": null}, {"name": "column2", "type": ["null", "string"], "default": null}, {"name": "updated_dt", "type": "string"}, {"name": "year", "type": "string"}, {"name": "month", "type": "string"}, {"name": "day", "type": "string"} ] } ``` **Environment Description** * Hudi version : 0.14.1 * Spark version : 3.4 * Storage (HDFS/S3/GCS..) : S3 * Running on Docker? (yes/no) : no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
