bvaradar commented on issue #2149:
URL: https://github.com/apache/hudi/issues/2149#issuecomment-709868001
@ashishmgofficial : With your provided avro file, I am able to ingest
without any errors.
```
spark-submit --packages org.apache.spark:spark-avro_2.11:2.4.4 --class
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer
$HUDI_UTILITIES_BUNDLE --table-type COPY_ON_WRITE --source-class
org.apache.hudi.utilities.sources.AvroDFSSource --source-ordering-field
_ts_ms --target-base-path /tmp/hudi_data --target-table stock_ticks_cow
--props
/Users/balaji.varadarajan/code/oss/upstream_hudi/docker/demo/config/cases_dfs_source.properties
--schemaprovider-class
org.apache.hudi.utilities.schema.DebeziumSchemaRegistryProvider
--transformer-class org.apache.hudi.utilities.transform.DebeziumTransformer
--hoodie-conf
hoodie.compaction.payload.class=org.apache.hudi.common.model.DebeziumAvroPayload
--hoodie-conf hoodie.deltastreamer.source.dfs.root=file:///tmp/data/ 2>&1 |
tee o.log
```
Logs
```
+-------------+---+------+----+-----------------------------------+--------------+------------------+----+----------------+
|_ts_ms |_op|inc_id|year|violation_desc
|violation_code|case_individual_id|flag|last_modified_ts|
+-------------+---+------+----+-----------------------------------+--------------+------------------+----+----------------+
|1602791105042|c |1 |2016|DRIVING WHILE INTOXICATED |11923
|17475366 |I |1600945380000000|
|1602791105042|c |3 |2016|AGGRAVATED UNLIC OPER 2ND/PREV CONV|5112A1
|17475367 |U |1600959600000000|
|1602791105043|c |4 |2019|AGGRAVATED UNLIC OPER 2ND/PREV |5112A2
|17475368 |I |1569337200000000|
|1602791105043|c |2 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |2180F
|17475569 |U |1601377200000000|
|1602791105043|c |9 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180E
|17475573 |I |1601377200000000|
|1602791105043|c |10 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180D
|17475574 |I |1601377800000000|
|1602791105044|c |11 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180D
|17475574 |I |1601381400000000|
|1602791105044|c |12 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180E
|17475574 |I |1601385000000000|
|1602791105044|c |13 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180E
|17475574 |I |1601388600000000|
|1602791105044|c |34 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180E
|17475574 |I |1601392200000000|
|1602791105045|c |35 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180E
|17475574 |I |1601395800000000|
|1602791105045|c |36 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180E
|17475574 |I |1601398800000000|
|1602791105045|c |37 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180E
|17475574 |I |1601399400000000|
|1602791105045|c |38 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180D
|17475574 |I |1601402400000000|
|1602791141537|d |3 |2016|AGGRAVATED UNLIC OPER 2ND/PREV CONV|5112A1
|17475367 |U |1600959600000000|
+-------------+---+------+----+-----------------------------------+--------------+------------------+----+----------------+
....
20/10/15 21:56:44 INFO DeltaSync: Commit 20201015215641 successful!
......
```
I am able to read the newly added data successfully too:
```
scala> df.show(100, false)
+-------------------+--------------------+------------------+----------------------+---------------------------------------------------------------------+-------------+---+------+----+-----------------------------------+--------------+------------------+----+----------------+
|_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path|_hoodie_file_name
|_ts_ms
|_op|inc_id|year|violation_desc
|violation_code|case_individual_id|flag|last_modified_ts|
+-------------------+--------------------+------------------+----------------------+---------------------------------------------------------------------+-------------+---+------+----+-----------------------------------+--------------+------------------+----+----------------+
|20201015215641 |20201015215641_0_1 |34 |
|738db628-641f-4d60-b7e5-c9c971a371f8-0_0-27-29_20201015215641.parquet|1602791105044|c
|34 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180E |17475574
|I |1601392200000000|
|20201015215641 |20201015215641_0_2 |10 |
|738db628-641f-4d60-b7e5-c9c971a371f8-0_0-27-29_20201015215641.parquet|1602791105043|c
|10 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180D |17475574
|I |1601377800000000|
|20201015215641 |20201015215641_0_3 |1 |
|738db628-641f-4d60-b7e5-c9c971a371f8-0_0-27-29_20201015215641.parquet|1602791105042|c
|1 |2016|DRIVING WHILE INTOXICATED |11923 |17475366
|I |1600945380000000|
|20201015215641 |20201015215641_0_4 |3 |
|738db628-641f-4d60-b7e5-c9c971a371f8-0_0-27-29_20201015215641.parquet|1602791141537|d
|3 |2016|AGGRAVATED UNLIC OPER 2ND/PREV CONV|5112A1 |17475367
|U |1600959600000000|
|20201015215641 |20201015215641_0_5 |36 |
|738db628-641f-4d60-b7e5-c9c971a371f8-0_0-27-29_20201015215641.parquet|1602791105045|c
|36 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180E |17475574
|I |1601398800000000|
|20201015215641 |20201015215641_0_6 |12 |
|738db628-641f-4d60-b7e5-c9c971a371f8-0_0-27-29_20201015215641.parquet|1602791105044|c
|12 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180E |17475574
|I |1601385000000000|
|20201015215641 |20201015215641_0_7 |9 |
|738db628-641f-4d60-b7e5-c9c971a371f8-0_0-27-29_20201015215641.parquet|1602791105043|c
|9 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180E |17475573
|I |1601377200000000|
|20201015215641 |20201015215641_0_8 |38 |
|738db628-641f-4d60-b7e5-c9c971a371f8-0_0-27-29_20201015215641.parquet|1602791105045|c
|38 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180D |17475574
|I |1601402400000000|
|20201015215641 |20201015215641_0_9 |2 |
|738db628-641f-4d60-b7e5-c9c971a371f8-0_0-27-29_20201015215641.parquet|1602791105043|c
|2 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |2180F |17475569
|U |1601377200000000|
|20201015215641 |20201015215641_0_10 |4 |
|738db628-641f-4d60-b7e5-c9c971a371f8-0_0-27-29_20201015215641.parquet|1602791105043|c
|4 |2019|AGGRAVATED UNLIC OPER 2ND/PREV |5112A2 |17475368
|I |1569337200000000|
|20201015215641 |20201015215641_0_11 |13 |
|738db628-641f-4d60-b7e5-c9c971a371f8-0_0-27-29_20201015215641.parquet|1602791105044|c
|13 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180E |17475574
|I |1601388600000000|
|20201015215641 |20201015215641_0_12 |11 |
|738db628-641f-4d60-b7e5-c9c971a371f8-0_0-27-29_20201015215641.parquet|1602791105044|c
|11 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180D |17475574
|I |1601381400000000|
|20201015215641 |20201015215641_0_13 |37 |
|738db628-641f-4d60-b7e5-c9c971a371f8-0_0-27-29_20201015215641.parquet|1602791105045|c
|37 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180E |17475574
|I |1601399400000000|
|20201015215641 |20201015215641_0_14 |35 |
|738db628-641f-4d60-b7e5-c9c971a371f8-0_0-27-29_20201015215641.parquet|1602791105045|c
|35 |2020|UNREASONABLE SPEED/SPECIAL HAZARDS |1180E |17475574
|I |1601395800000000|
+-------------------+--------------------+------------------+----------------------+---------------------------------------------------------------------+-------------+---+------+----+-----------------------------------+--------------+------------------+----+----------------+
```
Do yo see anything I am missing here ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
