prashanthpdesai edited a comment on issue #1653:
URL: https://github.com/apache/incubator-hudi/issues/1653#issuecomment-632406451
@bhasudha : I presume that offset for each partitions will be stored the
commit metadata in HDFS path, since its first run we used auto.offset.reset
earliest to consume it from beginning . for incremental run offset will be
picked up automatically rite from the HDFS path ? please correct me if my
understanding is wrong.
Command we used to run in kubenetes pod:
spark-submit "--props",
System.getenv("KAFKA_PROP_FILE"),
"--schemaprovider-class",
"org.apache.hudi.utilities.schema.SchemaRegistryProvider",
"--source-class","org.apache.hudi.utilities.sources.AvroKafkaSource",
"--target-base-path",System.getenv("HUDI_OUTPUT_LOC"),
"--target-table", "mcm.hudi.deltacow",
"--table-type", "COPY_ON_WRITE",
"--checkpoint",
"--commit-on-errors",
"--op", "UPSERT",
"--source-ordering-field", "modifiedDt".
For first run ,
#kafka prop file
hoodie.deltastreamer.source.kafka.topic=enriched-output-changelog
hoodie.auto.commit=false
enable.auto.commit=false
auto.offset.reset=earliest
we did try to both earliest and latest as well but we ended up with same
exception.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
