t0il3ts0ap commented on issue #2515: URL: https://github.com/apache/hudi/issues/2515#issuecomment-777992760
@vinothchandar nope. Attaching config for your reference. This is happening when a row is being deleted. The deletion is happening successfully, but archival post that fails. ``` spark-submit --packages org.apache.spark:spark-avro_2.12:3.0.1,org.apache.hudi:hudi-utilities-bundle_2.12:0.7.0 --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer --master yarn --num-executors 1 --conf spark.scheduler.mode=FAIR --conf spark.task.maxFailures=5 --conf spark.rdd.compress=true --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.shuffle.service.enabled=true --conf spark.sql.hive.convertMetastoreParquet=false --conf spark.executor.heartbeatInterval=120s --conf spark.network.timeout=600s --conf spark.eventLog.overwrite=true --conf spark.eventLog.enabled=true --conf spark.yarn.max.executor.failures=5 --conf spark.sql.catalogImplementation=hive --deploy-mode client /home/hadoop/deltastreamer-addons-1.0-SNAPSHOT.jar --enable-sync --hoodie-conf hoodie.deltastreamer.schemaprovider.registry.url=https://dataplatform-schema-registry.nnn.prod.tech.in/subjects/auth_service.public.users-value/versions/latest --hoodie-conf hoodie.datasource.hive_sync.database=auth_service --hoodie-conf hoodie.datasource.hive_sync.table=usersv2 --hoodie-conf hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.NonPartitionedExtractor --hoodie-conf hoodie.datasource.write.recordkey.field=id --hoodie-conf hoodie.datasource.write.precombine.field=__lsn --hoodie-conf hoodie.datasource.write.partitionpath.field='' --hoodie-conf hoodie.parquet.compression.codec=snappy --hoodie-conf hoodie.deltastreamer.source.kafka.topic=auth_service.public.users --hoodie-conf group.id=beta-delta-streamer-usersv2 --hoodie-conf partition.assignment.strategy=org.apache.kafka.clients.consumer.RangeAssignor --hoodie-conf hive.metastore.disallow.incompatible.col.type.changes=false --hoodie-conf hoodie.deltastreamer.schemaprovider.spark_avro_post_processor.enable=false --hoodie-conf auto.offset.reset=latest --table-type MERGE_ON_READ --source-ordering-field __lsn --source-class org.apache.hudi.utilities.sources.AvroKafkaSource --schemaprovider-class com.nnn.schema.CustomSchemaRegistryProvider --target-base-path s3://navi-emr-poc/delta-streamer-test/tables/usersv2 --target-table usersv2 --props s3://navi-emr-poc/delta-streamer-test/config/kafka-source-prod.properties --transformer-class com.nnn.transform.CustomTransformer --continuous ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
