JB-data opened a new issue #4055: URL: https://github.com/apache/hudi/issues/4055
**Describe the problem you faced** We are running a spark-submit command with an SQL transformation. The data from the kafka topic is then temorarily registered in hive's default db, and applies a very simple sql to it. We see the job fail quite soon, while without the sql transformation, it succeeds. This is more related to spark settings probably than to hudi. But maybe someone has experience in the context with hudi? **To Reproduce** Steps to reproduce the behavior: spark-submit command with sql transformation to take data from kafka topic, and write slighty modified to another table: 1. spark-submit --master yarn --deploy-mode cluster --driver-java-options "-Djava.security.auth.login.config=./client.jaas" \ --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./client.jaas" --conf spark.app.name=myapp \ --conf spark.yarn.maxAppAttempts=4 --conf spark.yarn.am.attemptFailuresValidityInterval=1h --conf spark.yarn.executor.failuresValidityInterval=1h \ --conf spark.task.maxFailures=8 --principal myprinc@myserver --keytab mykeytab \ --files client.jaas,mykeytab,kafka.client.truststore.jks --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer \ --jars /opt/cloudera/parcels/CDH-7.2.2-1.cdh7.2.2.p3.7839477/lib/hive/lib/hive-service-3.1.3000.7.2.2.3-1.jar,\ /opt/cloudera/parcels/CDH-7.2.2-1.cdh7.2.2.p3.7839477/lib/hive/lib/hive-jdbc-3.1.3000.7.2.2.3-1.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar abfs://[email protected]/config/streaming/fr24/hudi-utilities-bundle_2.11-0.6.0.jar \ \ --props abfs://stora/myproperties \ --target-base-path abfs://storagemytransfo/unmanaged/mytable \ --table-type COPY_ON_WRITE \ --source-class org.apache.hudi.utilities.sources.AvroKafkaSource \ --target-table refactored_created_by_hue_smalltransfo \ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \ --source-limit 40000 \ \ --transformer-class org.apache.hudi.utilities.transform.SqlQueryBasedTransformer \ --hoodie-conf hoodie.deltastreamer.transformer.sql='SELECT data[0].id as id FROM <SRC>' --executor-memory 4G \ --driver-memory 2G \ **Expected behavior** No error during that stage, correct landing of transformed data in target table. Environment Description Hudi version : 0.6.0 Spark version :2.4.5 Hive version :Hive 3.1 on Tez Hadoop version :3.1.1 Storage (HDFS/S3/GCS..) : abfs (azure blob storage) Running on Docker? (yes/no) :no **Additional context** Add any other context about the problem here. **Stacktrace** YARN: Driver stacktrace ```org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 8 times, most recent failure: Lost task ... ExecutorLostFailure ...Container from a bad node: container_e29_1632244056069_1301_01_000009 on host: xxx Exit status: 134. Diagnostics: 2 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.transformer.sql' was supplied but isn't a known config.``` Note: running same job also exits with exit code 143 instead of 134. SPARK UI: ``` after 1.6 minutes failure in 2nd stage: isEmpty at DeltaSync.java:344``` + same error as from yarn ( Container from a bad node: container_yy on host: xxx Exit status: 134) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
