liujinhui1994 opened a new issue #2209: URL: https://github.com/apache/hudi/issues/2209
**_Tips before filing an issue_** - Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)? - Join the mailing list to engage in conversations and get faster support at [email protected]. - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly. **Describe the problem you faced** A clear and concise description of the problem. **To Reproduce** Steps to reproduce the behavior: hoodie.deltastreamer.source.dfs.root There are only two parquet files in this configuration, the size is about 2.5MB This is my start command spark-submit --jars hdfs://nameservice1/data_lake/jars/hive-service-2.1.1.jar,hdfs://nameservice1/data_lake/jars/hive-jdbc-2.1.1.jar \ --master yarn \ --name t3_science.t_short_url \ --driver-memory 2G \ --num-executors 1 \ --executor-memory 2G \ --executor-cores 1 \ --deploy-mode cluster \ --conf spark.driver.userClassPathFirst=true \ --conf spark.dynamicAllocation.enabled=false \ --conf spark.yarn.submit.waitAppCompletion=false \ --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer hdfs://nameservice1/data_lake/jars/data_lake.jar \ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \ --source-class org.apache.hudi.utilities.sources.ParquetDFSSource \ --target-base-path hdfs://nameservice1/data_lake/data/full/t3_science.t_short_url/20201032 \ --op BULK_INSERT \ --target-table t3_science.t_short_url_4 \ --table-type COPY_ON_WRITE \ --source-limit 5242880 \ --props hdfs://nameservice1/data_lake/config/full/t3_science.t_short_url/dfs-source.properties \ **Stacktrace** 20/10/27 15:33:29 INFO util.ShutdownHookManager: Shutdown hook called 20/10/27 15:33:29 INFO executor.Executor: Not reporting error to driver during JVM shutdown. 20/10/27 15:33:29 ERROR util.SparkUncaughtExceptionHandler: [Container in shutdown] Uncaught exception in thread Thread[Executor task launch worker for task 1,5,main] java.lang.OutOfMemoryError: Java heap space at java.lang.StringCoding.decode(StringCoding.java:215) at java.lang.String.<init>(String.java:463) at java.lang.String.<init>(String.java:515) at org.apache.spark.unsafe.types.UTF8String.toString(UTF8String.java:1214) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.createExternalRow_0_0$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.apply(Unknown Source) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:302) at org.apache.hudi.AvroConversionUtils$$anonfun$1.apply(AvroConversionUtils.scala:44) at org.apache.hudi.AvroConversionUtils$$anonfun$1.apply(AvroConversionUtils.scala:44) at scala.collection.Iterator$$anon$11.next(Iterator.scala:410) at scala.collection.Iterator$$anon$11.next(Iterator.scala:410) at scala.collection.Iterator$$anon$10.next(Iterator.scala:394) at scala.collection.Iterator$class.foreach(Iterator.scala:891) at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48) at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310) at scala.collection.AbstractIterator.to(Iterator.scala:1334) at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1334) at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289) at scala.collection.AbstractIterator.toArray(Iterator.scala:1334) at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$29.apply(RDD.scala:1364) at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$29.apply(RDD.scala:1364) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2121) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2121) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:121) at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1408) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
