[GitHub] [hudi] ZeMirella opened a new issue #3699: [SUPPORT] Job hanging on toRdd at HoodieSparkUtils

[GitBox]

ZeMirella opened a new issue #3699:
URL: https://github.com/apache/hudi/issues/3699


   **Describe the problem you faced**
   My job hangs during the toRdd task here is some screenshots of the task size
   
   <img width="1771" alt="Captura de Tela 2021-09-21 às 19 07 20" 
src="https://user-images.githubusercontent.com/75490501/134254359-2f2d0cb9-8bb8-48d2-b02d-f84fd9dab9d6.png";>
   
   <img width="1746" alt="Captura de Tela 2021-09-21 às 19 07 46" 
src="https://user-images.githubusercontent.com/75490501/134254377-963509f6-3075-4509-bf88-7188dc71d548.png";>
   
   **My spark-submit**
   `spark-submit --deploy-mode cluster --conf spark.executor.cores=5 --conf 
spark.executor.memoryOverhead=6g --conf spark.executor.memory=43g --conf 
spark.dynamicAllocation.maxExecutors=50 --conf 
spark.sql.hive.convertMetastoreParquet=false --conf spark.rdd.compress=true 
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf 
spark.kryoserializer.buffer.max=512m --packages 
org.apache.hudi:hudi-spark-bundle_2.12:0.8.0,org.apache.spark:spark-avro_2.12:3.0.1,com.audienceproject:spark-dynamodb_2.12:1.1.2
 --py-files s3://bucket/modules.zip --files s3://bucket/config.yml 
s3://bucket/main.py`
   
   **My cluster configuration**
   
   <img width="1533" alt="Captura de Tela 2021-09-21 às 19 11 44" 
src="https://user-images.githubusercontent.com/75490501/134254642-ffdbc239-77b2-4d47-be1e-8a3872280533.png";>
   
   I also set shuffle.parallelism=2000
   
   **Expected behavior**
   It should run without hang
   
   **Environment Description**
   
   * Hudi version : 0.8
   
   * Spark version : 3.0
   
   * Hive version : 3.1.2
   
   * Hadoop version : Amazon 3.2.1
   
   * Storage (HDFS/S3/GCS..) : s3
   
   * Running on Docker? (yes/no) : no
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org