[GitHub] [hudi] rubenssoto commented on issue #1957: [SUPPORT] Small Table Upsert sometimes take a lot of time

[GitBox]

rubenssoto commented on issue #1957:
URL: https://github.com/apache/hudi/issues/1957#issuecomment-674193769


   Hi @bhasudha ,
   
   Yes, this is a simple job, reads parquet generated by aws DMS and writes 
Hudi.
   
   My spark submit:
   spark-submit --deploy-mode cluster --conf 
spark.dynamicAllocation.cachedExecutorIdleTimeout=60s --conf 
spark.dynamicAllocation.executorIdleTimeout=60s --conf 
spark.dynamicAllocation.maxExecutors=1 --conf 
spark.executor.memoryOverhead=2048 --conf spark.executor.cores=3 --conf 
spark.executor.memory=10g --conf 
spark.serializer=org.apache.spark.serializer.KryoSerializer --conf 
spark.sql.hive.convertMetastoreParquet=false --packages 
org.apache.hudi:hudi-spark-bundle_2.11:0.5.3,org.apache.spark:spark-avro_2.11:2.4.4
 --py-files python_modules main.py poc_available_history
   
   
   
   <img width="1680" alt="Captura de Tela 2020-08-14 às 14 51 08" 
src="https://user-images.githubusercontent.com/36298331/90278335-afaddd80-de3d-11ea-8cf0-9fd168fa643a.png";>
   <img width="1680" alt="Captura de Tela 2020-08-14 às 14 50 25" 
src="https://user-images.githubusercontent.com/36298331/90278349-b63c5500-de3d-11ea-91a9-02f3ae5bfc76.png";>
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org