SamarthRaval commented on issue #8438: URL: https://github.com/apache/hudi/issues/8438#issuecomment-1520664120
> @SamarthRaval Can you paste the screenshots of spark UI for the stages which are taking long. Also if possible provide the logs of the driver and one of the executor. Hello, Due to unavailability of spark history server(As its > 1 month) I don't have access to history server. But executor logs error has been provided above, and job gets finishes successfully but performance is the main concern here. Compaction stage takes ~ 1.2 hrs `spark-submit --deploy-mode cluster --class org.apache.hudi.utilities.HoodieCompactor --jars /usr/lib/hudi/hudi-utilities-bundle.jar,/usr/lib/hudi/hudi-spark-bundle.jar,s3://hudi-tier2-extractor-1.2.3.jar /usr/lib/hudi/hudi-utilities-bundle.jar --base-path s3://transactions.all_hudi/ --table-name transactions_all --parallelism 1000 --spark-memory 90g --mode schedule --hoodie-conf hoodie.embed.timeline.server=false --hoodie-conf hoodie.datasource.write.payload.class=com.spark.datalake.NullSafeDefaultHoodieRecordPayload --hoodie-conf hoodie.payload.ordering.field=lastmodifieddate --hoodie-conf hoodie.compaction.logfile.size.threshold=500000` Execution stage takes ~ 7 mins `spark-submit --deploy-mode cluster --class org.apache.hudi.utilities.HoodieCompactor --jars /usr/lib/hudi/hudi-utilities-bundle.jar,/usr/lib/hudi/hudi-spark-bundle.jar,s3://hudi-tier2-extractor-1.2.3.jar /usr/lib/hudi/hudi-utilities-bundle.jar --base-path s3://transactions.all_hudi/ --table-name transactions_all --parallelism 1000 --spark-memory 90g --hoodie-conf hoodie.embed.timeline.server=false --hoodie-conf hoodie.datasource.write.payload.class=com.spark.datalake.NullSafeDefaultHoodieRecordPayload --hoodie-conf hoodie.payload.ordering.field=lastmodifieddate --hoodie-conf hoodie.compaction.logfile.size.threshold=500000` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
