ghrahul commented on issue #11126: URL: https://github.com/apache/hudi/issues/11126#issuecomment-2264735034
Hi @ad1happy2go , Thank you for sharing the above details. We were able to follow the process and tested it with multiple tables, it worked. We rolled out the upgrade to `Spark 3.2.1` and HUDI `HUDI 0.13.1` for around 500+ table in a stepwise migration. Then rolled out the upgrade again for `Spark 3.4.1` and `HUDI 0.14.0`. This process worked for 95% of the tables. But we are seeing issues for some tables where daily `bulk_insert` is happening. These pipelines are taking 10-20x time and 10-20x cost after the upgrade. There is a specific spark job which is taking a lot of time to complete after the upgrade which we have observed: `save at DatasetBulkInsertCommitActionExecutor.java:81` Could you please tell me if we are missing something in the HUDI configuration. **Spark HUDI Bundle Used** https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark3.4-bundle_2.12/0.14.0/hudi-spark3.4-bundle_2.12-0.14.0.jar https://repo1.maven.org/maven2/org/apache/hudi/hudi-spark3.2-bundle_2.12/0.13.1/hudi-spark3.2-bundle_2.12-0.13.1.jar **HUDI CONFIG for a Table with this issue** ``` {'className': 'org.apache.hudi', 'hoodie.datasource.hive_sync.use_jdbc': 'false', 'hoodie.datasource.write.precombine.field': 'created_at', 'hoodie.datasource.write.recordkey.field': 'master_user_id,created_at', 'hoodie.table.name': 'table_name', 'hoodie.consistency.check.enabled': 'false', 'hoodie.datasource.hive_sync.table': 'table_name', 'hoodie.datasource.hive_sync.database': 'lake_useractivity', 'hoodie.datasource.hive_sync.enable': 'true', 'hoodie.datasource.hive_sync.mode': 'hms', 'path': 's3a://bucket_name/table_name/public/table_name/', 'hoodie.index.type': 'BLOOM', 'hoodie.bloom.index.update.partition.path': 'true', 'hoodie.bloom.index.filter.type': 'DYNAMIC_V0', 'hoodie.datasource.hive_sync.support_timestamp': 'true', 'hoodie.datasource.write.reconcile.schema': 'true', 'hoodie.write.markers.type': 'direct', 'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.ComplexKeyGenerator', 'hoodie.datasource.write.partitionpath.field': 'master_user_id_partition', 'hoodie.datasource.hive_sync.partition_extractor_class': 'org.apache.hudi.hive.MultiPartKeysValueExtractor', 'hoodie.datasource.hive_sync.partition_fields': 'master_user_id_partition', 'hoodie.datasource.write.hive_style_partitioning': 'true', 'hoodie.upsert.shuffle.parallelism': 40, 'hoodie.datasource.write.operation': 'bulk_insert', 'hoodie.cleaner.policy': 'KEEP_LATEST_COMMITS', 'hoodie.cleaner.commits.retained': 1} ``` We use the above config for a same job with `HUDI 0.11.1` it took 5 minutes and 0.36 dollars to complete and for `HUDI 0.13.1` it took `1 hour 44 mins` and `36 dollars` to complete and with `HUDI 0.14.0` as well we saw same kind of cost hike and time hike. We compared the jobs which are getting created in Spark UI and we saw this job is getting created which is taking a lot of after the upgrade. And this is failing as well for some tables. <img width="1499" alt="image" src="https://github.com/user-attachments/assets/c346c721-02e3-4a5e-b5e9-00c1be8ba964";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
