codejoyan edited a comment on issue #2620: URL: https://github.com/apache/hudi/issues/2620#issuecomment-886077052
Some additional details for the above runs. 1. The configs I am using - REGULAR BLOOM. 2. Max and Min file size in older partitions - 116 MB and 6 MB respectively 3. Avg record size - 50 bytes 4. Avg no of data files in older partitions - Between 157 to 225 I then changed the configs as below to have roughly 100k entries per file. But the performance is worse now. It basically gets stuck. Attached Spark Web UI screenshot 1. **Configs** - hoodie.insert.shuffle.parallelism - 1500 hoodie.upsert.shuffle.parallelism - 1500 hoodie.parquet.small.file.limit - 4200000 hoodie.parquet.max.file.size - 5000000 hoodie.index.type - BLOOM hoodie.copyonwrite.record.size.estimate - 50 hoodie.copyonwrite.insert.split.size - 100000 hoodie.bloom.index.prune.by.ranges - false hoodie.bloom.index.filter.type - DYNAMIC_V0 hoodie.index.bloom.num_entries - 30000 The performance is now okay witth BLOOM index when the incremental batch size is around 100 MB (around 4-5 mins for upsert). But it gets worse when batch size increases (> 5 GB) and the countByKey at BaseSparkCommitActionExecutor.java:154 step gets stuck. ** - What is the general rule of thumb between file size, insert split and incremental batch size? - Also this also proves that COW table is not good when the incremental batch size varies in size. Am I correct? ** <img width="1590" alt="Screenshot 2021-07-24 at 10 01 31 PM" src="https://user-images.githubusercontent.com/48707638/126875026-8c92b70e-6d83-43e3-966f-26a71c7a156f.png";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/48707638/126875026-8c92b70e-6d83-43e3-966f-26a71c7a156f.png"](https://user-images.githubusercontent.com/48707638/126875026-8c92b70e-6d83-43e3-966f-26a71c7a156f.png"){kind=link}