jtmzheng opened a new issue #2470: URL: https://github.com/apache/hudi/issues/2470
**Describe the problem you faced** We're seeing heavy skew in our Spark Streaming job when processing rollbacks (`mapToPair at ListingBasedRollbackHelper.java:100`). The example below took 2.2h to complete with a p75 of 15 minutes and a max of 2.2h (100 tasks, long tail of 8 tasks that took > 1 hour) <img width="1666" alt="Screen Shot 2021-01-20 at 11 36 04 AM" src="https://user-images.githubusercontent.com/3466206/105383583-4184f800-5bdf-11eb-8368-c345d452a6eb.png";> What can cause this skew and what can we do to alleviate/investigate this? We have Hudi configured with: ``` hudi_options = { "hoodie.table.name": "transactions", "hoodie.datasource.write.recordkey.field": "id.value", "hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator", "hoodie.datasource.write.partitionpath.field": "year,month,day", "hoodie.datasource.write.table.name": "transactions", "hoodie.datasource.write.table.type": "MERGE_ON_READ", "hoodie.datasource.write.operation": "upsert", "hoodie.consistency.check.enabled": "true", "hoodie.datasource.write.precombine.field": "publishedAtUnixNano", "hoodie.compact.inline": True, "hoodie.compact.inline.max.delta.commits": 10, "hoodie.cleaner.commits.retained": 1, } ``` **Environment Description** * Hudi version : 0.6.0 * Spark version : 2.4.6 (EMR 5.31) * Hive version : 2.3.7 * Hadoop version : Amazon 2.10.0 * Storage (HDFS/S3/GCS..) : S3 * Running on Docker? (yes/no) : no ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/3466206/105383583-4184f800-5bdf-11eb-8368-c345d452a6eb.png"](https://user-images.githubusercontent.com/3466206/105383583-4184f800-5bdf-11eb-8368-c345d452a6eb.png"){kind=link}