loukey-lj opened a new pull request, #7815: URL: https://github.com/apache/hudi/pull/7815
### Change Logs The online job runs for 13 days and finds that there are subtasks but no data processing, as shown in the figure below, this job uses the update time as the partition, uses the bucket index, the number of buckets is 128, and the write parallelism is 128. The key is uniform because the file size of each bucket is not much different from the storage point of view. After positioning, there is a skew in the shuffle algorithm.  Potential disadvantages of algorithmic tilt: 1. The memory usage is uneven, some nodes may have high pressure on the JVM, and TM nodes are prone to timeout 2. It may cause the checkpoint to time out, because the data will be flushed to hdfs during the snapshot state. If the skew is serious, it will cause some nodes to take too long and cause timeout. current algorithm：  Algorithm flaws: 1. curBucket ∈ [0, numBuckets -1] 2. For the number of globalHash values in the same partition <= numBuckets number, globalHash is divergent, and mod(globalHash, numPartitions) is easy to conflict 3. When numBuckets is relatively large, shuffleIndex is prone to conflicts, resulting in skew Algorithm optimization:  kb = key % b; kb ∈ [0, b-1] pw = pt % w; pw ∈ [0, w-1] shuffleIndex = (pw + kb) % w shuffleIndex ∈ [0, w-1] In fact, it is to calculate a pw according to the partition first. Pw can be understood as a slot Wn allocated to the partition. Different partitions have a slot. Then move b slots back on the basis of this slot as the writing of data for this partition  ### Impact NA ### Risk level (write none, low medium or high below) NA ### Documentation Update NA - _The config description must be updated if new configs are added or the default value of the configs are changed_ - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website._ ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
