advancedxy commented on PR #428: URL: https://github.com/apache/incubator-uniffle/pull/428#issuecomment-1363729040
> And I still can't find the key of this pr to solve the negative preAllocatedSize. Could you help describe more? @advancedxy Sorry, let me add a bit of details here. Before this pr, there are two places to decrease `preAllocatedSize`. 1. sendShuffleData -> one of multiple cache shuffle data -> decrease `preAllocatedSize`, and also removes the require buffer id 2. `preAllocatedBufferCheck` in shuffleTaskManager, which checks preAllocated is expired or not, then removes it in another check. Some cases could cause double release of `preAllocatedSize`: 1. anything went wrong in one of the cacheShuffleData calls(except the first one), then preAllocatedSize is partially decreased, then the whole size would be decreased in `preAllocatedBufferCheck` 2. `preAllocatedBufferCheck` checks one buffer expired, add it to `removeIds`. Client sends the `sendShuffledata` and that finishes first -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
