wankunde commented on PR #12089: URL: https://github.com/apache/gluten/pull/12089#issuecomment-4451196954
> > but evictPartitionBuffersMinSize() method will be skipped due to splitState_ == kInit condition. > > Can you explain a bit more about this? According to the code logic, `evictPartitionBuffersMinSize()` is used for reclaiming memory when `splitState_ == kInit` > > https://github.com/apache/gluten/blob/main/cpp/velox/shuffle/VeloxHashShuffleWriter.cc#L1389-L1391 When the shuffle writer is in `doSplit()` method, and split state may be kPreAlloc or kSplit, hash shuffle writer can not reclaim memory from itself. | spill number | split state | memory used | comment | | -------------- | ------------- | ------------- | ------------------------------------------------------------------- | | 0 | kInit | 0MB | | | 0 | kPreAlloc | 100MB | | | 0 | kSplit | 150MB | | | 1 | kInit | 150MB | | | 1 | kPreAlloc | 250MB | | | 1 | kSplit | 300MB | | | 2 | kInit | 300MB | | | 2 | kPreAlloc | 450MB | case1 : ShuffleWriter used + some other used > task offheap total | | 2 | kSplit | 500MB | case2 : ShuffleWriter used + some other used > task offheap total | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
