wForget commented on issue #11542: URL: https://github.com/apache/incubator-gluten/issues/11542#issuecomment-3850911804
> The VeloxSortShuffleWriter::getPeeledRowVector seems to be requesting a lot of memory. Does this mean I have a very large ColumnBatch or that the peeledRowVector is not being released early? This understanding is incorrect (but I don't understand why the call stack in the jemalloc memory dump shows getPeeledRowVector), a large amount of memory is held by `VeloxSortShuffleWriter.pages_`. ``` I20260205 11:43:05.898020 252506 VeloxSortShuffleWriter.cc:76] Received ColumnarBatch with 3850 rows and 7 columns. I20260205 11:43:05.898070 252506 VeloxSortShuffleWriter.cc:80] Writing RowVector with 3850 rows and 4857888 bytes. I20260205 11:43:05.898320 252506 VeloxSortShuffleWriter.cc:377] Acquire new buffer. current capacity: 67108768, size: 67108768, pageCursor: 67106391, unused: 2377 I20260205 11:43:05.898404 252506 VeloxSortShuffleWriter.cc:387] Allocated new buffer. capacity: 67108768, size: 67108768 I20260205 11:43:05.924463 252506 VeloxSortShuffleWriter.cc:88] After write, total pages: 19, total page bytes: 1275066592, velox pool used bytes: 1300234240 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
