JingsongLi commented on PR #8660: URL: https://github.com/apache/paimon/pull/8660#issuecomment-5058711282
The resizing logic has been moved into all `AbstractWritableVector.reset()` methods, but its utilization depends on `elementsAppended`; the path that directly calls the setter does not automatically maintain this value, making it easy to misclassify a vector with millions of elements that has been used normally as “virtually unused,” resulting in repeated resizing and resizing back up in subsequent batches. Current tests manually call `addElementsAppended` and do not cover real-world reader/vector-tree reuse scenarios. Another issue is that even when the row capacity of `HeapBytesVector` is reduced, the largest `buffer` byte array is fully retained, meaning memory for large objects is not fully released as intended. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
