huangxiaopingRD commented on PR #8660:
URL: https://github.com/apache/paimon/pull/8660#issuecomment-4988987177
> Aren't all vectors the same size? Why do we need to scale them back down?
For nested types, the parent vector only stores top-level metadata such as
nulls, offsets, and lengths. The child vector stores the actual elements and
can grow much larger than the parent.
For example, an `array<int>` vector with 4K top-level rows may have a child
`int` vector containing millions of array elements.
When the parent vector is reset, Paimon also resets writable child vectors.
This is implemented in `AbstractStructVector.reset()`, which iterates over
`children` and calls `reset()` on each child that implements
`WritableColumnVector`.
So the shrink logic is not assuming that child vectors are the same object
as the parent. Each vector still resets itself independently. The point is that
a child vector may retain a very large backing array after one unusually large
batch, and because the vector tree is reused for later batches, that retained
array can keep consuming heap even when later batches are small.
The change only scales down vectors whose capacity crossed the huge-vector
threshold. Normal-sized expanded vectors are still retained for reuse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
