alamb commented on PR #5610: URL: https://github.com/apache/arrow-rs/pull/5610#issuecomment-2046111670
> I believe the intention of this code is to avoid writing out data buffers that aren't referenced by any views, perhaps as a result of slicing. That is my interpretation too -- @ariesdevil can you confirm? > I think this is probably better handled as a more general "compact" operation, that also handles pruning the data buffers themselves. I believe this is tracked by https://github.com/apache/arrow-rs/issues/5513 Yes, that is my suggestion. The rationale for a separate gc operation is: 1. Since it is potentially expensive I think the user should decide when to do the check, rather than on each IPC call. I think some applications might have different tradeoffs between compacting / pruning and saving IPC space. 2. `gc` is more generally useful for operations other than IPC (consolidating memory down after a filter, for example) Having a variation of `gc` that simply prunes unreferenced buffers (but not copy) might also be interesting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
