RinChanNOWWW commented on issue #4864: URL: https://github.com/apache/arrow-rs/issues/4864#issuecomment-1737029785
> The challenge is bounding the memory usage of the reader and balancing this against the cost of decoding We can only save the arrays we need to output and release the others. > which for primitives is relatively low Besides decoding there is also decompression. And it may become a large cost for variable-length type like string. > it will probably require some sort of heuristic Yes. Such as all the output arrays are contained in predicate arrays. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
