zeevm commented on PR #4156: URL: https://github.com/apache/arrow-rs/pull/4156#issuecomment-1554231892
Understood. My considerations are: 1. Using arrow I'll have to allocate double the memory and incur a lot of mem copies into our engine native in memory format. 2. Our engine already does predicate push down, late materialization, aplying predicates to dictionaries and very fast scanning of raw dictionary IDs (skipping dictionary decoding entirely) with SIMD instructions and other optimizations Get Outlook for Android<https://aka.ms/AAb9ysg> ________________________________ From: Raphael Taylor-Davies ***@***.***> Sent: Friday, May 19, 2023 11:05:15 AM To: apache/arrow-rs ***@***.***> Cc: zeevm ***@***.***>; Comment ***@***.***> Subject: Re: [apache/arrow-rs] Cleanup ChunkReader (#4118) (PR #4156) Correct, the high-level API reads into arrow and supports object stores, in addition to predicate pushdown, late materialization, row filtering, etc... The low-level APIs support advanced usage, but are not nearly as batteries included. I would encourage you to try out the arrow APIs, you may find they are actually faster 😅 — Reply to this email directly, view it on GitHub<https://github.com/apache/arrow-rs/pull/4156#issuecomment-1554200543>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAA33OPY2XVBQOVZ6PSXOC3XG4STXANCNFSM6AAAAAAXPRRIHY>. You are receiving this because you commented.Message ID: ***@***.***> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
