XiangpengHao commented on issue #5855: URL: https://github.com/apache/arrow-rs/issues/5855#issuecomment-2155066031
On SIMD decoding varint: https://github.com/as-com/varint-simd Memory overhead does not show in flamegraph as it is implicit (caused by cpu memory stalls rather than function calls). It is mostly based on speculations... In https://github.com/apache/arrow-rs/issues/5854#issuecomment-2154921008 I discussed a simple trick that improves the performance by up to 30%, by reducing the memory footprint and reduce implicit memory movement. On memory bandwidth: it takes 300ms to decode 100k columns, which allocates 600MB memory, that is ~2GB per second. Getting rid of/sharing small allocations won't help much because the majority of the memory consumption comes from `ColumnChunk` which is currently 420 bytes -> 420B * 100k * 10 row group = 420MB. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
