zeroshade commented on issue #1446: URL: https://github.com/apache/arrow-go/issues/1446#issuecomment-6669843282
Thanks for the detailed analysis — I confirmed the allocation regression on current main. Reading the same 1M-row file (float64, int64 and binary columns, Snappy) through `pqarrow.ReadTable`: | Distinct values | v18.7.0 | main | with your fix | | --- | ---: | ---: | ---: | | 10k | 121.3 MiB | 122.7 MiB | 121.0 MiB | | 100k | 136.1 MiB | 147.8 MiB | 135.9 MiB | I didn't see the time regression on my machine (main was slightly faster than v18.7.0 there), so that part likely depends on the data. I've ported your fix to main in #1473 and credited you there. Review turned up two additional bounds that are in that PR too: an index the page's bytes can't reach is rejected before the dictionary grows, and the up-front allocation is capped at the page's own size so `ByteArray` dictionaries of short values can't preallocate far more than the page. On the writer observation: that change is intentional, from #1036. The cost-based dictionary fallback now runs only for uncompressed columns, because its uncompressed-size comparison doesn't predict compressed size. With a codec, the dictionary is kept until the page size limit forces a fallback. To get the previous behavior for a column, set `parquet.WithDictionaryCostFallbackFor(column, true)`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
