akoumjian commented on issue #39808: URL: https://github.com/apache/arrow/issues/39808#issuecomment-2935328204
For what it's worth, my team still avoids using Dataset entirely due to this issue. Instead we have implemented our own file discovery, scanning, and concatenation. It is highly unlikely this is intended behavior, as this is not about peak memory usage as in that comment but unreleased memory many times greater than the uncompressed data should represent. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
