blaginin commented on PR #17076: URL: https://github.com/apache/datafusion/pull/17076#issuecomment-3170664919
I agree with the idea overall: - `batch_size` doesn’t seem to me like a `FileSource` property; it’s rather a `FileScanConfig` param (and I feel like it’ll be even more straightforward once we do the big refactoring @adriangb mentioned) - `FileSource` doesn’t currently have any methods to inspect the current `batch_size`, so the user impact shouldn’t be too big Happy for us to merge this, but I feel like we may need to agree on the final approach first. If we’ll end up rewriting this in the next release, should we make small incremental updates now? I fear it may be annoying for library users to face a breaking change in every release - especially in the world of File Scan configs, which are a bit messy / hard to understand right now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
