pitrou commented on issue #45038: URL: https://github.com/apache/arrow/issues/45038#issuecomment-2547906730
Hmm, the current behavior seems to be deliberate according to the [PR description](https://github.com/apache/arrow/pull/10955): > `FileSystemDatasetWriteOptions` now has a `max_open_files` setting (ARROW-12321) which prevents opening too many files. Instead the writer will apply backpressure on the scanner while also closing the open file with the greatest # of rows already written (then resume writing once the file is closed). @westonpace Do you happen to remember why the option description doesn't match the intended behavior? Is it just a mistake in the docstring? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
