2010YOUY01 commented on issue #11472: URL: https://github.com/apache/datafusion/issues/11472#issuecomment-2230833594
> I haven't dug into the implementation, but I imagine it becomes harder to find the right split point for multi-threaded reading That is correct, DataFusion parallel CSV scan does: 1. For example it gets a 1000B CSV file and wants to read in 2 threads. 2. File is split into [0, 500), [500, 1000) two ranges, then probe the actual line boundary and adjust the range 3. Let `arrow-rs` handle reading a valid file byte range. So I think the first step will be to add arrow support -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
