houqp commented on issue #1532: URL: https://github.com/apache/arrow-datafusion/issues/1532#issuecomment-1008356293
Thank you @alamb for bringing this up! > I believe the current proposal is to make an official arrow branch in datafusion: #68 (comment), which is probably a step towards switching to arrow2 Yes, this aligns with what I have in mind. The official arrow2 branch was proposed so we can close that long running PR and have a centralized location for folks to collaborate on the migration until we are happy with the master merge. If the community is happy with merging directly into master and iterate there, that would work as well. > Will arrow-rs eventually support async file IO? Requiring a synchronous ChuckReader is currently a major limitation in supporting alternate ObjectStores I believe so. However, we could probably save this work and get it for free with the arrow2 switch. > I guess, what are the reasons switching would be a bad idea? Like what is the delta between what they both currently provide? IMHO, the main downside is the switch cost and downstream impact. But I think it's a one time cost that's worth paying. I think arrow2 at this point should have covered most of all our needs in datafusion as demonstrated in https://github.com/apache/arrow-datafusion/pull/68. All unit and integration tests are passing at the moment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
