Cheappie commented on issue #2185: URL: https://github.com/apache/arrow-datafusion/issues/2185#issuecomment-1094912949
First of all I am just a stranger that evaluates datafusion query engine, I might lack some context so my point might not be valid for this case. Yes sure that make sense. From what I see writer API adds point of failure to the upstream. For example how is It going to deal with data loss in case of process crash or missing permissions for write to the s3 bucket, etc... ? ObjectStore that just performs reads cannot corrupt datasource and from my perspective that is great. I would suggest to push this cross FS implementation into Rust Arrow repository same as C++ did then implementation would be even more reusable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
