alamb commented on issue #907: URL: https://github.com/apache/arrow-datafusion/issues/907#issuecomment-902214790
> Keep separate crates, like datafusion-s3 which has implementations for some interfaces, and maybe also exposes some helper functions Yes I think separate crates is a very good idea to avoid datafusion requiring a massive dependency stack and to keep compile time reasonable. > How would you think of keeping the code as part of the workspace? I think this is a reasonable for "core" integrations, though going this path also subjects those crates to the apache arrow governance model (among other things slower release cycle) which may not be needed here Maybe we could start it as its own repo / separate crate with whoever does it and then we can figure out if we want to bring it back into apache arrow if any. > was should we use Rusoto (in maintenance mode) or the new AWS rust sdk (in Alpha, not on crates.io until GA). My current thinking is probably to use Rusoto for now, as it will probably be a while until the AWS SDK gets into GA, but want to hear everyone thoughts on this. FWIW we took this approach in IOx (starting with Rusto) and it has been working well for us: https://github.com/influxdata/influxdb_iox/blob/main/object_store/Cargo.toml#L21-L23 (we have been using it in AWS for several months now and we have not hit any issues with it) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
