turbo1912 commented on PR #3540:
URL:
https://github.com/apache/arrow-datafusion/pull/3540#issuecomment-1252737721
> I think that rather than any options, the support for object_store should
be invisible to the user. So just running this SQL from datafusion-cli should
be enough to trigger the cli to register the s3 objectstore:
>
> ```
> create external table unicorns stored as parquet location
's3://my_bucket/lineitem/';
> ```
>
> You could do that by creating the logical plan from the user's given SQL,
then only registering an object store if they're creating an external location
that's in the cloud. Something like:
>
> ```rust
> let logical_plan = ctx.create_logical_plan(sql)?;
> match logical_plan {
> LogicalPlan::CreateExternalTable(external) => {
> if external.location.to_lowercase().starts_with("s3://") {
> // use
https://docs.rs/object_store/latest/object_store/path/struct.Path.html
> // parse the path, find the bucket
> // register the object store
> }
> }
> _ => {},
> }
> // continue on and execute the logical plan
> ```
Yes @kmitchener! I thought of this as well, but then wasn't sure if every
object-store type can be inferred from the url so took the less risky approach,
that doesn't close any doors for the future. For example, can
`file://localhost` mean multiple things?
That being said, looks like there is some [object-store
code](https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/datasource/object_store.rs#L151
) already that makes some assumptions in regards to the url.
I am happy to go with this approach if there is interest! @alamb, @tustvold
and @Jimexist any thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
