r4ntix commented on code in PR #5732:
URL: https://github.com/apache/arrow-datafusion/pull/5732#discussion_r1151342705
##########
datafusion/core/src/datasource/listing_table_factory.rs:
##########
@@ -138,6 +142,73 @@ impl TableProviderFactory for ListingTableFactory {
.with_file_sort_order(file_sort_order);
let table_path = ListingTableUrl::parse(&cmd.location)?;
+
+ // try obtaining all relevant information of object store from
cmd.options
+ match table_path.scheme() {
+ "s3" => {
+ let url: &Url = table_path.as_ref();
+ let bucket_name = url
+ .host_str()
+ .ok_or(DataFusionError::External("invaild bucket
name".into()))?;
+ let mut builder =
+ AmazonS3Builder::from_env().with_bucket_name(bucket_name);
+
+ if let (Some(access_key_id), Some(secret_access_key)) = (
+ cmd.options.get("access_key_id"),
+ cmd.options.get("secret_access_key"),
+ ) {
+ builder = builder
+ .with_access_key_id(access_key_id)
+ .with_secret_access_key(secret_access_key);
+ }
+
+ if let Some(session_token) = cmd.options.get("session_token") {
+ builder = builder.with_token(session_token);
+ }
+
+ if let Some(region) = cmd.options.get("region") {
+ builder = builder.with_region(region);
+ }
+
+ let store = Arc::new(builder.build()?);
+
+ state
+ .runtime_env()
+ .register_object_store(table_path.as_ref(), store);
Review Comment:
Thanks for the suggestion!
Considering that there is already a dependency on object_store in the
current datafusion core, I think it would be a natural thing to support
different object store implementations directly in datafusion core.
@alamb @yjshen
Is it possible to make this as an optional feature of datafusion core? That
way it would be optional and out-of-the-box for CLI or other services that use
Datafusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
