zhangfengcdt commented on code in PR #44:
URL: https://github.com/apache/sedona-db/pull/44#discussion_r2337238280
##########
rust/sedona/src/context.rs:
##########
@@ -232,18 +231,21 @@ impl SedonaContext {
options: GeoParquetReadOptions<'_>,
) -> Result<DataFrame> {
let urls = table_paths.to_urls()?;
- let provider =
- match geoparquet_listing_table(&self.ctx, urls.clone(),
options.clone()).await {
- Ok(provider) => provider,
- Err(e) => {
- if urls.is_empty() {
- return Err(e);
- }
-
- ensure_object_store_registered(&mut self.ctx.state(),
urls[0].as_str()).await?;
- geoparquet_listing_table(&self.ctx, urls, options).await?
- }
- };
+
+ // Pre-register object store with our custom options before creating
GeoParquetReadOptions
+ if !urls.is_empty() {
+ use
crate::object_storage::ensure_object_store_registered_with_options;
+ // Extract the table options from GeoParquetReadOptions for object
store registration
+ let table_options_map =
options.table_options().cloned().unwrap_or_default();
+ ensure_object_store_registered_with_options(
+ &mut self.ctx.state(),
+ urls[0].as_str(),
+ Some(&table_options_map),
+ )
Review Comment:
Yes, I think currenly due to the way we register the storage store by
scheme, once the store is initialized with nosig, it cannot be used to access
the private bucket. However, if user initializes it with private credentials
and then use it to access the public bucket, it will work.
For future fix, we can change to register the storage store by bucket (e.g.,
s3://bucket_name) instead of s3://, this will enable use to use different
access patterns (public vs private) for different buckets.
I will add a todo comment to implement this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
