qzyu999 opened a new pull request, #1625:
URL: https://github.com/apache/datafusion-python/pull/1625
# Which issue does this PR close?
Closes #1624.
# Rationale for this change
When using datafusion-python as a compute backend from multi-threaded Python
applications (e.g., PyIceberg), cloud credentials must currently be set via
`os.environ` before calling `register_parquet()`. Since `os.environ` is
process-global, concurrent threads mutating it causes credential
cross-contamination, requiring a global lock that serializes all I/O operations
and negates any parallelism benefit.
The existing `register_object_store()` method already solves thread-safety,
but requires callers to manually parse bucket names from paths, determine the
correct URL scheme, and make a separate call before `register_parquet`. This PR
adds a convenience `object_store` parameter that handles all of this internally.
# What changes are included in this PR?
1. **New private helper** `_register_object_store_for_path(path, store)`
parses a URL path via `urllib.parse.urlparse` to extract scheme and host, then
calls `register_object_store()` internally.
2. **New `object_store` parameter** added to all file-based register/read
methods:
- `register_parquet` / `read_parquet`
- `register_csv` / `read_csv`
- `register_json` / `read_json`
- `register_avro` / `read_avro`
- `register_arrow` / `read_arrow`
3. **Test suite** (`python/tests/test_object_store_param.py`) with 23 tests:
- 8 unit tests for URL parsing logic (s3, gs, az, https, error cases)
- 13 mock-based tests verifying each method correctly delegates to
`register_object_store`
- 2 end-to-end integration tests using `LocalFileSystem` with real
Parquet files
All changes are purely in the Python layer no Rust code modified.
# Are there any user-facing changes?
Yes. All file-based `register_*` and `read_*` methods on `SessionContext`
now accept an optional `object_store` keyword argument. When provided with a
pre-configured store instance (e.g., `AmazonS3`, `GoogleCloud`,
`MicrosoftAzure`), the store is automatically registered for the URL parsed
from the path.
Example usage:
\\\python
from datafusion import SessionContext
from datafusion.object_store import AmazonS3
ctx = SessionContext()
store = AmazonS3(bucket_name='my-bucket', region='us-east-1',
access_key_id=key, secret_access_key=secret)
ctx.register_parquet('my_table', 's3://my-bucket/data.parquet',
object_store=store)
\\\
This is a backward-compatible addition (the parameter defaults to `None` and
existing call sites are unaffected).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
