qzyu999 opened a new issue, #1623:
URL: https://github.com/apache/datafusion-python/issues/1623
## Is your feature request related to a problem or challenge?
When using datafusion-python as a compute backend from multi-threaded Python
applications (e.g., Apache Iceberg's PyIceberg), we need to read Parquet files
from cloud storage (S3/GCS/ADLS) in parallel across multiple threads. Each
thread creates its own SessionContext and calls egister_parquet().
Currently, **cloud credentials must be set via os.environ** before calling
egister_parquet() (the Rust object_store crate reads AWS_ACCESS_KEY_ID, etc.
from the environment). Since os.environ is process-global, concurrent threads
mutating it causes credential cross-contamination.
**The workaround** is a global hreading.RLock() that serializes all
file-based DataFusion operations effectively negating the benefit of the
thread pool. A scan over 200 S3 Parquet files runs ~4-8 slower than it should
because only one thread can read at a time.
## Describe the solution you'd like
Add an optional \object_store\ parameter to \ egister_parquet()\ (and other
\ egister_*\ / \ ead_*\ methods) that accepts a pre-configured store instance:
\\\python
from datafusion import SessionContext
from datafusion.object_store import AmazonS3
# Thread-safe: no os.environ mutation needed
ctx = SessionContext()
store = AmazonS3('my-bucket', region='us-east-1', access_key_id=key,
secret_access_key=secret)
ctx.register_object_store('s3://', store, host='my-bucket')
ctx.register_parquet('my_table', 's3://my-bucket/data.parquet')
\\\
I know \ egister_object_store()\ already exists (and works!), but the
two-step dance of \ egister_object_store\ + \ egister_parquet\ requires the
caller to:
1. Parse the bucket name from the file path
2. Determine the correct scheme (\s3://\, \gs://\, \z://\)
3. Call \ egister_object_store\ before \ egister_parquet\
A single \object_store=\ keyword on \ egister_parquet\ would handle all of
this internally.
## Current workaround (in PyIceberg)
\\\python
import threading, os
_ENV_LOCK = threading.RLock()
def read_parquet_with_credentials(ctx, path, io_properties):
env_vars = translate_properties_to_env(io_properties)
with _ENV_LOCK: # Serializes ALL concurrent reads
for k, v in env_vars.items():
os.environ[k] = v
try:
ctx.register_parquet('source', path)
result = ctx.sql('SELECT * FROM source').to_arrow_table()
finally:
for k in env_vars:
os.environ.pop(k, None)
return result
\\\
## Describe alternatives you've considered
1. **Use \ egister_object_store\ directly** This works for S3 (\AmazonS3\
accepts inline creds), but \GoogleCloud\ only accepts \service_account_path\
(no inline token/JSON), making it insufficient for GCS with ephemeral
credentials.
2. **Wait for PR #1476** The draft PR integrates \pyo3-object_store\ which
fully solves this, but it's a large change (361 lines) and has been in draft
for 3 months.
3. **Minimal fix** Add \object_store=\ parameter to \ egister_parquet\ that
calls \ egister_object_store\ internally after parsing the URL. This is ~20
lines of Python.
## Additional context
- Related: #899 (general ObjectStore ergonomics), PR #1476 (full
pyo3-object_store integration)
- Downstream consumer:
[apache/iceberg-python](https://github.com/apache/iceberg-python) pluggable
backend
- DataFusion version: 54.0.0
- The \AmazonS3\, \GoogleCloud\, \MicrosoftAzure\ classes already exist in
\datafusion.object_store\ the gap is just the ergonomics of passing them to \
egister_parquet\ and filling in \GoogleCloud\'s credential options
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
