timsaucer opened a new issue, #1737:
URL: https://github.com/apache/datafusion-python/issues/1737
# Is your feature request related to a problem or challenge?
Split out of #1676, which proposed adding an `object_stores` field to
`SessionExtensionComponents` so an extension library could declare its object
stores alongside its codecs, functions, and providers and have
`with_extensions` install everything in one call. The other fields in that
issue are actionable and are being implemented; this one is blocked, so it is
being tracked separately rather than holding up the rest.
An extension library that reads from a storage system datafusion-python does
not know about has no way to contribute an object store. Every other extension
point in the FFI surface — table providers, catalog providers, functions,
codecs, query planners, physical optimizer rules — has a `__datafusion_*__`
capsule getter that lets a separately compiled library hand over an
implementation. Object stores have none.
# Describe the solution you'd like
Ultimately, a `__datafusion_object_store__` capsule getter following the
same convention as the rest of the protocol, so a library can export an
`ObjectStore` implementation across the FFI boundary, plus an `object_stores`
field on `SessionExtensionComponents` keyed by scheme.
**This is blocked upstream and cannot be built here first.** There is no
`FFI_ObjectStore` in `datafusion-ffi` — no object-store module exists in the
crate at all. Without one there is nothing for a capsule to carry.
The Python-side surface is also closed today.
`SessionContext.register_object_store` takes `StorageContexts`, a closed enum
over five built-in pyclasses (`AmazonS3`, `GoogleCloudStorage`,
`MicrosoftAzure`, `LocalFileSystem`, `HTTP`), and `crates/core/src/context.rs`
matches all five exhaustively to pull out an `Arc<dyn ObjectStore>`. There is
no `RustWrappedPyObjectStore` equivalent to the wrappers the other extension
points have. So even a Python-native path would need new work, and a
third-party Rust cdylib could not participate at all — it would have to import
`datafusion.object_store` and call back into the host to construct one of our
own objects.
The dependency order is therefore: an `FFI_ObjectStore` upstream in
`datafusion-ffi`, then an importer and `__datafusion_object_store__` hook here,
then optionally the `object_stores` bundle field.
# Describe alternatives you've considered
**What works today.** A library that wants to ship a configured store
returns one of datafusion-python's own objects and lets the caller register it:
```python
ctx.register_object_store("s3://my-bucket", my_library.configured_s3_store())
```
That is one line and needs no protocol. It covers the case where the library
is packaging credentials or endpoint configuration for a store
datafusion-python already supports, which is probably the common case.
What it does not cover is a library implementing a genuinely new
`ObjectStore` — an internal blob service, a content-addressed store, a caching
layer in front of another store. That case needs the FFI type and has no
workaround short of the library vendoring its own DataFusion.
**Adding `object_stores` to `SessionExtensionComponents` anyway**, carrying
the five existing `StorageContexts` variants. Rejected in #1676: it would be
the only field in that dataclass carrying nothing foreign, there would be no
capsule to validate, and it would bake the `format!("{scheme}{derived_host}")`
key construction into a tuple shape for no benefit over the one-line call above.
# Additional context
Follow-up to #1676; see
https://github.com/apache/datafusion-python/issues/1676#issuecomment-5680959562
for the analysis this was split out of. Blocked on an upstream `datafusion-ffi`
change — worth raising in apache/datafusion before any work starts here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
