lisasgoh commented on issue #7075:
URL: https://github.com/apache/arrow-rs/issues/7075#issuecomment-2637921175
@tustvold The Rust API to directly provide an object store makes sense, but
I'll be interacting with polars via python so I was wondering what's the best
way to register a new object store in python. I might be misunderstanding, but
regarding your idea for python, does it involve creating some sort of generic
ObjectStore python impl that wraps the user provided python impl and invokes
its methods? Something like:
`Python::with_gil(|py| {
let _: PyResult<()> = obj.call_method1("put", (location, bytes));
})`
I am worried that for this approach, doesn't this mean that the interactions
with the object store will be -> rust -> python -> rust? This would have some
overheard and I'm not sure if there would be performance implications due to
the GIL. It seems like libloading is already used in polars for plugins
https://docs.pola.rs/api/python/stable/reference/plugins.html, which is where I
got the idea from.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
