jonasdedden commented on issue #286: URL: https://github.com/apache/arrow-rs-object-store/issues/286#issuecomment-6337856113
Hey @tustvold, A while ago I had a look how to add Lance support to LakeSail: https://github.com/lakehq/sail/issues/2573 It's a somewhat long discussion, but the TL;DR was that it would be optimal if there were some stable `object_store` ABI, because then Lance could directly use the object store interfaces of Sail that already provide a lot of value around caching, authentication and more that are nicely configurable through pure Spark logic already. An alternative would be to statically compile Lance and all its dependencies into the Sail binary, which not only locks all dependency versions exactly to what Lance needs, but also massively increases binary size. In your [last comment](https://github.com/apache/arrow-rs-object-store/issues/286#issuecomment-2741757166) you highlight that such an ABI layer actually could be somewhat simple, at least simpler than initially thought. Granted, all of this is entirely vibecoded end-to-end, and I never worked on stable ABIs, but I believe this could be an implementation with a bunch of examples as a first PoC of what you thought of => https://github.com/jonasdedden/object-store-abi/tree/main -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
