+1 for an OpenDALFileIO -Jack
On Fri, Aug 2, 2024 at 8:32 AM Xuanwo <xua...@apache.org> wrote: > Hi, renjie > > Thank you for your support. I'll delve into the details and first build a > PoC PR to make it clear. > > On Fri, Aug 2, 2024, at 22:51, Renjie Liu wrote: > > Hi: > > Thanks Xuanwo for raising this. > > As mentioned in another thread, I think using iceberg-rust in pyiceberg is > a good idea. > > About this idea, would you have a more detailed design? For example, > where should the pyo3 codes live, in iceberg-rust or in pyiceberg? What > kind of interface should we provide to pyiceberg, FileIO or OpenDAL? > > I think this is a good first step moving forward to make pyiceberg backed > iceberg-rust. In the future we can replace components gradually. > > On Fri, Aug 2, 2024 at 5:58 PM Xuanwo <xua...@apache.org> wrote: > > > > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL > implementations via pyo3 / fsspec bindings > <https://github.com/apache/opendal/issues/4511>? > > Hi, Raschkowski, good question! > > It's possible. There is an ongoing project developing fsspec bindings for > opendal at https://github.com/fsspec/opendalfs. Once complete, we can > directly use opendal through fsspec. > > This work is unrelated to Pyicberg or Iceberg-rust. Ideally, users should > be able to use opendalfs as an alternative implementation of the fsspec > AbstractFileSystem class. > > On Fri, Aug 2, 2024, at 17:44, Will Raschkowski wrote: > > Xuanwo, would PyIceberg and iceberg-rust share the underlying OpenDAL > implementations via pyo3 / fsspec bindings > <https://github.com/apache/opendal/issues/4511>? > > > ------------------------------ > > *From:* Joe Stein <crypt...@gmail.com> > *Sent:* Thursday, August 1, 2024 3:37 AM > *To:* dev@iceberg.apache.org <dev@iceberg.apache.org> > *Subject:* Re: [DISCUSS] Use iceberg-rust as pyiceberg file io > > *CAUTION:* This email originates from an external party (outside of > Palantir). If you believe this message is suspicious in nature, please use > the "Report Message" button built into Outlook. > > Kafka did this with librdkafka and was wildly successful. The underlying > bindings being in rust are great with a layer for access in Python +1 > > > ~ Joe Stein > > > On Wed, Jul 31, 2024 at 10:29 PM Xuanwo <xua...@apache.org> wrote: > > Hello everyone > > I start this thread to discuss the idea about using iceberg-rust as > pyiceberg file io. > > The idea is living at https://hackmd.io/@xuanwo/iceberg_rust_as_file_io > [hackmd.io] > <https://urldefense.com/v3/__https://hackmd.io/@xuanwo/iceberg_rust_as_file_io__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTPd6ppTI$> > > In summary, we can leverage the work from iceberg-rust to help pyiceberg > in developing a fast and compact file IO system that benefits users with > specific constraints. > > Welcome to join in the discussion. > > Xuanwo > > https://xuanwo.io/ [xuanwo.io] > <https://urldefense.com/v3/__https://xuanwo.io/__;!!NkS9JGVQ2sDq!7Js41FIzh2smsAOySXrKd527DXCmXdrwV8Uov8TIdQqLRcsCkfPnHzfbxbX_xctpoNpYw2XGfrduTNspr1jI$> > > Xuanwo > > https://xuanwo.io/ > > Xuanwo > > https://xuanwo.io/ > >
