Yes kaxil, agree on some overlap and we should provide the right message to users to help them choose what to use when.
OpenDAL comes with predefined backends(ex: s3, gcs, fs, ghac, memcached etc;), all these backends work on the interface file like operations, so from users POV they have to configure which backend to use and connection string and what action they want to perform read/write etc;, then rest works out of the box. When using databases as key-value storage, OpenDAL supports reading and writing data seamlessly. However, to align with OpenDAL’s design, the key column must follow a file path-like structure*, *essentially a string that looks like a file path (e.g., documents/file.txt). The value column should store the corresponding binary data, typically using a type like BYTEA in PostgreSQL. This setup allows OpenDAL to perform file-style operations (read, write, list, delete) on database storage just as it would with traditional file systems or object stores. Some more context about OpenDAL db support[1] @shahar Epstein, Yes i could see with OpenDAL interface the reading and writing part is simpler it can be a read from http and write to s3 etc; [1] https://xuanwo.io/2023/03-opendal-database-support/ Pavan On Mon, May 19, 2025 at 4:59 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > Good idea. However, there is some overlap with ObjectStorage too. OpenDAL > looks to be a superset of ObjectStorage for sure, but we will need to > figure out the messaging to users from POV of what they should be using. > > On Mon, 19 May 2025 at 13:31, Pavankumar Gopidesu <gopidesupa...@gmail.com > > > wrote: > > > Yes Vikram we can discuss. > > > > Pavan, > > > > On Mon, May 19, 2025 at 12:55 AM Vikram Koka > <vik...@astronomer.io.invalid > > > > > wrote: > > > > > Pavan, > > > > > > From a concept perspective and strategic direction, I am in 100% > > agreement. > > > I have also been thinking about this and submitted a talk for the > Airflow > > > Summit on this topic. > > > > > > I am unsure of this particular package, but will look into it to > > understand > > > more. > > > > > > The place where I do have concerns with the approach is on the > interface > > > and its integration with other providers. We have tried a very similar > > > approach in the past with the Universal Transfer Operator and it had > > > limited resonance / adoption. I have also seen other similar approaches > > > proposed since and with limited uptick. > > > > > > This is definitely a key topic I would like to collaborate on, so let’s > > > sync offline and get back to the dev list. > > > > > > Best regards, > > > Vikram > > > > > > On Sun, May 18, 2025 at 11:35 AM Shahar Epstein <sha...@apache.org> > > wrote: > > > > > > > +1 from me - the project seems well-maintained, and we should > > definitely > > > > collaborate and open the "providers" door to Apache projects that > could > > > fit > > > > well in Apache Airflow like this one. > > > > If it proves itself well in the future, maybe we could think of > > > deprecating > > > > the "transform" operators in favor of this one, or at least make > their > > > > execution method to utilize it. > > > > > > > > > >
