Good idea. However, there is some overlap with ObjectStorage too. OpenDAL looks to be a superset of ObjectStorage for sure, but we will need to figure out the messaging to users from POV of what they should be using.
On Mon, 19 May 2025 at 13:31, Pavankumar Gopidesu <gopidesupa...@gmail.com> wrote: > Yes Vikram we can discuss. > > Pavan, > > On Mon, May 19, 2025 at 12:55 AM Vikram Koka <vik...@astronomer.io.invalid > > > wrote: > > > Pavan, > > > > From a concept perspective and strategic direction, I am in 100% > agreement. > > I have also been thinking about this and submitted a talk for the Airflow > > Summit on this topic. > > > > I am unsure of this particular package, but will look into it to > understand > > more. > > > > The place where I do have concerns with the approach is on the interface > > and its integration with other providers. We have tried a very similar > > approach in the past with the Universal Transfer Operator and it had > > limited resonance / adoption. I have also seen other similar approaches > > proposed since and with limited uptick. > > > > This is definitely a key topic I would like to collaborate on, so let’s > > sync offline and get back to the dev list. > > > > Best regards, > > Vikram > > > > On Sun, May 18, 2025 at 11:35 AM Shahar Epstein <sha...@apache.org> > wrote: > > > > > +1 from me - the project seems well-maintained, and we should > definitely > > > collaborate and open the "providers" door to Apache projects that could > > fit > > > well in Apache Airflow like this one. > > > If it proves itself well in the future, maybe we could think of > > deprecating > > > the "transform" operators in favor of this one, or at least make their > > > execution method to utilize it. > > > > > >
