Finally caught up with this one, looked through code and discussions. I am a little torn on that one but I did some more research and I think it's a useful abstraction.
+1(binding) The big + of using fsspec is that it is already supported by the most important "consumers" that are likely to be used in Airflow. Pandas, Pyarrow, Iceberg. The fact that you will be able to take an S3/GCS ObjectStoragePath as an input directly and it will transparently use the connection of Airflow is a big plus. I would just add that we should get real-life DAG examples on how this might simplify code of their DAGs, it's cool. I think the quality and clarity of the documentation that will come with it - clearly explaining some cases and examples on how DAG authors can make use of it to make their DAG authoring "better" - is a key to success of this one. If we fail to explain it, it might become yet another rarely used feature of Airflow There is one worry I have - it adds "yet another abstraction" to learn and "yet another set of dependencies" to Airflow. We have a new "common.io" provider, we have many new dependencies, we have aiobotocore as a requirement for AWS integration for example. I already looked at the PR and attempted to help with some of the dependency questions and problems. but we will have a few more of those to solve and some decisions to mke should apache-airflow-provider-common-io be default? Should it be included in the reference image? etc. etc. This will make Airflow and its dependencies more complex than simpler. That's why I am not strong +1! just measly +1 - because I see how it can make airflow even "heavier" than it is now. J. On Thu, Oct 19, 2023 at 4:34 PM Igor Kholopov <ikholo...@google.com.invalid> wrote: > Thanks for incorporating the feedback! > > +1 (non-binding) > > On Thu, Oct 19, 2023 at 1:55 PM Dennis Akpenyi <dennisakpe...@gmail.com> > wrote: > > > +1 (non-binding) > > > > On Thu, Oct 19, 2023 at 12:24 PM Bolke de Bruin <bdbr...@gmail.com> > wrote: > > > > > Dear Community, > > > > > > I would like to start a vote for "AIP-58 Add Airflow ObjectStore". > > > > > > You can find the AIP here: > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263430565 > > > > > > Implementing PR (most of the discussion happened here): > > > https://github.com/apache/airflow/pull/34729 > > > > > > Discussion Thread (not much has happened here :-) ): > > > Note: the title has changed from its original. > > > > > > https://lists.apache.org/thread/l3fkr0h6j2g4tlmsov14fywmj58t3mtp > > > > > > This is my binding +1m the vote will last until 12:00 UTC on 26th > > October, > > > and until at least 3 binding votes have been cast. > > > > > > Please vote accordingly: > > > > > > [ ] + 1 approve > > > [ ] + 0 no opinion > > > [ ] - 1 disapprove with the reason > > > > > > Only votes from PMC members and committers are binding, but other > members > > > of the community are encouraged to check the AIP and vote with > > > "(non-binding)". > > > > > > Cheers > > > Bolke > > > -- > > > > > > -- > > > Bolke de Bruin > > > bdbr...@gmail.com > > > > > >
