Thanks for thorough consideration Jarek. I follow your concerns. The idea behind this AIP was to reduce the cognitive load on users by staying as much pythonic as we can and to be gentle with the Airflow-isms. So I hope to limit that "yet another abstraction". I do agree that having great examples and documentation are going to be important. As a random idea, this https://medium.com/@fninsiima/de-mini-series-part-two-57770ff7cdf9 , can now be significantly simplified.
One less worry I hope is that aiobotocore is actually starting to relax its botocore requirements bringing it much closer to latest release: https://github.com/aio-libs/aiobotocore/pull/1037 On the requirements side there are actually not that many additional dependencies being brought in. Core fsspec does not bring any requirements. s3fs brings in three which are all covered by current ones. adlfs brings in five, all already part of our current set. Of course it does bring some complexity, but I do hope you see that it is fairly limited and if it does bring in anything it is well supported. The reason for creating common.io as a provider was that it was suggested that we might want to move a bit faster than core on the very simple (yet powerful ;-) ) FileTransferOperator. Considering this I hope you would like to make your measly +1 into a strong +1 :-). Cheers Bolke On Thu, 19 Oct 2023 at 19:48, Jarek Potiuk <ja...@potiuk.com> wrote: > Finally caught up with this one, looked through code and discussions. I am > a little torn on that one but I did some more research and I think it's a > useful abstraction. > > +1(binding) > > The big + of using fsspec is that it is already supported by the most > important "consumers" that are likely to be used in Airflow. Pandas, > Pyarrow, Iceberg. The fact that you will be able to take an S3/GCS > ObjectStoragePath as an input directly and it will transparently use the > connection of Airflow is a big plus. > > I would just add that we should get real-life DAG examples on how this > might simplify code of their DAGs, it's cool. I think the quality and > clarity of the documentation that will come with it - clearly explaining > some cases and examples on how DAG authors can make use of it to make their > DAG authoring "better" - is a key to success of this one. If we fail to > explain it, it might become yet another rarely used feature of Airflow > > There is one worry I have - it adds "yet another abstraction" to learn and > "yet another set of dependencies" to Airflow. We have a new "common.io" > provider, we have many new dependencies, we have aiobotocore as a > requirement for AWS integration for example. I already looked at the PR and > attempted to help with some of the dependency questions and problems. but > we will have a few more of those to solve and some decisions to mke should > apache-airflow-provider-common-io be default? Should it be included in the > reference image? etc. etc. This will make Airflow and its dependencies more > complex than simpler. That's why I am not strong +1! just measly +1 - > because I see how it can make airflow even "heavier" than it is now. > > J. > > > > On Thu, Oct 19, 2023 at 4:34 PM Igor Kholopov <ikholo...@google.com.invalid > > > wrote: > > > Thanks for incorporating the feedback! > > > > +1 (non-binding) > > > > On Thu, Oct 19, 2023 at 1:55 PM Dennis Akpenyi <dennisakpe...@gmail.com> > > wrote: > > > > > +1 (non-binding) > > > > > > On Thu, Oct 19, 2023 at 12:24 PM Bolke de Bruin <bdbr...@gmail.com> > > wrote: > > > > > > > Dear Community, > > > > > > > > I would like to start a vote for "AIP-58 Add Airflow ObjectStore". > > > > > > > > You can find the AIP here: > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263430565 > > > > > > > > Implementing PR (most of the discussion happened here): > > > > https://github.com/apache/airflow/pull/34729 > > > > > > > > Discussion Thread (not much has happened here :-) ): > > > > Note: the title has changed from its original. > > > > > > > > https://lists.apache.org/thread/l3fkr0h6j2g4tlmsov14fywmj58t3mtp > > > > > > > > This is my binding +1m the vote will last until 12:00 UTC on 26th > > > October, > > > > and until at least 3 binding votes have been cast. > > > > > > > > Please vote accordingly: > > > > > > > > [ ] + 1 approve > > > > [ ] + 0 no opinion > > > > [ ] - 1 disapprove with the reason > > > > > > > > Only votes from PMC members and committers are binding, but other > > members > > > > of the community are encouraged to check the AIP and vote with > > > > "(non-binding)". > > > > > > > > Cheers > > > > Bolke > > > > -- > > > > > > > > -- > > > > Bolke de Bruin > > > > bdbr...@gmail.com > > > > > > > > > > -- -- Bolke de Bruin bdbr...@gmail.com
