+1 from me as well. Indeed LLMs for Apache Airflow is a no brainer, this will open a lot of opportunities for our users.
Having Astronomer supporting this initiative is great. I like the idea of system tests dashboard, especially for those that are not open source/easily testable without a paid account. Also, now that we have the ability to suspend providers we can still do that in case we change our mind in the future. Best regards, Pierre Le ven. 20 oct. 2023 à 08:35, Elad Kalif <elad...@apache.org> a écrit : > +1 from me > I think the dashboard idea is great! > > On Thu, Oct 19, 2023 at 7:05 PM Andrey Anshin <andrey.ans...@taragol.is> > wrote: > > > Because 4 out 5 new providers have a draft PR I would like to raise a > > question about which related to all new providers. Just to avoid the > same > > question in all PRs. > > > > Do we actually want to make new operators kindish of like > "PythonOperator"? > > Maybe I miss some important thing and can't see why it would work better > > rather than run hooks methods inside of PythonOperator / TaskFlow? > > > > For the reference Reference: > > Add Cohere Provider: > > https://github.com/apache/airflow/pull/34921#discussion_r1358525838 > > Enable pgvector support for Postgres provider: > > https://github.com/apache/airflow/pull/34891#discussion_r1362910782 > > Add OpenAI Provider: > > https://github.com/apache/airflow/pull/35023#discussion_r1365235167 > > Add Weaviate Provider: > > https://github.com/apache/airflow/pull/35060/files#r1365765741 > > > > ---- > > Best Wishes > > *Andrey Anshin* > > > > > > > > On Tue, 17 Oct 2023 at 22:42, Kaxil Naik <kaxiln...@apache.org> wrote: > > > > > Hey Everyone, > > > > > > As a follow-up to my Keynote talk, Building and deploying LLM > > applications > > > with Apache Airflow <https://www.youtube.com/watch?v=mgA6m3ggKhs&t=4s > >, > > I > > > am formally proposing the addition of these 5 providers to the Apache > > > Airflow repo: > > > > > > - > > > > > > PgVector <https://github.com/pgvector/pgvector> > > > - > > > > > > Weaviate <https://weaviate.io/> > > > - > > > > > > Pinecone <https://www.pinecone.io/> > > > - > > > > > > OpenAI <https://openai.com/> > > > - > > > > > > Cohere <https://cohere.com/> > > > > > > > > > Advancements in LLMs are moving at a rapid pace & transforming the way > we > > > work and our industry. Although LLMs are simple to use in prototyping, > > > using LLM for enterprise applications and for production still > presents a > > > lot of challenges. These > > > < > > > > > > https://speakerdeck.com/kaxil/building-and-deploying-llm-applications-with-apache-airflow?slide=8 > > > > > > > are some of the same problems that we tackle in Data Engineering, and > > > Airflow is a natural fit for them. > > > > > > We at Astronomer would like to add first-class support for the popular > > LLMs > > > (OpenAI & Cohere) and vector DBs (PgVector, Weaviate & Pinecone) so > that > > > Data Scientists and ML engineers can utilize them natively with > > easy-to-use > > > Operator & Hook abstractions while providing a native (and > > > Production-ready) approach for Authentication, retries, logging etc. > > > > > > We also think this is vital for the Apache Airflow project as we, the > > > project, embrace the LLM tide and continue to be a great example of > > > balancing innovation and maintaining backward-compatibility. > > > > > > The first versions of these providers will enable building one of the > > most > > > common use cases of LLMs i.e. Question and Answering / Chatbots using > > > Retrieval-augmented generation (RAG) done with the help of embeddings. > > > > > > Everyone is welcome and encouraged to contribute once the PRs are > merged. > > > Astronomer is committed to maintaining these providers in the Airflow > > repo, > > > including reviewing PRs, maintaining code quality, testing and keeping > > the > > > APIs up-to-date. > > > > > > Note: PgVector <https://github.com/pgvector/pgvector> is an > open-source > > > project, so we don’t need a formal vote for it as per our guidelines > > > < > > > > > > https://github.com/apache/airflow/blob/main/PROVIDERS.rst#accepting-new-community-providers > > > >. > > > So please consider this email as seeking a Lazy Consensus for it. > > > > > > I will open up a VOTING thread after discussing this for a few days. > > > > > > Thanks. > > > > > > Regards, > > > > > > Kaxil > > > > > >
