+1 -- I agree with Max & Rafal that this should be a provider living in a separate repo and maintained by Flyte (if possible) so you can quickly and easily make changes when you update your APIs.
Regards, Kaxil On Sun, 10 Apr 2022 at 13:04, Rafal Biegacz <[email protected]> wrote: > Samhita & Max, > > Maybe a good starting point would be to offer this provider in the form of > an installable PYPI module, similar to what is being done in case of "Great > Expectation" > provider (https://pypi.org/project/airflow-provider-great-expectations/) ? > > Regards, Rafal. > > On Wed, Apr 6, 2022 at 8:31 PM Max Payton <[email protected]> > wrote: > >> As someone who works at Lyft, we do use a version of this operator that >> we wrote internally, in about 5% of our DAGs (out of 2000). Flyte doesn't >> really have a concept of sensors, nor can it interact with other tasks in >> Airflow, so it primarily is useful for scheduling ml pipelines with >> dependencies on Airflow orchestrated tables. It would be useful to us to >> have an officially sponsored version of this operator maintained by the >> Flyte org directly. >> *Max Payton* >> He/Him/His >> Software Engineer >> 202.441.7757 <+12024417757> >> [image: Lyft] <http://www.lyft.com/> >> >> >> On Fri, Apr 1, 2022 at 12:23 AM Samhita Alla <[email protected]> wrote: >> >>> Hello, >>> >>> I work on an open-source project called Flyte >>> <https://github.com/flyteorg/flyte>, which is a container-native, >>> structured programming and distributed processing platform that enables >>> highly concurrent, scalable, and maintainable workflows for machine >>> learning and data processing pipelines. >>> >>> As a more significant chunk of the users who are into *pipelines* are >>> using Airflow, we've been thinking about building a provider that bridges >>> the gap between Airflow and Flyte, to help the Airflow users retain their >>> existing pipelines for ETL and use Flyte from within the Airflow DAGs to >>> run machine learning jobs (say). At Lyft, where Airflow and Flyte are used >>> together, they extensively use this operator to enable Airflow DAGs interop >>> with Flyte. Lately, our users have also been requesting for this feature. >>> >>> We've had this operator in the back of our minds for a long time; here's >>> the issue <https://github.com/flyteorg/flyte/issues/544>. The Flyte >>> team would like to know the community's thoughts on this provider. >>> Many thanks, >>> Samhita >>> >>>
