As someone who works at Lyft, we do use a version of this operator that we wrote internally, in about 5% of our DAGs (out of 2000). Flyte doesn't really have a concept of sensors, nor can it interact with other tasks in Airflow, so it primarily is useful for scheduling ml pipelines with dependencies on Airflow orchestrated tables. It would be useful to us to have an officially sponsored version of this operator maintained by the Flyte org directly. *Max Payton* He/Him/His Software Engineer 202.441.7757 <+12024417757> [image: Lyft] <http://www.lyft.com/>
On Fri, Apr 1, 2022 at 12:23 AM Samhita Alla <[email protected]> wrote: > Hello, > > I work on an open-source project called Flyte > <https://github.com/flyteorg/flyte>, which is a container-native, > structured programming and distributed processing platform that enables > highly concurrent, scalable, and maintainable workflows for machine > learning and data processing pipelines. > > As a more significant chunk of the users who are into *pipelines* are > using Airflow, we've been thinking about building a provider that bridges > the gap between Airflow and Flyte, to help the Airflow users retain their > existing pipelines for ETL and use Flyte from within the Airflow DAGs to > run machine learning jobs (say). At Lyft, where Airflow and Flyte are used > together, they extensively use this operator to enable Airflow DAGs interop > with Flyte. Lately, our users have also been requesting for this feature. > > We've had this operator in the back of our minds for a long time; here's > the issue <https://github.com/flyteorg/flyte/issues/544>. The Flyte team > would like to know the community's thoughts on this provider. > Many thanks, > Samhita > >
