Hello,

I work on an open-source project called Flyte
<https://github.com/flyteorg/flyte>, which is a container-native,
structured programming and distributed processing platform that enables
highly concurrent, scalable, and maintainable workflows for machine
learning and data processing pipelines.

As a more significant chunk of the users who are into *pipelines* are using
Airflow, we've been thinking about building a provider that bridges the gap
between Airflow and Flyte, to help the Airflow users retain their existing
pipelines for ETL and use Flyte from within the Airflow DAGs to run machine
learning jobs (say). At Lyft, where Airflow and Flyte are used together,
they extensively use this operator to enable Airflow DAGs interop with
Flyte. Lately, our users have also been requesting for this feature.

We've had this operator in the back of our minds for a long time; here's
the issue <https://github.com/flyteorg/flyte/issues/544>. The Flyte team
would like to know the community's thoughts on this provider.
Many thanks,
Samhita

Reply via email to