ruoyu90 commented on issue #9321: URL: https://github.com/apache/airflow/issues/9321#issuecomment-646884033
(Thanks @casassg for pointing us here) Hello from [TFX](https://github.com/tensorflow/tfx/) community! It is excited to see this proposal for Airflow as we are also [proposing](https://github.com/tensorflow/community/pull/253) some pipeline patterns that seem to align well with the vision here. Some context: TFX is designed to be a portable ML production platform and we already have a [demo implementation](https://github.com/tensorflow/tfx/blob/master/tfx/examples/chicago_taxi_pipeline/taxi_pipeline_simple.py) on top of Airflow stack for traditional workflow-based pipelines. Our new [proposal](https://github.com/tensorflow/community/pull/253) is trying to define a more advanced semantics for asynchronous execution pipelines where every nodes in a pipeline is loosely connected through data signals. The existing Airflow implementation is not good at dealing with such pattern but we are excited to see the proposal in this thread which makes it feasible to realize such pattern on Airflow. How about we get in touch and see whether we can have another successful story for TFX on Airflow? /cc @zhitaoli /cc @rcrowe-google /cc @theadactyl ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
