bolkedebruin commented on issue #5254: [AIRFLOW-4473] Add papermill operator URL: https://github.com/apache/airflow/pull/5254#issuecomment-490247174 @dimberman indeed. We intend to run Kubeflow and Airflow side by side as they fit different needs. @feng-tao it’s not too hard actually. We need to come up with a Metadata model that fits Airflow, but also can interface with the outside world in order to have end2end lineage. Ie. Spark generates its own lineage (for Atlas), for us it’s important to be able to connect what Airflow does with Spark. In the Neo4J case you don’t have the lineage that Spark (or Hive for that matter) provides so Airflow needs to fill that gap. From that point of view your integration, at the moment, is actually *simpler* cause you don’t have that interfacing requirement, yet.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
