davidzollo commented on issue #9178: URL: https://github.com/apache/seatunnel/issues/9178#issuecomment-5391653714
> [@davidzollo](https://github.com/davidzollo) [@DanielLeens](https://github.com/DanielLeens) [@Hisoka-X](https://github.com/Hisoka-X) I reviewed the previous work in [apache/airflow#50226](https://github.com/apache/airflow/pull/50226) and checked the current Airflow process for accepting new providers > > The Airflow process has changed since that PR was opened. A new provider now starts in incubation and requires > > * a working implementation or design PoC > * at least two named provider stewards > * an Airflow committer willing to sponsor the provider > * a `[DISCUSS]` thread on the Airflow dev mailing list > * no objections for at least 7 days before proceeding with the implementation PR > * unit tests documentation and a system test plan > * ongoing maintenance and quarterly provider health updates > > The current process is documented here https://github.com/apache/airflow/blob/main/providers/ACCEPTING_PROVIDERS.rst > > I do not think we should reopen the old implementation as-is because the SeaTunnel and Airflow APIs have moved since then. I suggest using it only as a reference and rebuilding this in focused slices > > ### Slice 1: Zeta REST integration > * Airflow connection and `SeaTunnelHook` > * submit jobs through `/submit-job` > * query status through `/job-info/{jobId}` > * stop jobs through `/stop-job` > * support HOCON JSON and SQL configurations > * return the SeaTunnel job ID to Airflow > > ### Slice 2: Airflow execution lifecycle > * `SeaTunnelOperator` > * deferrable trigger so long-running jobs do not occupy an Airflow worker > * reusable job sensor > * proper terminal-state and failure handling > * cancel the SeaTunnel job when the Airflow task is killed or cancelled > > ### Slice 3: packaging and verification > * provider metadata and connection documentation > * example DAGs > * focused unit tests for submit status failure and cancellation > * Docker-based Zeta system test > * provider build and Airflow CI integration > > ### Slice 4: CLI and other engines > * separate CLI operator for Zeta Flink and Spark > * use the current engine-specific commands and deployment options > * document that SeaTunnel must be installed on the Airflow worker > * keep this separate from Zeta REST because Flink and Spark do not expose the SeaTunnel REST API and need their own submission and monitoring paths > > Before starting I would like to confirm > > 1. Is this still an active SeaTunnel ecosystem priority > 2. Is [@ocean-zhc](https://github.com/ocean-zhc) planning to continue or interested in collaborating? or i can take this up and take help as required. > 3. Does the community agree with Zeta REST as the first provider slice > 4. Can someone join as the second steward > 5. Do we have an Airflow committer who may be willing to sponsor the provider > > If this direction looks reasonable I can prepare the technical design and working Zeta REST PoC for the Airflow mailing-list discussion @goutamadwant Don't proceed with this integration now. I'll close it first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
