jason810496 commented on PR #65958: URL: https://github.com/apache/airflow/pull/65958#issuecomment-4532817766
I have been thinking about the schema versioning flow across https://github.com/apache/airflow/pull/67235 and this PR. Each language SDK/runtime version should already know which supervisor schema version it supports. > Would it be simpler if the SDK/runtime reports or negotiates that with the coordinator during startup, instead of requiring the version to be present in bundle metadata such as a manifest/comment/footer? Yes, it's a valid option if we only support task execution for multi-lang. However, if we consider the upcoming dag processing feature (we will raise the AIP for sure), it would be better to discover the valid bundles via file metadata to avoid overhead for trying to start each bundle as a subprocess and see which one return a valid result. > That would keep schema versioning as an SDK/runtime concern. Users would only choose the SDK version they depend on, and would not need to understand or maintain the supervisor schema version separately. From my perspective, users will not know "supervisor schema version" at all. For each Lang-SDK, we will eventually provide a build-tool (Gradle extension for Java-SDK like https://github.com/apache/airflow/issues/66328, `airflow-go-pack` for Go-SDK -- https://github.com/apache/airflow/pull/67156) to pack the "supervisor schema version" into the bundle (it's a internal spec between supervisor and Lang-SDK). The Lang-SDK specific build tool is the only thing that expose to users. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
