jason810496 opened a new issue, #67599: URL: https://github.com/apache/airflow/issues/67599
### Background Python's ``airflow.models.variable.Variable.get`` and ``airflow.models.connection.Connection.get`` register sensitive values with a ``SecretsMasker`` so any subsequent task log line that contains the secret is automatically redacted. The Go SDK currently does not — secrets retrieved via ``GetVariable`` / ``GetConnection`` flow straight into task logs unredacted. The gap is documented as four coordinated TODOs across the SDK and the runtime. None of them is useful in isolation: log redaction needs both a producer (the lookup site registering the secret) and a consumer (the log handler running the redaction). Until all four land, the SDK ships without log-side masking parity. HTTP-backed client (already on ``main``): ```go // go-sdk/sdk/client.go - GetVariable // TODO: register secret-named variables with a SecretsMasker so the // returned value is automatically redacted from subsequent task logs, // matching Python's airflow.models.variable.Variable.get behaviour. // Pairs with the "TODO: mask secrets here" hook in // pkg/worker/runner.go's task log handler — both halves are needed // before secret masking actually works end-to-end. ``` ```go // go-sdk/sdk/client.go - GetConnection // TODO: register Connection.Password and sensitive-keyed entries of // Connection.Extra with a SecretsMasker so they are auto-redacted from // subsequent task logs, matching Python's // airflow.models.connection.Connection.get behaviour. Pairs with the // "TODO: mask secrets here" hook in pkg/worker/runner.go's task log // handler and the matching TODO on GetVariable above. ``` Coordinator-mode client (lands with #67317): ```go // go-sdk/pkg/execution/client.go - GetVariable and GetConnection // (same TODO text, same SecretsMasker hook) ``` Log-handler hook (lands with #67318): the ``// TODO: mask secrets here`` comment in ``go-sdk/pkg/worker/runner.go`` is where the consumer side of the redaction belongs. ### What needs to happen 1. Design and add a ``SecretsMasker`` type to the Go SDK with a register-and-redact API roughly equivalent to ``airflow.utils.log.secrets_masker.SecretsMasker`` (case-insensitive secret-name detection, replace-with-``***`` semantics, per-task scoping). 2. Register secret-named variables in both ``sdk.client.GetVariable`` and ``execution.CoordinatorClient.GetVariable``. 3. Register ``Connection.Password`` and sensitive-keyed entries of ``Connection.Extra`` in both ``sdk.client.GetConnection`` and ``execution.CoordinatorClient.GetConnection``. 4. Wire the masker into the task log handler at the ``// TODO: mask secrets here`` site in ``pkg/worker/runner.go`` so emitted log lines are redacted before they leave the runtime. 5. Remove the four coordinated TODOs above. ### Acceptance criteria - Looking up a variable named e.g. ``my_password`` via either ``sdk.client`` or ``CoordinatorClient`` causes any subsequent task log line containing the variable's value to be redacted to ``***`` (or the project's chosen sentinel). - ``Connection.Password`` and sensitive-keyed ``Connection.Extra`` entries produce the same redaction behaviour from both clients. - Tests cover: (a) registration from each client, (b) redaction in the log handler, (c) negative case where a non-sensitive variable name is not registered. - All four TODOs cross-referencing this work are removed from the codebase. ### Dependency Wait until both #67317 and #67318 merge. #67317 lands the coordinator-side ``GetVariable`` / ``GetConnection`` TODO sites; #67318 lands the ``pkg/worker/runner.go`` task log handler that the masker plugs into. Until both are on ``main``, the consumer-side hook this feature needs does not exist. ### Context - Originating review: self-review thread on #67317 (coordinator comms + client PR). - HTTP-side TODOs (already on ``main``): ``go-sdk/sdk/client.go`` ``GetVariable`` and ``GetConnection``. - Coordinator-side TODOs (land with #67317): ``go-sdk/pkg/execution/client.go`` ``GetVariable`` and ``GetConnection``. - Log-handler hook (lands with #67318): ``go-sdk/pkg/worker/runner.go``. - Python parity: ``airflow.utils.log.secrets_masker.SecretsMasker``, ``airflow.models.variable.Variable.get``, ``airflow.models.connection.Connection.get``. - Related project board: https://github.com/orgs/apache/projects/499/views/1 --- Drafted-by: Claude Code (Opus 4.7) (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
