Taragolis commented on code in PR #36449: URL: https://github.com/apache/airflow/pull/36449#discussion_r1459030435
########## airflow/providers/yandex/secrets/secrets_manager.py: ########## @@ -0,0 +1,280 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Objects relating to sourcing secrets from Yandex Cloud Lockbox.""" +from __future__ import annotations + +import logging +from functools import cached_property +from typing import Any + +import yandex.cloud.lockbox.v1.payload_pb2 as payload_pb +import yandex.cloud.lockbox.v1.payload_service_pb2 as payload_service_pb +import yandex.cloud.lockbox.v1.payload_service_pb2_grpc as payload_service_pb_grpc +import yandex.cloud.lockbox.v1.secret_pb2 as secret_pb +import yandex.cloud.lockbox.v1.secret_service_pb2 as secret_service_pb +import yandex.cloud.lockbox.v1.secret_service_pb2_grpc as secret_service_pb_grpc +import yandexcloud + +from airflow.models import Connection +from airflow.providers.yandex.utils.credentials import get_credentials +from airflow.providers.yandex.utils.defaults import default_conn_name +from airflow.providers.yandex.utils.fields import get_field_from_extras +from airflow.providers.yandex.utils.user_agent import provider_user_agent +from airflow.secrets import BaseSecretsBackend + + +class LockboxSecretBackend(BaseSecretsBackend): + """ + Retrieves Connection or Variables or Configs from Yandex Lockbox. + + Configurable via ``airflow.cfg`` like so: + + .. code-block:: ini + + [secrets] + backend = airflow.providers.yandex.secrets.secrets_manager.SecretsManagerBackend + backend_kwargs = {"connections_prefix": "airflow/connections"} + + For example, when ``{"connections_prefix": "airflow/connections"}`` is set, if a secret is defined with + the path ``airflow/connections/smtp_default``, the connection with conn_id ``smtp_default`` would be + accessible. + + When ``{"variables_prefix": "airflow/variables"}`` is set, if a secret is defined with + the path ``airflow/variables/hello``, the variable with the name ``hello`` would be accessible. + + When ``{"config_prefix": "airflow/config"}`` is set, if a secret is defined with + the path ``airflow/config/sql_alchemy_conn``, the config with key ``sql_alchemy_conn`` would be + accessible. + + When the prefix is empty, keys will use the Lockbox Secrets without any prefix. + + .. code-block:: ini + + [secrets] + backend = airflow.providers.yandex.secrets.secrets_manager.SecretsManagerBackend + backend_kwargs = {"yc_connection_id": "<connection_ID>", "folder_id": "<folder_ID>"} + + You need to specify credentials or id of yandexcloud connection to connect to Yandex Lockbox with. + Credentials will be used with this priority: + + * OAuth Token + * Service Account JSON file + * Service Account JSON + * Yandex Cloud Connection + + If no credentials specified, default connection id will be used. + + Also, you need to specify the Yandex Cloud folder ID to search for Yandex Lockbox secrets in. + + :param yc_oauth_token: Specifies the user account OAuth token to connect to Yandex Lockbox with. + Looks like ``y3_xxxxx``. + :param yc_sa_key_json: Specifies the service account auth JSON. + Looks like ``{"id": "...", "service_account_id": "...", "private_key": "..."}``. + :param yc_sa_key_json_path: Specifies the service account auth JSON file path. + Looks like ``/home/airflow/authorized_key.json``. + File content looks like ``{"id": "...", "service_account_id": "...", "private_key": "..."}``. + :param yc_connection_id: Specifies the connection ID to connect to Yandex Lockbox with. + Default: "yandexcloud_default" + :param folder_id: Specifies the folder ID to search for Yandex Lockbox secrets in. + If set to None (null in JSON), requests will use the connection folder_id if specified. + :param connections_prefix: Specifies the prefix of the secret to read to get Connections. + If set to None (null in JSON), requests for connections will not be sent to Yandex Lockbox. + Default: "airflow/connections" + :param variables_prefix: Specifies the prefix of the secret to read to get Variables. + If set to None (null in JSON), requests for variables will not be sent to Yandex Lockbox. + Default: "airflow/variables" + :param config_prefix: Specifies the prefix of the secret to read to get Configurations. + If set to None (null in JSON), requests for variables will not be sent to Yandex Lockbox. + Default: "airflow/config" + :param sep: Specifies the separator used to concatenate secret_prefix and secret_id. + Default: "/" + :param endpoint: Specifies an API endpoint. + Leave blank to use default. + """ + + def __init__( + self, + yc_oauth_token: str | None = None, + yc_sa_key_json: str | None = None, + yc_sa_key_json_path: str | None = None, + yc_connection_id: str | None = None, Review Comment: > This problem is addressed in this PR That might be not such as straight forward. Secrets Manager has particular issue not only directly with connection but with other components and it is very difficult to detect in regular unit tests because during unit tests everything already initialised, however in regular usage it is not. Something what we have in the past with other Secrets Backends: - Try to obtain Connection to the DB from the secrets backend was raised circular import error: https://github.com/apache/airflow/pull/26784/files - Try to obtain Fernet key from the secrets backend was raised circular import error. This past issues not directly related to this PR, but still worthwhile to check it with this provider, I guess one of this might produce some interesting errors, or not. >I guess Deployment Manager can just choose to not use Connection in SecretManager configuration if that connection metadata is not trustworthy enough. Yeah, I've checked that couple other components also use conn_id, e.g. remote logging mechanism. So maybe this one not so critical than I thought initially > On the other hand, ability to not duplicate creds between SecretsManager and Hook looks profitable for simple installations. This is good idea to have some wrapper class / sets of function to convert provided parameters to the valid parameters for the Hook and SecretsManager, the same approach use into the Amazon Provider and I guess into the Google provider. My point here more about whether or not is a good idea to use Connection ID in Secrets Manager. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
