Piatachock commented on code in PR #36449: URL: https://github.com/apache/airflow/pull/36449#discussion_r1457025440
########## airflow/providers/yandex/secrets/secrets_manager.py: ########## @@ -0,0 +1,280 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Objects relating to sourcing secrets from Yandex Cloud Lockbox.""" +from __future__ import annotations + +import logging +from functools import cached_property +from typing import Any + +import yandex.cloud.lockbox.v1.payload_pb2 as payload_pb +import yandex.cloud.lockbox.v1.payload_service_pb2 as payload_service_pb +import yandex.cloud.lockbox.v1.payload_service_pb2_grpc as payload_service_pb_grpc +import yandex.cloud.lockbox.v1.secret_pb2 as secret_pb +import yandex.cloud.lockbox.v1.secret_service_pb2 as secret_service_pb +import yandex.cloud.lockbox.v1.secret_service_pb2_grpc as secret_service_pb_grpc +import yandexcloud + +from airflow.models import Connection +from airflow.providers.yandex.utils.credentials import get_credentials +from airflow.providers.yandex.utils.defaults import default_conn_name +from airflow.providers.yandex.utils.fields import get_field_from_extras +from airflow.providers.yandex.utils.user_agent import provider_user_agent +from airflow.secrets import BaseSecretsBackend + + +class LockboxSecretBackend(BaseSecretsBackend): + """ + Retrieves Connection or Variables or Configs from Yandex Lockbox. + + Configurable via ``airflow.cfg`` like so: + + .. code-block:: ini + + [secrets] + backend = airflow.providers.yandex.secrets.secrets_manager.SecretsManagerBackend + backend_kwargs = {"connections_prefix": "airflow/connections"} + + For example, when ``{"connections_prefix": "airflow/connections"}`` is set, if a secret is defined with + the path ``airflow/connections/smtp_default``, the connection with conn_id ``smtp_default`` would be + accessible. + + When ``{"variables_prefix": "airflow/variables"}`` is set, if a secret is defined with + the path ``airflow/variables/hello``, the variable with the name ``hello`` would be accessible. + + When ``{"config_prefix": "airflow/config"}`` is set, if a secret is defined with + the path ``airflow/config/sql_alchemy_conn``, the config with key ``sql_alchemy_conn`` would be + accessible. + + When the prefix is empty, keys will use the Lockbox Secrets without any prefix. + + .. code-block:: ini + + [secrets] + backend = airflow.providers.yandex.secrets.secrets_manager.SecretsManagerBackend + backend_kwargs = {"yc_connection_id": "<connection_ID>", "folder_id": "<folder_ID>"} + + You need to specify credentials or id of yandexcloud connection to connect to Yandex Lockbox with. + Credentials will be used with this priority: + + * OAuth Token + * Service Account JSON file + * Service Account JSON + * Yandex Cloud Connection + + If no credentials specified, default connection id will be used. + + Also, you need to specify the Yandex Cloud folder ID to search for Yandex Lockbox secrets in. + + :param yc_oauth_token: Specifies the user account OAuth token to connect to Yandex Lockbox with. + Looks like ``y3_xxxxx``. + :param yc_sa_key_json: Specifies the service account auth JSON. + Looks like ``{"id": "...", "service_account_id": "...", "private_key": "..."}``. + :param yc_sa_key_json_path: Specifies the service account auth JSON file path. + Looks like ``/home/airflow/authorized_key.json``. + File content looks like ``{"id": "...", "service_account_id": "...", "private_key": "..."}``. + :param yc_connection_id: Specifies the connection ID to connect to Yandex Lockbox with. + Default: "yandexcloud_default" + :param folder_id: Specifies the folder ID to search for Yandex Lockbox secrets in. + If set to None (null in JSON), requests will use the connection folder_id if specified. + :param connections_prefix: Specifies the prefix of the secret to read to get Connections. + If set to None (null in JSON), requests for connections will not be sent to Yandex Lockbox. + Default: "airflow/connections" + :param variables_prefix: Specifies the prefix of the secret to read to get Variables. + If set to None (null in JSON), requests for variables will not be sent to Yandex Lockbox. + Default: "airflow/variables" + :param config_prefix: Specifies the prefix of the secret to read to get Configurations. + If set to None (null in JSON), requests for variables will not be sent to Yandex Lockbox. + Default: "airflow/config" + :param sep: Specifies the separator used to concatenate secret_prefix and secret_id. + Default: "/" + :param endpoint: Specifies an API endpoint. + Leave blank to use default. + """ + + def __init__( + self, + yc_oauth_token: str | None = None, + yc_sa_key_json: str | None = None, + yc_sa_key_json_path: str | None = None, + yc_connection_id: str | None = None, Review Comment: My thoughts here is: 1. This problem is addressed in this PR 2. I guess Deployment Manager can just choose to not use Connection in SecretManager configuration if that connection metadata is not trustworthy enough. On the other hand, ability to not duplicate creds between SecretsManager and Hook looks profitable for simple installations. ########## airflow/providers/yandex/secrets/secrets_manager.py: ########## @@ -0,0 +1,280 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Objects relating to sourcing secrets from Yandex Cloud Lockbox.""" +from __future__ import annotations + +import logging +from functools import cached_property +from typing import Any + +import yandex.cloud.lockbox.v1.payload_pb2 as payload_pb +import yandex.cloud.lockbox.v1.payload_service_pb2 as payload_service_pb +import yandex.cloud.lockbox.v1.payload_service_pb2_grpc as payload_service_pb_grpc +import yandex.cloud.lockbox.v1.secret_pb2 as secret_pb +import yandex.cloud.lockbox.v1.secret_service_pb2 as secret_service_pb +import yandex.cloud.lockbox.v1.secret_service_pb2_grpc as secret_service_pb_grpc +import yandexcloud + +from airflow.models import Connection +from airflow.providers.yandex.utils.credentials import get_credentials +from airflow.providers.yandex.utils.defaults import default_conn_name +from airflow.providers.yandex.utils.fields import get_field_from_extras +from airflow.providers.yandex.utils.user_agent import provider_user_agent +from airflow.secrets import BaseSecretsBackend + + +class LockboxSecretBackend(BaseSecretsBackend): + """ + Retrieves Connection or Variables or Configs from Yandex Lockbox. + + Configurable via ``airflow.cfg`` like so: + + .. code-block:: ini + + [secrets] + backend = airflow.providers.yandex.secrets.secrets_manager.SecretsManagerBackend + backend_kwargs = {"connections_prefix": "airflow/connections"} + + For example, when ``{"connections_prefix": "airflow/connections"}`` is set, if a secret is defined with + the path ``airflow/connections/smtp_default``, the connection with conn_id ``smtp_default`` would be + accessible. + + When ``{"variables_prefix": "airflow/variables"}`` is set, if a secret is defined with + the path ``airflow/variables/hello``, the variable with the name ``hello`` would be accessible. + + When ``{"config_prefix": "airflow/config"}`` is set, if a secret is defined with + the path ``airflow/config/sql_alchemy_conn``, the config with key ``sql_alchemy_conn`` would be + accessible. + + When the prefix is empty, keys will use the Lockbox Secrets without any prefix. + + .. code-block:: ini + + [secrets] + backend = airflow.providers.yandex.secrets.secrets_manager.SecretsManagerBackend + backend_kwargs = {"yc_connection_id": "<connection_ID>", "folder_id": "<folder_ID>"} + + You need to specify credentials or id of yandexcloud connection to connect to Yandex Lockbox with. + Credentials will be used with this priority: + + * OAuth Token + * Service Account JSON file + * Service Account JSON + * Yandex Cloud Connection + + If no credentials specified, default connection id will be used. + + Also, you need to specify the Yandex Cloud folder ID to search for Yandex Lockbox secrets in. + + :param yc_oauth_token: Specifies the user account OAuth token to connect to Yandex Lockbox with. + Looks like ``y3_xxxxx``. + :param yc_sa_key_json: Specifies the service account auth JSON. + Looks like ``{"id": "...", "service_account_id": "...", "private_key": "..."}``. + :param yc_sa_key_json_path: Specifies the service account auth JSON file path. + Looks like ``/home/airflow/authorized_key.json``. + File content looks like ``{"id": "...", "service_account_id": "...", "private_key": "..."}``. + :param yc_connection_id: Specifies the connection ID to connect to Yandex Lockbox with. + Default: "yandexcloud_default" + :param folder_id: Specifies the folder ID to search for Yandex Lockbox secrets in. + If set to None (null in JSON), requests will use the connection folder_id if specified. + :param connections_prefix: Specifies the prefix of the secret to read to get Connections. + If set to None (null in JSON), requests for connections will not be sent to Yandex Lockbox. + Default: "airflow/connections" + :param variables_prefix: Specifies the prefix of the secret to read to get Variables. + If set to None (null in JSON), requests for variables will not be sent to Yandex Lockbox. + Default: "airflow/variables" + :param config_prefix: Specifies the prefix of the secret to read to get Configurations. + If set to None (null in JSON), requests for variables will not be sent to Yandex Lockbox. + Default: "airflow/config" + :param sep: Specifies the separator used to concatenate secret_prefix and secret_id. + Default: "/" + :param endpoint: Specifies an API endpoint. + Leave blank to use default. + """ + + def __init__( + self, + yc_oauth_token: str | None = None, + yc_sa_key_json: str | None = None, + yc_sa_key_json_path: str | None = None, + yc_connection_id: str | None = None, Review Comment: My thoughts here are: 1. This problem is addressed in this PR 2. I guess Deployment Manager can just choose to not use Connection in SecretManager configuration if that connection metadata is not trustworthy enough. On the other hand, ability to not duplicate creds between SecretsManager and Hook looks profitable for simple installations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
