dddevis opened a new issue, #25706:
URL: https://github.com/apache/airflow/issues/25706
### Official Helm Chart version
1.6.0 (latest released)
### Apache Airflow version
2.3.0
### Kubernetes Version
1.22
### Helm Chart configuration
I'm trying to set up remote logging on S3. The `config.logging` stanza
looks like this:
```yaml
logging:
remote_logging: "True"
remote_base_log_folder: "s3://my-bucket/airflow-logs"
remote_log_conn_id: "airflow-remote-logging-conn"
encrypt_s3_logs: "False"
colored_console_log: "False"
```
I have an AWS Secrets Manager secret called
`airflow/connections/airflow-remote-logging-conn`. Via Helm I have configured
Airflow to use Secrets Manager as the secrets backend. Here is my
`config.secrets`:
```yaml
secrets:
backend:
"airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend"
backend_kwargs: '{"connections_prefix": "airflow/connections",
"variables_prefix": "airflow/variables", "config_prefix": "airflow/config"}'
```
### Docker Image customisations
A minimal extension to the official Docker image:
```dockerfile
ARG AIRFLOW_VERSION=2.3.0
ARG PYTHON_VERSION=3.9
FROM apache/airflow:${AIRFLOW_VERSION}-python${PYTHON_VERSION}
COPY requirements.txt requirements.txt
COPY dags /opt/airflow/dags
USER airflow
RUN python -m pip install --user -r requirements.txt
```
where `requirements.txt` is
```
apache-airflow-providers-amazon==3.0.0
apache-airflow-providers-cncf-kubernetes==3.0.2
apache-airflow-providers-postgres==5.0.0
boto3==1.21.7
```
### What happened
Remote logging does not work. All task logs in the UI report:
```
*** Failed to verify remote log exists
s3://my-bucket/airflow_logs/dag_id=my_dag/run_id=scheduled__2022-08-10T06:00:00+00:00/task_id=my_task/attempt=1.log.
The conn_id `airflow-remote-logging-conn` isn't defined
*** Falling back to local log
*** Trying to get logs (last 100 lines) from worker pod
mytask-26fd147458d74729a4d609638f0a03db ***
[2022-08-11, 06:00:14 UTC] {dagbag.py:507} INFO - Filling up the DagBag from
/opt/airflow/dags/repo/airflow/dags/my_dag.py
[2022-08-11, 06:00:16 UTC] {task_command.py:369} INFO - Running
<TaskInstance:my_dag.my_task scheduled__2022-08-10T06:00:00+00:00 [queued]> on
host mytask-26fd147458d74729a4d609638f0a03db
```
### What you think should happen instead
Remote logging to S3 should work and the Airflow deployment should recognize
that the connection `airflow-remote-logging-conn` exists, since the Secrets
Manager secret name is `airflow/connections/airflow-remote-logging-conn`,
adhering to the specification in `config.secrets.backend_kwargs`.
I have essentially the same setup for a different Airflow instance I manage
with Helm chart v1.1.0, and I have no problems there.
### How to reproduce
1. In `values.yaml` for the v1.6.0 Helm chart, change `config.logging` and
`config.secrets` as above.
2. Change `workers.serviceAccount` along these lines:
```yaml
serviceAccount:
create: true
name: "airflow-worker"
annotations:
eks.amazonaws.com/role-arn:
"arn:aws:iam::123456789:role/airflow-worker"
```
where the IRSA `airflow-worker` has these policies attached:
```json
{
"Statement": [
{
"Action": [
"s3:ListBucket",
"s3:*Object*"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::my-bucket/*",
"arn:aws:s3:::my-bucket"
],
"Sid": "MyBucketReadWritePolicyDocument"
}
],
"Version": "2012-10-17"
}
```
and
```json
{
"Statement": [
{
"Action": [
"secretsmanager:ListSecretVersionIds",
"secretsmanager:GetSecretValue",
"secretsmanager:GerResourcePolicy",
"secretsmanager:DescribeSecret"
],
"Effect": "Allow",
"Resource":
"arn:aws:secretsmanager:us-east-1.123456789:secret:airflow/*",
"Sid": "SecretsManagerAirflowReadOnlyPolicy"
}
],
"Version": "2012-10-17"
}
```
Similarly for IRSAs `airflow-scheduler` and `airflow-webserver`.
3. Create a Secrets Manager secret called
`airflow/connections/airflow-remote-logging-conn` and populate it with
`s3://my-bucket`.
4. Run the example DAG and see the error reporting that the connection
`airflow-remote-logging-conn` isn't defined.
### Anything else
This happens every time for any task with chart version 1.6.0. The same
setup with 1.1.0 works.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
