dispensable commented on issue #59602:
URL: https://github.com/apache/airflow/issues/59602#issuecomment-3803728780
The airflow api server requests log from airflow worker with worker podname
(which can not be solved in airflow-worker deployment mode). so we should
change the worker deploy to statefulset.
but statefuleset will mount a pvc for logs persistent which is useless when
s3 remote logging enabled.
so all we need are:
1. enable statefulset for worker deployment
2. disable PVC generated by persistent value config
3. enable loggroomer sidecar to clean logs periodic
by use helm `--post-renderer` script with kustomize and helmfiles i find a
working around:
for 2
```yaml
jsonPatches:
- target:
group: apps
version: v1
kind: StatefulSet
name: <name>-worker
patch:
- op: remove
path: /spec/volumeClaimTemplates
```
and in chart values.yaml
for 1 and 3
```yaml
workers:
persistence:
# Enable persistent volumes
enabled: true
logGroomerSidecar:
enabled: true
retentionDays: 1
frequencyMinutes: 60
extraVolumes:
- name: logs
emptyDir: {}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
