IvanovOleg opened a new issue, #38937:
URL: https://github.com/apache/airflow/issues/38937
### Official Helm Chart version
1.13.1 (latest released)
### Apache Airflow version
2.8.3
### Kubernetes Version
Openshift 4.14.12
### Helm Chart configuration
```yaml
scheduler:
securityContext:
fsGroup: 1000720000
runAsGroup: 1000720000
runAsUser: 1000720000
waitForMigrations:
securityContexts:
container:
runAsGroup: 0
workers:
securityContext:
fsGroup: 1000720000
runAsGroup: 1000720000
runAsUser: 1000720000
waitForMigrations:
securityContexts:
container:
runAsGroup: 0
triggerer:
securityContext:
fsGroup: 1000720000
runAsGroup: 1000720000
runAsUser: 1000720000
waitForMigrations:
securityContexts:
container:
runAsGroup: 0
webserver:
securityContext:
fsGroup: 1000720000
runAsUser: 1000720000
statsd:
securityContext:
runAsUser: 1000720000
redis:
securityContext:
runAsUser: 1000720000
runAsGroup: 1000720000
postgresql:
primary:
podSecurityContext:
enabled: false
containerSecurityContext:
enabled: false
readReplicas:
podSecurityContext:
enabled: false
containerSecurityContext:
enabled: false
```
### Docker Image customizations
_No response_
### What happened
Webserver, worker, triggerer, shceduler stuck with following init container
issue:
```
[2024-04-11T15:10:52.326+0000] {db.py:799} INFO - Waiting for migrations...
0 second(s)
[2024-04-11T15:10:53.329+0000] {db.py:799} INFO - Waiting for migrations...
1 second(s)
[2024-04-11T15:10:54.330+0000] {db.py:799} INFO - Waiting for migrations...
2 second(s)
[2024-04-11T15:10:55.334+0000] {db.py:799} INFO - Waiting for migrations...
3 second(s)
[2024-04-11T15:10:56.335+0000] {db.py:799} INFO - Waiting for migrations...
4 second(s)
[2024-04-11T15:10:57.337+0000] {db.py:799} INFO - Waiting for migrations...
5 second(s)
[2024-04-11T15:10:58.339+0000] {db.py:799} INFO - Waiting for migrations...
6 second(s)
[2024-04-11T15:10:59.341+0000] {db.py:799} INFO - Waiting for migrations...
7 second(s)
[2024-04-11T15:11:00.342+0000] {db.py:799} INFO - Waiting for migrations...
8 second(s)
[2024-04-11T15:11:01.343+0000] {db.py:799} INFO - Waiting for migrations...
9 second(s)
[2024-04-11T15:11:02.345+0000] {db.py:799} INFO - Waiting for migrations...
10 second(s)
[2024-04-11T15:11:03.347+0000] {db.py:799} INFO - Waiting for migrations...
11 second(s)
[2024-04-11T15:11:04.349+0000] {db.py:799} INFO - Waiting for migrations...
12 second(s)
[2024-04-11T15:11:05.350+0000] {db.py:799} INFO - Waiting for migrations...
13 second(s)
[2024-04-11T15:11:06.353+0000] {db.py:799} INFO - Waiting for migrations...
14 second(s)
[2024-04-11T15:11:07.355+0000] {db.py:799} INFO - Waiting for migrations...
15 second(s)
[2024-04-11T15:11:08.356+0000] {db.py:799} INFO - Waiting for migrations...
16 second(s)
[2024-04-11T15:11:09.358+0000] {db.py:799} INFO - Waiting for migrations...
17 second(s)
[2024-04-11T15:11:10.359+0000] {db.py:799} INFO - Waiting for migrations...
18 second(s)
[2024-04-11T15:11:11.361+0000] {db.py:799} INFO - Waiting for migrations...
19 second(s)
[2024-04-11T15:11:12.363+0000] {db.py:799} INFO - Waiting for migrations...
20 second(s)
[2024-04-11T15:11:13.365+0000] {db.py:799} INFO - Waiting for migrations...
21 second(s)
[2024-04-11T15:11:14.366+0000] {db.py:799} INFO - Waiting for migrations...
22 second(s)
[2024-04-11T15:11:15.368+0000] {db.py:799} INFO - Waiting for migrations...
23 second(s)
[2024-04-11T15:11:16.369+0000] {db.py:799} INFO - Waiting for migrations...
24 second(s)
[2024-04-11T15:11:17.370+0000] {db.py:799} INFO - Waiting for migrations...
25 second(s)
[2024-04-11T15:11:18.372+0000] {db.py:799} INFO - Waiting for migrations...
26 second(s)
[2024-04-11T15:11:19.373+0000] {db.py:799} INFO - Waiting for migrations...
27 second(s)
[2024-04-11T15:11:20.374+0000] {db.py:799} INFO - Waiting for migrations...
28 second(s)
[2024-04-11T15:11:21.375+0000] {db.py:799} INFO - Waiting for migrations...
29 second(s)
[2024-04-11T15:11:22.377+0000] {db.py:799} INFO - Waiting for migrations...
30 second(s)
[2024-04-11T15:11:23.378+0000] {db.py:799} INFO - Waiting for migrations...
31 second(s)
[2024-04-11T15:11:24.380+0000] {db.py:799} INFO - Waiting for migrations...
32 second(s)
[2024-04-11T15:11:25.382+0000] {db.py:799} INFO - Waiting for migrations...
33 second(s)
[2024-04-11T15:11:26.383+0000] {db.py:799} INFO - Waiting for migrations...
34 second(s)
[2024-04-11T15:11:27.384+0000] {db.py:799} INFO - Waiting for migrations...
35 second(s)
[2024-04-11T15:11:28.385+0000] {db.py:799} INFO - Waiting for migrations...
36 second(s)
[2024-04-11T15:11:29.386+0000] {db.py:799} INFO - Waiting for migrations...
37 second(s)
[2024-04-11T15:11:30.400+0000] {db.py:799} INFO - Waiting for migrations...
38 second(s)
[2024-04-11T15:11:31.422+0000] {db.py:799} INFO - Waiting for migrations...
39 second(s)
[2024-04-11T15:11:32.424+0000] {db.py:799} INFO - Waiting for migrations...
40 second(s)
[2024-04-11T15:11:33.426+0000] {db.py:799} INFO - Waiting for migrations...
41 second(s)
[2024-04-11T15:11:34.427+0000] {db.py:799} INFO - Waiting for migrations...
42 second(s)
[2024-04-11T15:11:35.429+0000] {db.py:799} INFO - Waiting for migrations...
43 second(s)
[2024-04-11T15:11:36.430+0000] {db.py:799} INFO - Waiting for migrations...
44 second(s)
[2024-04-11T15:11:37.431+0000] {db.py:799} INFO - Waiting for migrations...
45 second(s)
[2024-04-11T15:11:38.434+0000] {db.py:799} INFO - Waiting for migrations...
46 second(s)
[2024-04-11T15:11:39.436+0000] {db.py:799} INFO - Waiting for migrations...
47 second(s)
[2024-04-11T15:11:40.438+0000] {db.py:799} INFO - Waiting for migrations...
48 second(s)
[2024-04-11T15:11:41.439+0000] {db.py:799} INFO - Waiting for migrations...
49 second(s)
[2024-04-11T15:11:42.441+0000] {db.py:799} INFO - Waiting for migrations...
50 second(s)
[2024-04-11T15:11:43.443+0000] {db.py:799} INFO - Waiting for migrations...
51 second(s)
[2024-04-11T15:11:44.444+0000] {db.py:799} INFO - Waiting for migrations...
52 second(s)
[2024-04-11T15:11:45.446+0000] {db.py:799} INFO - Waiting for migrations...
53 second(s)
[2024-04-11T15:11:46.448+0000] {db.py:799} INFO - Waiting for migrations...
54 second(s)
[2024-04-11T15:11:47.449+0000] {db.py:799} INFO - Waiting for migrations...
55 second(s)
[2024-04-11T15:11:48.471+0000] {db.py:799} INFO - Waiting for migrations...
56 second(s)
[2024-04-11T15:11:49.473+0000] {db.py:799} INFO - Waiting for migrations...
57 second(s)
[2024-04-11T15:11:50.475+0000] {db.py:799} INFO - Waiting for migrations...
58 second(s)
[2024-04-11T15:11:51.476+0000] {db.py:799} INFO - Waiting for migrations...
59 second(s)
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line
57, in main
args.func(args)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_config.py",
line 49, in command
return func(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/providers_configuration_loader.py",
line 55, in wrapped_function
return func(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py",
line 190, in check_migrations
db.check_migrations(timeout=args.migration_wait_timeout)
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/db.py", line
800, in check_migrations
raise TimeoutError(
TimeoutError: There are still unapplied migrations after 60 seconds.
MigrationHead(s) in DB: set() | Migration Head(s) in Source Code:
{'88344c1d9134'}
```
Waiting doesn't help. In postgres I see following errors:
```
postgresql 15:08:12.02 INFO ==>
postgresql 15:08:12.03 INFO ==> Welcome to the Bitnami postgresql container
postgresql 15:08:12.03 INFO ==> Subscribe to project updates by watching
https://github.com/bitnami/containers
postgresql 15:08:12.04 INFO ==> Submit issues and feature requests at
https://github.com/bitnami/containers/issues
postgresql 15:08:12.04 INFO ==>
postgresql 15:08:12.05 INFO ==> ** Starting PostgreSQL setup **
postgresql 15:08:12.07 INFO ==> Validating settings in POSTGRESQL_* env
vars..
postgresql 15:08:12.08 INFO ==> Loading custom pre-init scripts...
postgresql 15:08:12.08 INFO ==> Initializing PostgreSQL database...
postgresql 15:08:12.10 INFO ==> pg_hba.conf file not detected. Generating
it...
postgresql 15:08:12.10 INFO ==> Generating local authentication
configuration
postgresql 15:08:13.32 INFO ==> Starting PostgreSQL in background...
postgresql 15:08:13.44 INFO ==> Changing password of postgres
postgresql 15:08:13.47 INFO ==> Configuring replication parameters
postgresql 15:08:13.49 INFO ==> Configuring synchronous_replication
postgresql 15:08:13.49 INFO ==> Configuring fsync
postgresql 15:08:13.53 INFO ==> Stopping PostgreSQL...
waiting for server to shut down.... done
server stopped
postgresql 15:08:13.63 INFO ==> Loading custom scripts...
postgresql 15:08:13.64 INFO ==> Enabling remote connections
postgresql 15:08:13.64 INFO ==> ** PostgreSQL setup finished! **
postgresql 15:08:13.65 INFO ==> ** Starting PostgreSQL **
2024-04-11 15:08:13.675 GMT [1] LOG: pgaudit extension initialized
2024-04-11 15:08:13.686 GMT [1] LOG: starting PostgreSQL 16.1 on
x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2024-04-11 15:08:13.686 GMT [1] LOG: listening on IPv4 address "0.0.0.0",
port 5432
2024-04-11 15:08:13.686 GMT [1] LOG: listening on IPv6 address "::", port
5432
2024-04-11 15:08:13.694 GMT [1] LOG: listening on Unix socket
"/tmp/.s.PGSQL.5432"
2024-04-11 15:08:13.707 GMT [123] LOG: database system was shut down at
2024-04-11 15:08:13 GMT
2024-04-11 15:08:13.722 GMT [1] LOG: database system is ready to accept
connections
2024-04-11 15:08:21.948 GMT [134] ERROR: relation "log" does not exist at
character 13
2024-04-11 15:08:21.948 GMT [134] STATEMENT: INSERT INTO log (dttm, event,
owner, extra) VALUES ('2024-04-11T15:08:14.855577+00:00'::timestamptz,
'cli_check', 'default', '{"host_name": "airflow-webserver-76c8cb49f-ms2gr",
"full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}')
RETURNING log.id
2024-04-11 15:08:22.713 GMT [135] ERROR: relation "log" does not exist at
character 13
2024-04-11 15:08:22.713 GMT [135] STATEMENT: INSERT INTO log (dttm, event,
owner, extra) VALUES ('2024-04-11T15:08:15.605933+00:00'::timestamptz,
'cli_check', 'default', '{"host_name": "airflow-scheduler-58c4b49fc-r98hh",
"full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}')
RETURNING log.id
2024-04-11 15:08:22.905 GMT [136] ERROR: relation "log" does not exist at
character 13
2024-04-11 15:08:22.905 GMT [136] STATEMENT: INSERT INTO log (dttm, event,
owner, extra) VALUES ('2024-04-11T15:08:15.815946+00:00'::timestamptz,
'cli_check', 'default', '{"host_name": "airflow-worker-0", "full_command":
"[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:08:23.033 GMT [137] ERROR: relation "log" does not exist at
character 13
2024-04-11 15:08:23.033 GMT [137] STATEMENT: INSERT INTO log (dttm, event,
owner, extra) VALUES ('2024-04-11T15:08:15.928602+00:00'::timestamptz,
'cli_check', 'default', '{"host_name": "airflow-triggerer-0", "full_command":
"[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:09:30.672 GMT [229] ERROR: relation "log" does not exist at
character 13
2024-04-11 15:09:30.672 GMT [229] STATEMENT: INSERT INTO log (dttm, event,
owner, extra) VALUES ('2024-04-11T15:09:30.657757+00:00'::timestamptz,
'cli_check', 'default', '{"host_name": "airflow-scheduler-58c4b49fc-r98hh",
"full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}')
RETURNING log.id
2024-04-11 15:09:30.684 GMT [230] ERROR: relation "log" does not exist at
character 13
2024-04-11 15:09:30.684 GMT [230] STATEMENT: INSERT INTO log (dttm, event,
owner, extra) VALUES ('2024-04-11T15:09:30.657674+00:00'::timestamptz,
'cli_check', 'default', '{"host_name": "airflow-webserver-76c8cb49f-ms2gr",
"full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}')
RETURNING log.id
2024-04-11 15:09:31.501 GMT [231] ERROR: relation "log" does not exist at
character 13
2024-04-11 15:09:31.501 GMT [231] STATEMENT: INSERT INTO log (dttm, event,
owner, extra) VALUES ('2024-04-11T15:09:31.492209+00:00'::timestamptz,
'cli_check', 'default', '{"host_name": "airflow-worker-0", "full_command":
"[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:09:31.543 GMT [232] ERROR: relation "log" does not exist at
character 13
2024-04-11 15:09:31.543 GMT [232] STATEMENT: INSERT INTO log (dttm, event,
owner, extra) VALUES ('2024-04-11T15:09:31.533532+00:00'::timestamptz,
'cli_check', 'default', '{"host_name": "airflow-triggerer-0", "full_command":
"[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:10:49.517 GMT [338] ERROR: relation "log" does not exist at
character 13
2024-04-11 15:10:49.517 GMT [338] STATEMENT: INSERT INTO log (dttm, event,
owner, extra) VALUES ('2024-04-11T15:10:49.507573+00:00'::timestamptz,
'cli_check', 'default', '{"host_name": "airflow-scheduler-58c4b49fc-r98hh",
"full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}')
RETURNING log.id
2024-04-11 15:10:50.612 GMT [353] ERROR: relation "log" does not exist at
character 13
2024-04-11 15:10:50.612 GMT [353] STATEMENT: INSERT INTO log (dttm, event,
owner, extra) VALUES ('2024-04-11T15:10:50.602994+00:00'::timestamptz,
'cli_check', 'default', '{"host_name": "airflow-worker-0", "full_command":
"[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:10:51.768 GMT [355] ERROR: relation "log" does not exist at
character 13
2024-04-11 15:10:51.768 GMT [355] STATEMENT: INSERT INTO log (dttm, event,
owner, extra) VALUES ('2024-04-11T15:10:51.760706+00:00'::timestamptz,
'cli_check', 'default', '{"host_name": "airflow-webserver-76c8cb49f-ms2gr",
"full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}')
RETURNING log.id
2024-04-11 15:10:52.789 GMT [357] ERROR: relation "log" does not exist at
character 13
2024-04-11 15:10:52.789 GMT [357] STATEMENT: INSERT INTO log (dttm, event,
owner, extra) VALUES ('2024-04-11T15:10:52.779490+00:00'::timestamptz,
'cli_check', 'default', '{"host_name": "airflow-triggerer-0", "full_command":
"[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:12:22.476 GMT [489] ERROR: relation "log" does not exist at
character 13
2024-04-11 15:12:22.476 GMT [489] STATEMENT: INSERT INTO log (dttm, event,
owner, extra) VALUES ('2024-04-11T15:12:22.462657+00:00'::timestamptz,
'cli_check', 'default', '{"host_name": "airflow-scheduler-58c4b49fc-r98hh",
"full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}')
RETURNING log.id
2024-04-11 15:12:24.925 GMT [491] ERROR: relation "log" does not exist at
character 13
2024-04-11 15:12:24.925 GMT [491] STATEMENT: INSERT INTO log (dttm, event,
owner, extra) VALUES ('2024-04-11T15:12:24.915888+00:00'::timestamptz,
'cli_check', 'default', '{"host_name": "airflow-webserver-76c8cb49f-ms2gr",
"full_command": "[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}')
RETURNING log.id
2024-04-11 15:12:24.978 GMT [492] ERROR: relation "log" does not exist at
character 13
2024-04-11 15:12:24.978 GMT [492] STATEMENT: INSERT INTO log (dttm, event,
owner, extra) VALUES ('2024-04-11T15:12:24.970969+00:00'::timestamptz,
'cli_check', 'default', '{"host_name": "airflow-triggerer-0", "full_command":
"[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:12:25.957 GMT [493] ERROR: relation "log" does not exist at
character 13
2024-04-11 15:12:25.957 GMT [493] STATEMENT: INSERT INTO log (dttm, event,
owner, extra) VALUES ('2024-04-11T15:12:25.947215+00:00'::timestamptz,
'cli_check', 'default', '{"host_name": "airflow-worker-0", "full_command":
"[''/home/airflow/.local/bin/airflow'', ''db'', ''check'']"}') RETURNING log.id
2024-04-11 15:13:13.806 GMT [121] LOG: checkpoint starting: time
2024-04-11 15:13:18.052 GMT [121] LOG: checkpoint complete: wrote 45
buffers (0.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=4.210 s,
sync=0.012 s, total=4.246 s; sync files=12, longest=0.003 s, average=0.001 s;
distance=261 kB, estimate=261 kB; lsn=0/1528A00, redo lsn=0/15289C8
```
### What you think should happen instead
Migration completed successfully
### How to reproduce
deploy helm chart
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
