t4n1o edited a comment on issue #19192:
URL: https://github.com/apache/airflow/issues/19192#issuecomment-1003779286
> 1. Do you actually have scheduler running at all? Does it have
coninuous access to the DB?
Yes, the scheduler is visible in the running processes, and also systemd.
The scheduler is the only program accessing the db. The db is on the same
RAID array/filesystem as the data.
> 2. Are you absolutely sure you are not using SequentialExecutior ?
What does your `airflow info` say - can you paste-bin output of it ? (airlfow
has built-in flag to send to pastebin). Please make sure also that you do it in
exactly the way your scheduler works. Miost likely you run your scheduler with
a different configuration than your webserver and that causes the problem.
I can't find the flag for pastebin but I pasted above.
> 3. Are you sure you are using Postgres and not Sqlite?
Yes
> 4. Where is your Python code (non-DAG)? Did you .airflowignore
non-DAG files from airflow's DAG folder?
/opt/data-workflows <= code
/opt/data-workflows/rust <= code
/opt/data-workflows/dags
There are only dags in the dag folder, I kept everything modular and
separated.
> 5. can you upgrade to Airlfow 2.2.3 (latest released) - it has
built-in warnings in case you use Sequential Executor/SQLite in the UI.
Yes, still the same problem on 2.2.3. No warnings about sqllite or
sequential executor
> 6. Can you change your DAGs to:
When I run your dag with the `sleep 1000`, everything works properly, can
launch many dags in parallel. But launching my standalone scripts (any of them
- python or rust ones) with BashOperator in a new process creates this problem
;/ I considered whether there could be contention between reading/writing from
the disks, because the postgres db is on the same disks, but this weird state
can persist indefinitely, and I believe linux would allow postgres to at least
take turns reading/writing from the disk.
Perhaps there is something weird in the way BashOperator spawns subprocesses
that can block the entire scheduler when used this way. Or a bug with
subprocess. Or maybe it doesn't like the CWD to be changed.
```
bash_command="umask 002 && cd /opt/data_workflows/rust/ && "
"./parser",
```
By the way, I'm sorry this turned into an unpaid debugging session, it
wasn't my intention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
