c-thiel commented on issue #15793: URL: https://github.com/apache/airflow/issues/15793#issuecomment-839662753
@ashb I added the instructions above. Please find following some Screenshots from the setup:  This is the log of one of the python tasks: ``` *** Reading local file: /opt/airflow/logs/test-python/python-185/2021-05-11T00:00:00+00:00/1.log [2021-05-12 10:16:05,723] {taskinstance.py:877} INFO - Dependencies all met for <TaskInstance: test-python.python-185 2021-05-11T00:00:00+00:00 [queued]> [2021-05-12 10:16:06,020] {taskinstance.py:877} INFO - Dependencies all met for <TaskInstance: test-python.python-185 2021-05-11T00:00:00+00:00 [queued]> [2021-05-12 10:16:06,021] {taskinstance.py:1068} INFO - -------------------------------------------------------------------------------- [2021-05-12 10:16:06,021] {taskinstance.py:1069} INFO - Starting attempt 1 of 1 [2021-05-12 10:16:06,021] {taskinstance.py:1070} INFO - -------------------------------------------------------------------------------- [2021-05-12 10:16:06,629] {taskinstance.py:1089} INFO - Executing <Task(PythonOperator): python-185> on 2021-05-11T00:00:00+00:00 [2021-05-12 10:16:06,720] {standard_task_runner.py:52} INFO - Started process 126 to run task [2021-05-12 10:16:06,727] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'test-python', 'python-185', '2021-05-11T00:00:00+00:00', '--job-id', '218', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/dag.py', '--cfg-path', '/tmp/tmpagjtm408', '--error-file', '/tmp/tmp8l_qm7bp'] [2021-05-12 10:16:06,727] {standard_task_runner.py:77} INFO - Job 218: Subtask python-185 [2021-05-12 10:16:13,428] {logging_mixin.py:104} INFO - Running <TaskInstance: test-python.python-185 2021-05-11T00:00:00+00:00 [running]> on host airflow-worker-1.airflow-worker.default.svc.cluster.local [2021-05-12 10:16:15,824] {taskinstance.py:1281} INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=test-python AIRFLOW_CTX_TASK_ID=python-185 AIRFLOW_CTX_EXECUTION_DATE=2021-05-11T00:00:00+00:00 AIRFLOW_CTX_DAG_RUN_ID=scheduled__2021-05-11T00:00:00+00:00 [2021-05-12 10:16:35,921] {python.py:118} INFO - Done. Returned value was: None [2021-05-12 10:16:36,032] {taskinstance.py:1185} INFO - Marking task as SUCCESS. dag_id=test-python, task_id=python-185, execution_date=20210511T000000, start_date=20210512T101605, end_date=20210512T101636 [2021-05-12 10:16:36,434] {taskinstance.py:1246} INFO - 0 downstream tasks scheduled from follow-on schedule check [2021-05-12 10:16:36,621] {local_task_job.py:146} INFO - Task exited with return code 0 [2021-05-12 10:19:45,235] {taskinstance.py:877} INFO - Dependencies all met for <TaskInstance: test-python.python-185 2021-05-11T00:00:00+00:00 [queued]> [2021-05-12 10:19:45,330] {taskinstance.py:867} INFO - Dependencies not met for <TaskInstance: test-python.python-185 2021-05-11T00:00:00+00:00 [queued]>, dependency 'Pool Slots Available' FAILED: ('Not scheduling since there are %s open slots in pool %s and require %s pool slots', -10, 'kubernetes', 5) [2021-05-12 10:19:45,421] {taskinstance.py:1053} WARNING - -------------------------------------------------------------------------------- [2021-05-12 10:19:45,421] {taskinstance.py:1054} WARNING - Rescheduling due to concurrency limits reached at task runtime. Attempt 1 of 1. State set to NONE. [2021-05-12 10:19:45,421] {taskinstance.py:1061} WARNING - -------------------------------------------------------------------------------- [2021-05-12 10:19:45,430] {local_task_job.py:93} INFO - Task is not able to be run [2021-05-12 10:19:47,725] {taskinstance.py:877} INFO - Dependencies all met for <TaskInstance: test-python.python-185 2021-05-11T00:00:00+00:00 [queued]> [2021-05-12 10:19:47,735] {taskinstance.py:867} INFO - Dependencies not met for <TaskInstance: test-python.python-185 2021-05-11T00:00:00+00:00 [queued]>, dependency 'Pool Slots Available' FAILED: ('Not scheduling since there are %s open slots in pool %s and require %s pool slots', -10, 'kubernetes', 5) [2021-05-12 10:19:47,823] {taskinstance.py:1053} WARNING - -------------------------------------------------------------------------------- [2021-05-12 10:19:47,823] {taskinstance.py:1054} WARNING - Rescheduling due to concurrency limits reached at task runtime. Attempt 1 of 1. State set to NONE. [2021-05-12 10:19:47,823] {taskinstance.py:1061} WARNING - -------------------------------------------------------------------------------- [2021-05-12 10:19:47,835] {local_task_job.py:93} INFO - Task is not able to be run [2021-05-12 10:19:49,530] {taskinstance.py:877} INFO - Dependencies all met for <TaskInstance: test-python.python-185 2021-05-11T00:00:00+00:00 [queued]> [2021-05-12 10:19:49,626] {taskinstance.py:867} INFO - Dependencies not met for <TaskInstance: test-python.python-185 2021-05-11T00:00:00+00:00 [queued]>, dependency 'Pool Slots Available' FAILED: ('Not scheduling since there are %s open slots in pool %s and require %s pool slots', -10, 'kubernetes', 5) [2021-05-12 10:19:49,727] {taskinstance.py:1053} WARNING - -------------------------------------------------------------------------------- [2021-05-12 10:19:49,727] {taskinstance.py:1054} WARNING - Rescheduling due to concurrency limits reached at task runtime. Attempt 1 of 1. State set to NONE. [2021-05-12 10:19:49,727] {taskinstance.py:1061} WARNING - -------------------------------------------------------------------------------- [2021-05-12 10:19:49,736] {local_task_job.py:93} INFO - Task is not able to be run [2021-05-12 10:19:51,839] {taskinstance.py:877} INFO - Dependencies all met for <TaskInstance: test-python.python-185 2021-05-11T00:00:00+00:00 [queued]> [2021-05-12 10:19:52,024] {taskinstance.py:867} INFO - Dependencies not met for <TaskInstance: test-python.python-185 2021-05-11T00:00:00+00:00 [queued]>, dependency 'Pool Slots Available' FAILED: ('Not scheduling since there are %s open slots in pool %s and require %s pool slots', -10, 'kubernetes', 5) [2021-05-12 10:19:52,140] {taskinstance.py:1053} WARNING - -------------------------------------------------------------------------------- [2021-05-12 10:19:52,140] {taskinstance.py:1054} WARNING - Rescheduling due to concurrency limits reached at task runtime. Attempt 1 of 1. State set to NONE. [2021-05-12 10:19:52,140] {taskinstance.py:1061} WARNING - -------------------------------------------------------------------------------- [2021-05-12 10:19:52,227] {local_task_job.py:93} INFO - Task is not able to be run ``` It is completly random which ones fail and which ones don't. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
