You should look in airflow-worker.out for an exception that's happening at about the same time.
Damian -----Original Message----- From: heng gu <[email protected]> Sent: Wednesday, February 26, 2020 11:55 To: [email protected] Subject: Re: task failed without running I got an advice from slack to check airflow-worker.err, and here is what I found File "python3.6/site-packages/airflow/executors/celery_executor.py", line 67, in execute_command close_fds=True, env=env) File "pypacks/lib/python3.6/subprocess.py", line 311, in check_call raise CalledProcessError(retcode, cmd) I am using rabbitmq. And here is the parallelism and concurrency set in cfg file: # The amount of parallelism as a setting to the executor. This defines# the max number of task instances that should run simultaneously# on this airflow installationparallelism = 32 # The number of task instances allowed to run concurrently by the schedulerdag_concurrency = 16 for this dag, I set concurrency=16, max_active_runs=8 Thanks, Heng On Wednesday, February 26, 2020, 03:18:27 AM EST, Mehmet Ersoy <[email protected]> wrote: Hi Heng, I have same problem. Failed jobs do not dump log files. Do you have this problem too? Have you been using Redis as message queue? Also can you send configurations related parallelism and concurrent in airflow.cfg file? Best regards, Mehmet. heng gu <[email protected]>, 24 Şub 2020 Pzt, 17:51 tarihinde şunu yazdı: > I have this dag with a branchpythonoperator task kicking off many of > 24 tasks, in this case, 4 tasks. 2 of the tasks were successful, the > other two (register_YZ, register_ZY) were failed without running (see > the attached UI screen shots). There is no log for tasks register_YZ > and register_ZY. I am using Celery Executor and running 12 workers > executing register_XX tasks. I am using airflow version 1.10.6. Any idea how > to fix it? > -- Mehmet ERSOY =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===============================================================================
