Damian, None of the tasks in the dag take that long. The task that appears to be coinciding with the scheduler error only takes about 1-3min. on another server that I am trying to migrate the dag from (which uses airflow v1.9.0 as opposed to 1.10..0 here on the new server) and manually running the bash commands in the terminal that the task is programmed to do only takes about 1min. as well.
On Mon, Dec 9, 2019 at 11:24 AM Shaw, Damian P. < [email protected]> wrote: > If you’re using the Sequential Executor maybe some of your tasks are > taking longer than 45 mins? > > > > For the sake of task isolation and many other reasons it’s very helpful to > use a different Executor, first step is Local Executor and then maybe > something more fancy. > > > > Damian > > > > *From:* Reed Villanueva <[email protected]> > *Sent:* Monday, December 9, 2019 3:48 PM > *To:* [email protected] > *Subject:* Airflow scheduler complains no heartbeat when running daemon > > > > Have problem where the airflow (v1.10.5) webserver will complain... > > The scheduler does not appear to be running. Last heartbeat was received > 45 minutes ago. > > But checking the scheduler daemon process (started via airflow scheduler > -D) can see... > > [airflow@airflowetl airflow]$ cat airflow-scheduler.pid > > 64186 > > [airflow@airflowetl airflow]$ ps -aux | grep 64186 > > airflow 64186 0.0 0.1 663340 67796 ? S 15:03 0:00 > /usr/bin/python3 /home/airflow/.local/bin/airflow scheduler -D > > airflow 94305 0.0 0.0 112716 964 pts/4 R+ 16:01 0:00 grep > --color=auto 64186 > > and after some period of time the error message *goes away again*). > > This happens very frequently off-and-on even after restarting both the > webserver and scheduler. > > The airflow-scheduler.err file is empty and the .out and .log files > appear innocuous (need more time to look through deeper). > > Running the scheduler in the terminal to see the feed live, everything > seems to run fine until I see this output in the middle of the dag execution > > [2019-11-29 15:51:57,825] {__init__.py:51} INFO - Using executor > SequentialExecutor > > [2019-11-29 15:51:58,259] {dagbag.py:90} INFO - Filling up the DagBag from > /home/airflow/airflow/dags/my_dag_file.py > > Once this pops up, I can see in the web UI that the scheduler heartbeat > error message appears. (Oddly, killing the scheduler process here does not > generate the heartbeat error message in the web UI). Checking for the > scheduler process, I see... > > [airflow@airflowetl airflow]$ ps -aux | grep scheduler > > airflow 3409 0.2 0.1 523336 67384 ? S Oct24 115:06 airflow > scheduler -- DagFileProcessorManager > > airflow 25569 0.0 0.0 112716 968 pts/4 S+ 16:00 0:00 grep > --color=auto scheduler > > airflow 56771 0.0 0.1 662560 67264 ? S Nov26 4:09 airflow > scheduler -- DagFileProcessorManager > > airflow 64187 0.0 0.1 662564 67096 ? S Nov27 0:00 airflow > scheduler -- DagFileProcessorManager > > airflow 153959 0.1 0.1 662568 67232 ? S 15:01 0:06 airflow > scheduler -- DagFileProcessorManager > > IDK if this is this normal or not. > > Thought the problem may have been that there were older scheduler > processes that were not deleted that were still running... > > [airflow@airflowetl airflow]$ kill -9 3409 36771 > > bash: kill: (36771) - No such process > > [airflow@airflowetl airflow]$ ps -aux | grep scheduler > > airflow 56771 0.0 0.1 662560 67264 ? S Nov26 4:09 airflow > scheduler -- DagFileProcessorManager > > airflow 64187 0.0 0.1 662564 67096 ? S Nov27 0:00 airflow > scheduler -- DagFileProcessorManager > > airflow 153959 0.0 0.1 662568 67232 ? S Nov29 0:06 airflow > scheduler -- DagFileProcessorManager > > airflow 155741 0.0 0.0 112712 968 pts/2 R+ 15:54 0:00 grep > --color=auto scheduler > > Notice all the various start times in the output. > > Doing a kill -9 56771 64187 ... and then rerunning airflow scheduler -D does > not seem to have fixed the problem. > > Note: the scheduler seems to consistently stop running after a task fails > to move a file from an FTP location to an HDFS one... > > hadoop fs -Dfs.mapr.trace=debug -get \ > > ftp://$FTP_CLIENT:$FTP_PASS@$FTP_IP/$FTP_DIR"$TABLENAME.TSV"; \ > > $PROJECT_HOME/tmp/"$TABLENAME.TSV" \ > > | hadoop fs -moveFromLocal $PROJECT_HOME/tmp/"$TABLENAME.TSV" > "$DATASTORE" > > # see https://stackoverflow.com/a/46433847/8236733 > > Note there *is* a logic error in this line since $DATASTORE is a hdfs dir > path, not a file path, but either way I don't think that the airflow > scheduler should be missing heartbeats like this from something seemingly > so unrelated. > > Anyone know what could be going on here or how to fix? > > > This electronic message is intended only for the named > recipient, and may contain information that is confidential or > privileged. If you are not the intended recipient, you are > hereby notified that any disclosure, copying, distribution or > use of the contents of this message is strictly prohibited. If > you have received this message in error or are not the named > recipient, please notify us immediately by contacting the > sender at the electronic mail address noted above, and delete > and destroy all copies of this message. Thank you. > > > > > ============================================================================== > Please access the attached hyperlink for an important electronic > communications disclaimer: > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > > ============================================================================== > -- This electronic message is intended only for the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
