That snippet shouldn’t be necessary; I assume the bq command was not the last one of your script. Add set -e to the top to exit on the first failure (this is generally what people expect from experience with scripting languages).
Alek On Tue, Nov 14, 2017 at 9:41 AM, Andrew Maguire <[email protected]> wrote: > Thanks, > > I've added this into my script and it now triggers error in Airflow. > > # capture status of last command and exit if error > status=$? > if [ $status -ne 0 ]; then > echo "Return code was not zero but $status" > exit $status > fi > > Now it triggers below output. > > I'll try figure out what's going on as would be a bit ugly to have to add > this everywhere (although for now will work fine if i add to just the last > step even). > > [2017-11-14 15:33:57,545] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:33:57,545] {bash_operator.py:94} INFO - > ################################## > [2017-11-14 15:33:57,546] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:33:57,545] {bash_operator.py:94} INFO - LOAD PREDICTIONS > [2017-11-14 15:33:57,546] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:33:57,545] {bash_operator.py:94} INFO - > ################################## > [2017-11-14 15:34:00,147] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:34:00,146] {bash_operator.py:94} INFO - BigQuery error in > load operation: Error processing job > [2017-11-14 15:34:00,147] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:34:00,147] {bash_operator.py:94} INFO - > 'pmc-analytical-data-mart:bqjob_r57a28bce8e342beb_0000015fbb2a633b_1': Not > [2017-11-14 15:34:00,147] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:34:00,147] {bash_operator.py:94} INFO - found: Uris > gs://pmc-ml/clickmodel/vy/pred//prediction.results* > [2017-11-14 15:34:00,209] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:34:00,208] {bash_operator.py:94} INFO - Return code was not > zero but 2 > [2017-11-14 15:34:00,209] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:34:00,209] {bash_operator.py:97} INFO - Command exited with > return code 2 > [2017-11-14 15:34:00,210] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:34:00,209] {models.py:1433} ERROR - Bash command failed > [2017-11-14 15:34:00,210] {base_task_runner.py:95} INFO - Subtask: > Traceback (most recent call last): > [2017-11-14 15:34:00,210] {base_task_runner.py:95} INFO - Subtask: File > "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1390, in > run > [2017-11-14 15:34:00,210] {base_task_runner.py:95} INFO - Subtask: > result = task_copy.execute(context=context) > [2017-11-14 15:34:00,210] {base_task_runner.py:95} INFO - Subtask: File > "/usr/local/lib/python2.7/dist-packages/airflow/ > operators/bash_operator.py", > line 100, in execute > [2017-11-14 15:34:00,211] {base_task_runner.py:95} INFO - Subtask: > raise AirflowException("Bash command failed") > [2017-11-14 15:34:00,211] {base_task_runner.py:95} INFO - Subtask: > AirflowException: Bash command failed > [2017-11-14 15:34:00,211] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:34:00,210] {models.py:1449} INFO - Marking task as > UP_FOR_RETRY > [2017-11-14 15:34:00,474] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:34:00,474] {email.py:109} INFO - Sent an alert email to [' > [email protected]'] > [2017-11-14 15:34:02,181] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:34:02,181] {models.py:1478} ERROR - Bash command failed > [2017-11-14 15:34:02,182] {base_task_runner.py:95} INFO - Subtask: > Traceback (most recent call last): > [2017-11-14 15:34:02,183] {base_task_runner.py:95} INFO - Subtask: File > "/usr/local/bin/airflow", line 28, in <module> > [2017-11-14 15:34:02,183] {base_task_runner.py:95} INFO - Subtask: > args.func(args) > [2017-11-14 15:34:02,183] {base_task_runner.py:95} INFO - Subtask: File > "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 422, in > run > [2017-11-14 15:34:02,184] {base_task_runner.py:95} INFO - Subtask: > pool=args.pool, > [2017-11-14 15:34:02,184] {base_task_runner.py:95} INFO - Subtask: File > "/usr/local/lib/python2.7/dist-packages/airflow/utils/db.py", line 53, in > wrapper > [2017-11-14 15:34:02,185] {base_task_runner.py:95} INFO - Subtask: > result = func(*args, **kwargs) > [2017-11-14 15:34:02,185] {base_task_runner.py:95} INFO - Subtask: File > "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1390, in > run > [2017-11-14 15:34:02,186] {base_task_runner.py:95} INFO - Subtask: > result = task_copy.execute(context=context) > [2017-11-14 15:34:02,187] {base_task_runner.py:95} INFO - Subtask: File > "/usr/local/lib/python2.7/dist-packages/airflow/ > operators/bash_operator.py", > line 100, in execute > [2017-11-14 15:34:02,187] {base_task_runner.py:95} INFO - Subtask: > raise AirflowException("Bash command failed") > [2017-11-14 15:34:02,188] {base_task_runner.py:95} INFO - Subtask: > airflow.exceptions.AirflowException: Bash command failed > [2017-11-14 15:34:06,994] {jobs.py:2125} INFO - Task exited with return > code 1 > > On Tue, Nov 14, 2017 at 3:33 PM Bolke de Bruin <[email protected]> wrote: > > > Hi Andrew, > > > > Your task is exiting with “code 0” which means success. I would verify > > that you are not swallowing the error/return code somewhere. > > > > Cheers > > Bolke > > > > Verstuurd vanaf mijn iPad > > > > > Op 14 nov. 2017 om 16:25 heeft Andrew Maguire <[email protected]> > > het volgende geschreven: > > > > > > Oh, good to know. > > > > > > It was just an image of this log info: > > > > > > [2017-11-14 15:09:34,595] {base_task_runner.py:95} INFO - Subtask: > > > [2017-11-14 15:09:34,594] {bash_operator.py:94} INFO - > > > ################################## > > > [2017-11-14 15:09:34,595] {base_task_runner.py:95} INFO - Subtask: > > > [2017-11-14 15:09:34,594] {bash_operator.py:94} INFO - LOAD PREDICTION > > MAP > > > [2017-11-14 15:09:34,595] {base_task_runner.py:95} INFO - Subtask: > > > [2017-11-14 15:09:34,594] {bash_operator.py:94} INFO - > > > ################################## > > > *[2017-11-14 15:09:36,695] {base_task_runner.py:95} INFO - Subtask: > > > [2017-11-14 15:09:36,694] {bash_operator.py:94} INFO - BigQuery error > in > > > load operation: Error processing job* > > > *[2017-11-14 15:09:36,695] {base_task_runner.py:95} INFO - Subtask: > > > [2017-11-14 15:09:36,695] {bash_operator.py:94} INFO - > > > 'pmc-analytical-data-mart:bqjob_r4352c5357a8f26c6_0000015fbb14107a_1': > > Not* > > > *[2017-11-14 15:09:36,695] {base_task_runner.py:95} INFO - Subtask: > > > [2017-11-14 15:09:36,695] {bash_operator.py:94} INFO - found: Uris > > > gs://pmc-ml/clickmodel/vy/pred//prediction_map** > > > *[2017-11-14 15:09:38,277] {base_task_runner.py:95} INFO - Subtask: > > > [2017-11-14 15:09:38,276] {bash_operator.py:94} INFO - BigQuery error > in > > > show operation: Not found: Table* > > > *[2017-11-14 15:09:38,277] {base_task_runner.py:95} INFO - Subtask: > > > [2017-11-14 15:09:38,277] {bash_operator.py:94} INFO - > > > pmc-analytical-data-mart:variety.clickmodel_predictions_map_20171112* > > > [2017-11-14 15:09:38,332] {base_task_runner.py:95} INFO - Subtask: > > > [2017-11-14 15:09:38,331] {bash_operator.py:97} INFO - Command exited > > with > > > return code 0 > > > [2017-11-14 15:09:40,783] {jobs.py:2125} INFO - Task exited with return > > > code 0 > > > > > > So you can see the lines in bold are failed bq commands, but for some > > > reason (maybe how the bq cli operates) Airflow still thinks the task > was > > > successful. > > > > > > I think if i was to put something like the below into the bash script > > then > > > that would be enough to trigger a failure to airflow - thoughts? > > > > > > # capture status of last command and exit if error > > > status=$? > > > if [ $status -ne 0 ]; then > > > echo "Return code was not zero but $status" > > > exit $status > > > fi > > > > > >> On Tue, Nov 14, 2017 at 3:21 PM Alek Storm <[email protected]> > > wrote: > > >> > > >> Hi Andy, > > >> > > >> The list doesn't allow inline images to be posted - can you paste your > > >> script content as text? > > >> > > >> Alek > > >> > > >>> On Nov 14, 2017 9:16 AM, "Andrew Maguire" <[email protected]> > > wrote: > > >>> > > >>> Hi, > > >>> > > >>> I have some bash operators that are failing but airflow is not > picking > > >> the > > >>> failure up. > > >>> > > >>> Here is an example: > > >>> > > >>> [image: image.png] > > >>> > > >>> This is a bash script that makes some "bq" and "gcloud" cli commands. > > >>> > > >>> I've used $? to get the status of such failed cli commands in the > past > > >> and > > >>> then do something. > > >>> > > >>> I've just wondering - how could i use the $? from the failed bq > command > > >> to > > >>> in turn pass an error to Airflow? > > >>> > > >>> Cheers, > > >>> Andy > > >>> > > >> > > >
