Hi Andrew, Your task is exiting with “code 0” which means success. I would verify that you are not swallowing the error/return code somewhere.
Cheers Bolke Verstuurd vanaf mijn iPad > Op 14 nov. 2017 om 16:25 heeft Andrew Maguire <[email protected]> het > volgende geschreven: > > Oh, good to know. > > It was just an image of this log info: > > [2017-11-14 15:09:34,595] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:09:34,594] {bash_operator.py:94} INFO - > ################################## > [2017-11-14 15:09:34,595] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:09:34,594] {bash_operator.py:94} INFO - LOAD PREDICTION MAP > [2017-11-14 15:09:34,595] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:09:34,594] {bash_operator.py:94} INFO - > ################################## > *[2017-11-14 15:09:36,695] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:09:36,694] {bash_operator.py:94} INFO - BigQuery error in > load operation: Error processing job* > *[2017-11-14 15:09:36,695] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:09:36,695] {bash_operator.py:94} INFO - > 'pmc-analytical-data-mart:bqjob_r4352c5357a8f26c6_0000015fbb14107a_1': Not* > *[2017-11-14 15:09:36,695] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:09:36,695] {bash_operator.py:94} INFO - found: Uris > gs://pmc-ml/clickmodel/vy/pred//prediction_map** > *[2017-11-14 15:09:38,277] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:09:38,276] {bash_operator.py:94} INFO - BigQuery error in > show operation: Not found: Table* > *[2017-11-14 15:09:38,277] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:09:38,277] {bash_operator.py:94} INFO - > pmc-analytical-data-mart:variety.clickmodel_predictions_map_20171112* > [2017-11-14 15:09:38,332] {base_task_runner.py:95} INFO - Subtask: > [2017-11-14 15:09:38,331] {bash_operator.py:97} INFO - Command exited with > return code 0 > [2017-11-14 15:09:40,783] {jobs.py:2125} INFO - Task exited with return > code 0 > > So you can see the lines in bold are failed bq commands, but for some > reason (maybe how the bq cli operates) Airflow still thinks the task was > successful. > > I think if i was to put something like the below into the bash script then > that would be enough to trigger a failure to airflow - thoughts? > > # capture status of last command and exit if error > status=$? > if [ $status -ne 0 ]; then > echo "Return code was not zero but $status" > exit $status > fi > >> On Tue, Nov 14, 2017 at 3:21 PM Alek Storm <[email protected]> wrote: >> >> Hi Andy, >> >> The list doesn't allow inline images to be posted - can you paste your >> script content as text? >> >> Alek >> >>> On Nov 14, 2017 9:16 AM, "Andrew Maguire" <[email protected]> wrote: >>> >>> Hi, >>> >>> I have some bash operators that are failing but airflow is not picking >> the >>> failure up. >>> >>> Here is an example: >>> >>> [image: image.png] >>> >>> This is a bash script that makes some "bq" and "gcloud" cli commands. >>> >>> I've used $? to get the status of such failed cli commands in the past >> and >>> then do something. >>> >>> I've just wondering - how could i use the $? from the failed bq command >> to >>> in turn pass an error to Airflow? >>> >>> Cheers, >>> Andy >>> >>
