Oh, good to know.
It was just an image of this log info:
[2017-11-14 15:09:34,595] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:34,594] {bash_operator.py:94} INFO -
##################################
[2017-11-14 15:09:34,595] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:34,594] {bash_operator.py:94} INFO - LOAD PREDICTION MAP
[2017-11-14 15:09:34,595] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:34,594] {bash_operator.py:94} INFO -
##################################
*[2017-11-14 15:09:36,695] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:36,694] {bash_operator.py:94} INFO - BigQuery error in
load operation: Error processing job*
*[2017-11-14 15:09:36,695] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:36,695] {bash_operator.py:94} INFO -
'pmc-analytical-data-mart:bqjob_r4352c5357a8f26c6_0000015fbb14107a_1': Not*
*[2017-11-14 15:09:36,695] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:36,695] {bash_operator.py:94} INFO - found: Uris
gs://pmc-ml/clickmodel/vy/pred//prediction_map**
*[2017-11-14 15:09:38,277] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:38,276] {bash_operator.py:94} INFO - BigQuery error in
show operation: Not found: Table*
*[2017-11-14 15:09:38,277] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:38,277] {bash_operator.py:94} INFO -
pmc-analytical-data-mart:variety.clickmodel_predictions_map_20171112*
[2017-11-14 15:09:38,332] {base_task_runner.py:95} INFO - Subtask:
[2017-11-14 15:09:38,331] {bash_operator.py:97} INFO - Command exited with
return code 0
[2017-11-14 15:09:40,783] {jobs.py:2125} INFO - Task exited with return
code 0
So you can see the lines in bold are failed bq commands, but for some
reason (maybe how the bq cli operates) Airflow still thinks the task was
successful.
I think if i was to put something like the below into the bash script then
that would be enough to trigger a failure to airflow - thoughts?
# capture status of last command and exit if error
status=$?
if [ $status -ne 0 ]; then
echo "Return code was not zero but $status"
exit $status
fi
On Tue, Nov 14, 2017 at 3:21 PM Alek Storm <[email protected]> wrote:
> Hi Andy,
>
> The list doesn't allow inline images to be posted - can you paste your
> script content as text?
>
> Alek
>
> On Nov 14, 2017 9:16 AM, "Andrew Maguire" <[email protected]> wrote:
>
> > Hi,
> >
> > I have some bash operators that are failing but airflow is not picking
> the
> > failure up.
> >
> > Here is an example:
> >
> > [image: image.png]
> >
> > This is a bash script that makes some "bq" and "gcloud" cli commands.
> >
> > I've used $? to get the status of such failed cli commands in the past
> and
> > then do something.
> >
> > I've just wondering - how could i use the $? from the failed bq command
> to
> > in turn pass an error to Airflow?
> >
> > Cheers,
> > Andy
> >
>
