Hi Boris,
Cool, it works fine.
Thanks for the hints using sample codes.
Jason
On Tue, Oct 18, 2016 at 1:12 PM, Boris Tyukin <bo...@boristyukin.com> wrote:
> hi Jason,
>
> here is an example below - in task0_python_callable i did 1/0 to raise an
> error and Airflow then called task0_failure_cb. I could see FAIL in the
> task log so my task0_failure_cb was definitely invoked.
>
> The task would still have failed status if look at UI. Are you expecting a
> different behavior?
>
>
> def task0_python_callable(ds, **kwargs):
> print ds
> print 1/0
> return 'Success!!! Execution time = {}'.format(kwargs['execution_
> date'])
>
>
> def task0_failure_cb(ds, **kwargs):
> logging.info("FAIL")
>
> task0 = PythonOperator(
> task_id='task0',
> python_callable=task0_python_callable,
> on_failure_callback=task0_failure_cb,
> provide_context=True,
> dag=dag)
>
>
>
> [2016-10-18 16:07:02,631] {models.py:154} INFO - Filling up the DagBag from
> /home/oracle/airflow/dags/fail_callback.py
> [2016-10-18 16:07:03,239] {models.py:154} INFO - Filling up the DagBag from
> /home/oracle/airflow/dags/fail_callback.py
> [2016-10-18 16:07:03,312] {models.py:1196} INFO -
> ------------------------------------------------------------
> --------------------
> Starting attempt 1 of 1
> ------------------------------------------------------------
> --------------------
>
> [2016-10-18 16:07:03,317] {models.py:1219} INFO - Executing
> <Task(PythonOperator): task0> on 2016-10-18 16:06:02
> [2016-10-18 16:07:03,322] {models.py:1286} ERROR - integer division or
> modulo by zero
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line
> 1245, in run
> result = task_copy.execute(context=context)
> File
> "/usr/local/lib/python2.7/site-packages/airflow/
> operators/python_operator.py",
> line 66, in execute
> return_value = self.python_callable(*self.op_args, **self.op_kwargs)
> File "/home/oracle/airflow/dags/fail_callback.py", line 33, in
> task0_python_callable
> print 1/0
> ZeroDivisionError: integer division or modulo by zero
> [2016-10-18 16:07:03,324] {models.py:1306} INFO - Marking task as FAILED.
> [2016-10-18 16:07:03,324] {fail_callback.py:38} INFO - FAIL
> [2016-10-18 16:07:03,327] {models.py:1327} ERROR - integer division or
> modulo by zero
>
>
> On Tue, Oct 18, 2016 at 2:59 PM, Jason Chen <chingchien.c...@gmail.com>
> wrote:
>
> > Hi airflow team,
> >
> > Is there any sample code to use "on_failure_callback" ?
> > I tried to use that as a callback to "post-process" when a task fails.
> > However, I cannot make it work.
> >
> > The definition of my task is as below (just one task in my dag).
> >
> > It invokes "task0_python_callable" which executes a command-line script
> and
> > it could drops exception. It in turns making task0 failed.
> > In that case, I am thinking it should trigger the "on_failure_callback"
> > function "task0_failure_cb", but it does not happen.
> >
> > Any suggestion?
> > Thanks.
> >
> > Jason
> >
> > =========
> > task0 = PythonOperator(
> > task_id='task0',
> > python_callable=task0_python_callable,
> > on_failure_callback=task0_failure_cb,
> > dag=dag)
> >
>
