Alexander Petrovsky created AIRFLOW-1358:
--------------------------------------------
Summary: SQL file renderer and on_failure_callback
Key: AIRFLOW-1358
URL: https://issues.apache.org/jira/browse/AIRFLOW-1358
Project: Apache Airflow
Issue Type: New Feature
Components: core
Reporter: Alexander Petrovsky
I have the follow dag:
https://gist.github.com/juise/cc006584e17be72e78550aab53a1be9c
When I broke my firs step (prepare), on_failure_callback function was called,
but it's crashed too, due looks like in sql rendered something is broken, and
he can't work with sql files (by example, by cleanup.sql file, see more info
below). But, when I use flat sql instead of sql file, everything works
perfectly.
My cleanup.sql file:
{code:java}
$ cat cleanup.sql
DROP TABLE IF EXISTS etl.{{ params.target_table }}_{{ ts_nodash }} CASCADE;
{code:java}
Log:
{code:java}
[2017-06-29 15:14:13,474] {models.py:168} INFO - Filling up the DagBag from
/opt/airflow/dags/ga_full_stats_test/ga_full_stats.py
[2017-06-29 15:14:13,852] {base_hook.py:67} INFO - Using connection to:
v2.rtty.in
[2017-06-29 15:14:13,858] {base_hook.py:67} INFO - Using connection to:
vga.rtty.in
[2017-06-29 15:14:13,893] {base_task_runner.py:112} INFO - Running: ['bash',
'-c', u'airflow run ga_full_stats_test_v2 prepare 2017-06-06T10:27:00 --job_id
123798 --raw -sd DAGS_FOLDER/ga_full_stats_test/ga_full_stats.py']
[2017-06-29 15:14:14,391] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:14,391] {__init__.py:57} INFO - Using executor CeleryExecutor
[2017-06-29 15:14:14,924] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:14,924] {models.py:168} INFO - Filling up the DagBag from
/opt/airflow/dags/ga_full_stats_test/ga_full_stats.py
[2017-06-29 15:14:15,088] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,087] {base_hook.py:67} INFO - Using connection to: v2.rtty.in
[2017-06-29 15:14:15,091] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,091] {base_hook.py:67} INFO - Using connection to: vga.rtty.in
[2017-06-29 15:14:15,109] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,109] {models.py:1128} INFO - Dependencies all met for <TaskInstance:
ga_full_stats_test_v2.prepare 2017-06-06 10:27:00 [queued]>
[2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,112] {models.py:1128} INFO - Dependencies all met for <TaskInstance:
ga_full_stats_test_v2.prepare 2017-06-06 10:27:00 [queued]>
[2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,112] {models.py:1328} INFO -
[2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask:
--------------------------------------------------------------------------------
[2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask: Starting
attempt 1 of 1
[2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask:
--------------------------------------------------------------------------------
[2017-06-29 15:14:15,112] {base_task_runner.py:95} INFO - Subtask:
[2017-06-29 15:14:15,120] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,120] {models.py:1352} INFO - Executing <Task(VerticaOperator):
prepare> on 2017-06-06 10:27:00
[2017-06-29 15:14:15,138] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,137] {vertica_operator.py:45} INFO - Executing: CREATE TABLE
etl.stats_full_daily_test_20170606T102700 1 LIKE public.stats_full_daily_test;
[2017-06-29 15:14:15,141] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,141] {base_hook.py:67} INFO - Using connection to: vga.rtty.in
[2017-06-29 15:14:15,158] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,158] {dbapi_hook.py:167} INFO - CREATE TABLE
etl.stats_full_daily_test_20170606T102700 1 LIKE public.stats_full_daily_test;
[2017-06-29 15:14:15,164] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,164] {models.py:1427} ERROR - Severity: ERROR, Message: Syntax error
at or near "1", Sqlstate: 42601, Position: 56, Routine: base_yyerror, File:
/scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043, SQL:
'CREATE TABLE etl.stats_full_daily_test_20170606T102700 1 LIKE
public.stats_full_daily_test;'
[2017-06-29 15:14:15,164] {base_task_runner.py:95} INFO - Subtask: Traceback
(most recent call last):
[2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: File
"/usr/lib/python2.7/site-packages/airflow/models.py", line 1384, in run
[2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: result =
task_copy.execute(context=context)
[2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: File
"/usr/lib/python2.7/site-packages/airflow/contrib/operators/vertica_operator.py",
line 47, in execute
[2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask:
hook.run(self.sql)
[2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: File
"/usr/lib/python2.7/site-packages/airflow/hooks/dbapi_hook.py", line 171, in run
[2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask:
cur.execute(s)
[2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: File
"/usr/lib/python2.7/site-packages/vertica_python/vertica/cursor.py", line 111,
in execute
[2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: raise
errors.QueryError.from_error_response(self._message, operation)
[2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask:
VerticaSyntaxError: Severity: ERROR, Message: Syntax error at or near "1",
Sqlstate: 42601, Position: 56, Routine: base_yyerror, File:
/scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043, SQL:
'CREATE TABLE etl.stats_full_daily_test_20170606T102700 1 LIKE
public.stats_full_daily_test;'
[2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,164] {models.py:1451} INFO - Marking task as FAILED.
[2017-06-29 15:14:15,165] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,165] {vertica_operator.py:45} INFO - Executing: cleanup.sql
[2017-06-29 15:14:15,169] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,169] {base_hook.py:67} INFO - Using connection to: vga.rtty.in
[2017-06-29 15:14:15,181] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,181] {dbapi_hook.py:167} INFO - cleanup.sql
[2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,185] {models.py:1466} ERROR - Failed at executing callback
[2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,185] {models.py:1467} ERROR - Severity: ERROR, Message: Syntax error
at or near "cleanup", Sqlstate: 42601, Position: 1, Routine: base_yyerror,
File: /scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043,
SQL: 'cleanup.sql'
[2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: Traceback
(most recent call last):
[2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: File
"/usr/lib/python2.7/site-packages/airflow/models.py", line 1464, in
handle_failure
[2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask:
task.on_failure_callback(context)
[2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: File
"/opt/airflow/dags/ga_full_stats_test/ga_full_stats.py", line 64, in
cleanup_on_failure
[2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask:
cleanup.execute(context)
[2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: File
"/usr/lib/python2.7/site-packages/airflow/contrib/operators/vertica_operator.py",
line 47, in execute
[2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask:
hook.run(self.sql)
[2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask: File
"/usr/lib/python2.7/site-packages/airflow/hooks/dbapi_hook.py", line 171, in run
[2017-06-29 15:14:15,185] {base_task_runner.py:95} INFO - Subtask:
cur.execute(s)
[2017-06-29 15:14:15,186] {base_task_runner.py:95} INFO - Subtask: File
"/usr/lib/python2.7/site-packages/vertica_python/vertica/cursor.py", line 111,
in execute
[2017-06-29 15:14:15,186] {base_task_runner.py:95} INFO - Subtask: raise
errors.QueryError.from_error_response(self._message, operation)
[2017-06-29 15:14:15,186] {base_task_runner.py:95} INFO - Subtask:
VerticaSyntaxError: Severity: ERROR, Message: Syntax error at or near
"cleanup", Sqlstate: 42601, Position: 1, Routine: base_yyerror, File:
/scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043, SQL:
'cleanup.sql'
[2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: [2017-06-29
15:14:15,191] {models.py:1472} ERROR - Severity: ERROR, Message: Syntax error
at or near "1", Sqlstate: 42601, Position: 56, Routine: base_yyerror, File:
/scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043, SQL:
'CREATE TABLE etl.stats_full_daily_test_20170606T102700 1 LIKE
public.stats_full_daily_test;'
[2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: Traceback
(most recent call last):
[2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: File
"/usr/bin/airflow", line 28, in <module>
[2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask:
args.func(args)
[2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: File
"/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 422, in run
[2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask:
pool=args.pool,
[2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: File
"/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in wrapper
[2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: result =
func(*args, **kwargs)
[2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: File
"/usr/lib/python2.7/site-packages/airflow/models.py", line 1384, in run
[2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: result =
task_copy.execute(context=context)
[2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask: File
"/usr/lib/python2.7/site-packages/airflow/contrib/operators/vertica_operator.py",
line 47, in execute
[2017-06-29 15:14:15,191] {base_task_runner.py:95} INFO - Subtask:
hook.run(self.sql)
[2017-06-29 15:14:15,192] {base_task_runner.py:95} INFO - Subtask: File
"/usr/lib/python2.7/site-packages/airflow/hooks/dbapi_hook.py", line 171, in run
[2017-06-29 15:14:15,192] {base_task_runner.py:95} INFO - Subtask:
cur.execute(s)
[2017-06-29 15:14:15,192] {base_task_runner.py:95} INFO - Subtask: File
"/usr/lib/python2.7/site-packages/vertica_python/vertica/cursor.py", line 111,
in execute
[2017-06-29 15:14:15,192] {base_task_runner.py:95} INFO - Subtask: raise
errors.QueryError.from_error_response(self._message, operation)
[2017-06-29 15:14:15,192] {base_task_runner.py:95} INFO - Subtask:
vertica_python.errors.VerticaSyntaxError: Severity: ERROR, Message: Syntax
error at or near "1", Sqlstate: 42601, Position: 56, Routine: base_yyerror,
File: /scratch_a/release/svrtar18593/vbuild/vertica/Parser/scan.l, Line: 1043,
SQL: 'CREATE TABLE etl.stats_full_daily_test_20170606T102700 1 LIKE
public.stats_full_daily_test;'
[2017-06-29 15:14:15,893] {jobs.py:2107} INFO - Task exited with return code 1
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
