Hi,
Does anyone have an example of how to pass --conf json string to
trigger_dag from the cli?
I need to trigger a range of historic dag's but i don't want to do backfill
as the dag runs each hour and that would create many duplicate runs. Also
as part of the dag the first step is to check if source data has changed
and if not then do nothing - this is to try be data driven with the
execution.
Issue is i've found a bug in my sql logic. So i want to force it to rerun
based on a param i want to send in via the command line (am guessing --conf
is way to do this?).
So instead of a backfill i want to do a series of
airflow trigger_dag
Commands to essentially mimic running the dag once per day for a given date
with a param to tell it to ignore the usual data checks and just run
regardless.
I tried:
airflow trigger_dag ga_data --exec_date 2017-06-26 --conf "{'forcerun':1}"
Where 'forcerun' is a param i want to pass to tell the dag to ignore the
usual data driven checks and just run regardless.
I get this error:
/usr/local/lib/python2.7/dist-packages/airflow-1.9.0.dev0+apache.incubating-py2.7.egg/airflow/configuration.py:553:
DeprecationWarning: This method will be removed in future versions.
Use 'parser.read_file()' instead.
self.readfp(StringIO.StringIO(string))
[2017-07-27 18:37:03,706] {__init__.py:56} INFO - Using executor LocalExecutor
[2017-07-27 18:37:03,812] {driver.py:120} INFO - Generating grammar
tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2017-07-27 18:37:03,842] {driver.py:120} INFO - Generating grammar
tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
[2017-07-27 18:37:04,113] {models.py:167} INFO - Filling up the DagBag
from /home/andrew_maguire/pmc-analytical-data-mart/airflow/dags
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 4, in <module>
__import__('pkg_resources').run_script('airflow==1.9.0.dev0+apache.incubating',
'airflow')
File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py",
line 719, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py",
line 1504, in run_script
exec(code, namespace, namespace)
File
"/usr/local/lib/python2.7/dist-packages/airflow-1.9.0.dev0+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
line 28, in <module>
args.func(args)
File
"/usr/local/lib/python2.7/dist-packages/airflow-1.9.0.dev0+apache.incubating-py2.7.egg/airflow/bin/cli.py",
line 180, in trigger_dag
execution_date=args.exec_date)
File
"/usr/local/lib/python2.7/dist-packages/airflow-1.9.0.dev0+apache.incubating-py2.7.egg/airflow/api/client/local_client.py",
line 24, in trigger_dag
execution_date=execution_date)
File
"/usr/local/lib/python2.7/dist-packages/airflow-1.9.0.dev0+apache.incubating-py2.7.egg/airflow/api/common/experimental/trigger_dag.py",
line 49, in trigger_dag
run_conf = json.loads(conf)
File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 380, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting property name: line 1 column 2 (char 1)
I'm sure i either need to pass more key values to --conf of maybe structure
the json differently but i cant seem to find any examples anywhere of how
to go about this.
Any pointers very welcome.
Cheers,
Andy
