{'forcerun': 1} is not valid JSON because of the single quotes, thus the
ValueError when decoding. Try --conf '{"forcerun": 1}'.-Daniel On Thu, Jul 27, 2017 at 11:52 AM, Andrew Maguire <[email protected]> wrote: > Hi, > > Does anyone have an example of how to pass --conf json string to > trigger_dag from the cli? > > I need to trigger a range of historic dag's but i don't want to do backfill > as the dag runs each hour and that would create many duplicate runs. Also > as part of the dag the first step is to check if source data has changed > and if not then do nothing - this is to try be data driven with the > execution. > > Issue is i've found a bug in my sql logic. So i want to force it to rerun > based on a param i want to send in via the command line (am guessing --conf > is way to do this?). > > So instead of a backfill i want to do a series of > > airflow trigger_dag > > Commands to essentially mimic running the dag once per day for a given date > with a param to tell it to ignore the usual data checks and just run > regardless. > > I tried: > > airflow trigger_dag ga_data --exec_date 2017-06-26 --conf "{'forcerun':1}" > > > Where 'forcerun' is a param i want to pass to tell the dag to ignore the > usual data driven checks and just run regardless. > > I get this error: > > /usr/local/lib/python2.7/dist-packages/airflow-1.9.0.dev0+ > apache.incubating-py2.7.egg/airflow/configuration.py:553: > DeprecationWarning: This method will be removed in future versions. > Use 'parser.read_file()' instead. > self.readfp(StringIO.StringIO(string)) > [2017-07-27 18:37:03,706] {__init__.py:56} INFO - Using executor > LocalExecutor > [2017-07-27 18:37:03,812] {driver.py:120} INFO - Generating grammar > tables from /usr/lib/python2.7/lib2to3/Grammar.txt > [2017-07-27 18:37:03,842] {driver.py:120} INFO - Generating grammar > tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt > [2017-07-27 18:37:04,113] {models.py:167} INFO - Filling up the DagBag > from /home/andrew_maguire/pmc-analytical-data-mart/airflow/dags > Traceback (most recent call last): > File "/usr/local/bin/airflow", line 4, in <module> > __import__('pkg_resources').run_script('airflow==1.9.0. > dev0+apache.incubating', > 'airflow') > File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", > line 719, in run_script > self.require(requires)[0].run_script(script_name, ns) > File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", > line 1504, in run_script > exec(code, namespace, namespace) > File "/usr/local/lib/python2.7/dist-packages/airflow-1.9.0. > dev0+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow", > line 28, in <module> > args.func(args) > File "/usr/local/lib/python2.7/dist-packages/airflow-1.9.0. > dev0+apache.incubating-py2.7.egg/airflow/bin/cli.py", > line 180, in trigger_dag > execution_date=args.exec_date) > File "/usr/local/lib/python2.7/dist-packages/airflow-1.9.0. > dev0+apache.incubating-py2.7.egg/airflow/api/client/local_client.py", > line 24, in trigger_dag > execution_date=execution_date) > File "/usr/local/lib/python2.7/dist-packages/airflow-1.9.0. > dev0+apache.incubating-py2.7.egg/airflow/api/common/ > experimental/trigger_dag.py", > line 49, in trigger_dag > run_conf = json.loads(conf) > File "/usr/lib/python2.7/json/__init__.py", line 339, in loads > return _default_decoder.decode(s) > File "/usr/lib/python2.7/json/decoder.py", line 364, in decode > obj, end = self.raw_decode(s, idx=_w(s, 0).end()) > File "/usr/lib/python2.7/json/decoder.py", line 380, in raw_decode > obj, end = self.scan_once(s, idx) > ValueError: Expecting property name: line 1 column 2 (char 1) > > > I'm sure i either need to pass more key values to --conf of maybe structure > the json differently but i cant seem to find any examples anywhere of how > to go about this. > > Any pointers very welcome. > > Cheers, > Andy >
