[
https://issues.apache.org/jira/browse/AIRFLOW-2517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495495#comment-16495495
]
ASF subversion and git services commented on AIRFLOW-2517:
----------------------------------------------------------
Commit 3ed25a9459a2a0b26b849b1a3ea7906f26a33707 in incubator-airflow's branch
refs/heads/master from [~milton0825]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3ed25a9 ]
[AIRFLOW-2517] backfill support passing key values through CLI
### JIRA
- [x] My PR addresses the following [Airflow JIRA]
(https://issues.apache.org/jira/browse/AIRFLOW/)
issues and references them in the PR title. For
example, "\[AIRFLOW-XXX\] My Airflow PR"
-
https://issues.apache.org/jira/browse/AIRFLOW-2517
- In case you are fixing a typo in the
documentation you can prepend your commit with
\[AIRFLOW-XXX\], code changes always need a JIRA
issue.
### Description
- [x] Here are some details about my PR, including
screenshots of any UI changes:
In backfill, we can provide key-value pairs
through CLI and those pairs can be accessed
through macros. This is just like the way
`trigger_dag -c` works [1].
Let's walk through an example.
In the airflow CLI we specify a key-value pair.
```
airflow backfill hello_world -s 2018-02-01 -e
2018-02-08 -c '{"text": "some text"}'
```
In the DAG file, I have a `BashOperator` that
contains a template command and I want
{{ dag_run.conf.text }} resolves to the text I
passed in CLI.
```python
templated_command = """
echo "ds = {{ ds }}"
echo "prev_ds = {{
macros.datetime.strftime(prev_execution_date,
"%Y-%m-%d") }}"
echo "next_ds = {{
macros.datetime.strftime(next_execution_date,
"%Y-%m-%d") }}"
echo "text_through_conf = {{ dag_run.conf.text }}"
"""
bash_operator = BashOperator(
task_id='bash_task',
bash_command=templated_command,
dag=dag
)
```
Rendered Bash command in Airflow UI.
<img width="1246" alt="screen shot 2018-05-22 at 4
33 59 pm" src="https://user-images.githubuserconte
nt.com/6065051/40395666-04c41574-5dde-11e8-9ec2-c0
312b7203e6.png">
[1]
https://airflow.apache.org/cli.html#trigger_dag
### Tests
- [x] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason:
### Commits
- [x] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds
documentation that describes how to use it.
- When adding new operators/hooks/sensors, the
autoclass documentation generation needs to be
added.
### Code Quality
- [x] Passes `git diff upstream/master -u --
"*.py" | flake8 --diff`
Closes #3406 from milton0825/backfill-support-conf
> backfill support passing key values through CLI
> -----------------------------------------------
>
> Key: AIRFLOW-2517
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2517
> Project: Apache Airflow
> Issue Type: Improvement
> Reporter: Chao-Han Tsai
> Assignee: Chao-Han Tsai
> Priority: Major
>
> In backfill, we can provide key-value pairs through CLI and those pairs can
> be accessed through macros. This is just like the way `trigger_dag -c` works
> [1].
> Let's walk through an example.
> In the airflow CLI we specify a key-value pair.
> {code:java}
> airflow backfill hello_world -s 2018-02-01 -e 2018-02-08 -c ' {"text": "some
> text"}
> {code}
> In the DAG file, I have a `BashOperator` that contains a template command and
> I want
> \{{ dag_run.conf.text }} resolves to the text I passed in CLI.
> {code:java}
> templated_command = """
> echo "ds = {{ ds }}"
> echo "prev_ds = {{ macros.datetime.strftime(prev_execution_date, "%Y-%m-%d")
> }}"
> echo "next_ds = {{ macros.datetime.strftime(next_execution_date, "%Y-%m-%d")
> }}"
> echo "text_through_conf = {{ dag_run.conf.text }}"
> """
> bash_operator = BashOperator(
> task_id='bash_task',
> bash_command=templated_command,
> dag=dag
> )
> {code}
> [1] [https://airflow.apache.org/cli.html#trigger_dag]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
