Bjorn Olsen created AIRFLOW-6824:
------------------------------------
Summary: EMRAddStepsOperator does not work well with multi-step
XCom
Key: AIRFLOW-6824
URL: https://issues.apache.org/jira/browse/AIRFLOW-6824
Project: Apache Airflow
Issue Type: Bug
Components: aws
Affects Versions: 1.10.9
Reporter: Bjorn Olsen
Assignee: Bjorn Olsen
EmrAddStepsOperator allows you to add several steps to EMR for processing - the
steps must be supplied as a list.
This works well when passing an actual Python list as the 'steps' value, but we
want to be able to generate the list of steps from a previous task - using an
XCom.
We must use the operator as follows, for the templating to work correctly and
for it to resolve the XCom:
{code:java}
add_steps_task = EmrAddStepsOperator(
task_id='add_steps',
job_flow_id=job_flow_id,
aws_conn_id='aws_default',
provide_context=True,
steps="{{task_instance.xcom_pull(task_ids='generate_steps')}}"
){code}
The value in XCom from the 'generate_steps' task looks like (simplified):
{code:java}
[{'Name':'Step1'}, {'Name':'Step2'}]
{code}
However this is passed as a string to the operator, which cannot be passed to
the underlying boto3 library which expects a list object.
The following won't work either:
{code:java}
add_steps_task = EmrAddStepsOperator(
task_id='add_steps',
job_flow_id=job_flow_id,
aws_conn_id='aws_default',
provide_context=True,
steps={{task_instance.xcom_pull(task_ids='generate_steps')}}
){code}
Since this is not valid Python.
We have to pass the steps as a string to the operator, and then convert it into
a list after the render_template_fields has happened (immediately before the
execute). Therefore the only option is to do the conversion from string to list
in the operator's execute method.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
