I *think* it might be because xcom_pull only returns one element per task:
https://github.com/apache/incubator-airflow/blob/master/airflow/example_dags/example_xcom.py#L35 You're calling with just a single task_id, so I think you should be doing `for element in ...` rather than `for key, value in ...`. On Thu, Jun 16, 2016 at 11:51 AM, Jeremiah Lowin <jlo...@apache.org> wrote: > Or, I'm afraid my jinja-fu is sorely lacking and I'm not going to be much > help to you here... perhaps Maxime or one of the other devs is more > experienced with templates and can offer some advice? > > Apologies! > Jeremiah > > On Thu, Jun 16, 2016 at 2:40 AM Or Sher <or.sh...@gmail.com> wrote: > > > Thanks Jeremiah! > > > > Looking at the code it seems like get_many can return only many xcom > values > > from different task_ids and not different keys from the same task_id. > > But it doesn't really matter as I would really like to do the iteration > > somehow inside the template. > > I feel like building the template inside the previous python operator is > > kind of missing the whole point of templates as I can already create the > > whole "already rendered" email message at the same time. > > > > I tried pushing one big json xcom to reflect all of the tests, but now > > I'm struggling > > with iterating string json inside the template. > > The following template fails on the "for" line: > > VALIDATION_EMAIL_CONTENT = """ > > <b>Date:</b> {{ ds }}<br> > > <p> > > <table> > > {% for key, value in > > task_instance.xcom_pull(task_ids='data_validation_tests', > > key='tests_results') %} > > {{ key|string() }} > > {{ value|string() }} > > {% endfor %} > > .. > > .. > > > > File "<template>", line 5, in top-level template code > > ValueError: too many values to unpack > > > > I'm guessing it's because the xcom_pull returns a string but I didn't > > managed to cast it to a json yet.. > > > > On Wed, Jun 15, 2016 at 5:33 PM Jeremiah Lowin <jlo...@apache.org> > wrote: > > > > > Hi Or, > > > > > > There is support for pulling multiple XComs at once (see > XCom.get_many()) > > > but it won't be directly available inside a jinja template, where you > you > > > are typically calling ti.xcom_pull() and returning a single XCom at a > > time, > > > using a key you know in advance. > > > > > > I would suggest that you do this processing either inside your existing > > > PythonOperator or in a subsequent PythonOperator that calls > > XCom.get_many() > > > and produces the final template for your email operator. It is > > > theoretically possible for you to do this in your email template using > > > template macros, but I only have a basic knowledge of that (and it's a > > > convoluted process) so I don't want to give you bad advice. If you can > do > > > your preprocessing in your PythonOperator I think you will be happier, > > > especially because the result will be visible in the Airflow UI and > logs. > > > > > > Best, > > > Jeremiah > > > > > > On Wed, Jun 15, 2016 at 9:49 AM Or Sher <or.sh...@gmail.com> wrote: > > > > > > > Hi all, > > > > > > > > For the purpose of data validation tests I have two adjacent > operators: > > > > Python - Parsing predefined json of tests and saves the result as an > > xcom > > > > where each key is a test id. > > > > > > > > Email - I'd like to be able to iterate over all keys from the > previous > > > task > > > > inside the predefined email template so I won't need to change it > every > > > > time I change the tests json. > > > > > > > > It seems like it's not supported. Is that right? > > > > What are my choices? > > > > If I won't find anything else, I assume I can always use one key for > > all > > > of > > > > the tests combined into one json. > > > > > > > > > >
