Good morning

2016-10-18 Thread Alex
Good morning , I have a project i want to bring to you and i want you to help me discuss it . Please reply to me as soon as possible for more details . Thanks & Regards

Re: Usage of "on_failure_callback" ?

2016-10-18 Thread Boris Tyukin
hi Jason, here is an example below - in task0_python_callable i did 1/0 to raise an error and Airflow then called task0_failure_cb. I could see FAIL in the task log so my task0_failure_cb was definitely invoked. The task would still have failed status if look at UI. Are you expecting a

Re: Airflow Logging

2016-10-18 Thread Maycock, Luke
Thank you for the response Maxime. We will consider the points you made when making the changes and may come back with further thoughts as we work on it. I can't say I am too familiar with Kibana, Logstash and Elasticsearch but I am assuming Airflow would still produce logs and you would be

Re: Fwd: Issue with Dynamically created tasks in a DAG

2016-10-18 Thread Laura Lorenz
I think the source of the confusion you're experiencing is that the UI is based off of the DAG definition file at time of webserver load, which I believe is on the one hand defensible since the scheduler operates in a somewhat similar way, but on the other hand rightfully confusing (and doesn't

Re: ETL best practices for airflow

2016-10-18 Thread Gerard Toonstra
Hi Laura, Looks very good. What I had to do first when I started was to figure out relevant concepts for ETL, I don't have a BI background. When I follow the tutorial and look at the examples, it's clear what airflow can do conceptually, but as soon as I want to get started on something, there's