Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

2017-02-25 Thread Daniel Huang
I think the skipped task issue mentioned is the same issue described in AIRFLOW-872 . I had a DAG that was consistently hitting it. I believe the skipped task has to have at least one downstream task for this to occur (e.g. LatestOnlyOperator >>

Cutting down on testing time - updated

2017-02-25 Thread Bolke de Bruin
Hi All, (Welcome to new MacBook Pro that has a send “button” on the touch bar) Jeremiah and I have been looking into optimising the time that is spend on tests. The reason for this was that Travis’ runs are taking more and more time and we are being throttled by travis. As part of that we

Cutting down on testing time

2017-02-25 Thread Bolke de Bruin
Hi All, Jeremiah and I have been looking into optimising the time that is spend on tests. The reason for this was that Travis’ runs are taking more and more time and we are being throttled by travis. As part of that we enabled color coding of test outcomes and timing of tests. The results kind

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

2017-02-25 Thread Jeremiah Lowin
Interesting if this is related to what I was seeing -- but to be clear the error I observed is non-deterministic and doesn't happen every time (obviously, because otherwise there would be no passing Travis runs). Is that the case for what you're describing, Dan/Alex? On Sat, Feb 25, 2017 at 4:13

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

2017-02-25 Thread Alex Van Boxel
About: Skipped tasks potentially cause a dagrun to be marked failure/success prematurely. Isn't that related to the discussion I had with Max about the ONE_SUCCESS trigger? When skipping tasks for now you need to put ONE_SUCCESS. I had kind of a fix but it was rejected because it changed

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

2017-02-25 Thread Bolke de Bruin
Not trying to muddy the waters, but the observation of Jeremiah (non deterministic outcomes) might have to do something with #3. I didn’t dive in deeper, yet. == ERROR: test_backfill_examples (tests.BackfillJobTest)

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

2017-02-25 Thread Bolke de Bruin
Hi Dan, - Backfill indeed runs only one dagrun at the time, see line 1755 of jobs.py. I’ll think about how to fix this over the weekend (I think it was my change that introduced this). Suggestions always welcome. Depending the impact it is a blocker or not. We don’t often use backfills and