Re: [Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

2020-01-07 Thread Jacob Ferriero
Image not working on dev list here is link to the github review comment containing said image: https://github.com/apache/airflow/pull/6370#issuecomment-546582724. On Tue, Jan 7, 2020 at 5:40 PM Jacob Ferriero wrote: > Hello Dev List, > > The inspiration for this is to allow operators to start a

[Discussion] In Prep for AIP: Stateful XComs and Poke Rescheduling in Operators

2020-01-07 Thread Jacob Ferriero
Hello Dev List, The inspiration for this is to allow operators to start a long running task on an external system and reschedule pokes for completion (e.g spark job on dataproc), instead of blocking a worker (sketched out in #6210 ) to allow freeing up

Re: Grouping tests using pytest markers

2020-01-07 Thread Jarek Potiuk
Since we need to tackle CI stability I would like to bump this one if anyone wants to say something :) I am full speed ahead with implementing those. *Proposed Breeze changes:* > >- `./breeze` by default will start only the main 'airflow-testing' >image. This way no huge resource usage

Often failing tests in CI (and a way to fix them quickly and future-proof)

2020-01-07 Thread Jarek Potiuk
*TL;DR; I have a proposal how we can remedy often failing CI tests and I have a kind request to other committers to help me to fix it in a good way.* As we all noticed we had recently some often (far too often) failing tests in Travis. The situation is not very good and we have to remedy it