@bolke, this sounds like a good list. On Wed, Jan 11, 2017 at 12:01 PM, Bolke de Bruin <[email protected]> wrote:
> Ok. > > For now to call it “beta” 4 items seems to be left: > > Blocker: > * retry_delay not respected > * poison pill due to re-queue before process has finished (to be > investigated) > > Features: > * cgroups + impersonation > * dag.catchup (Ben Tallman -> Only documentation is missing). > > PRs that contain documentation would really be appreciated. In my opinion > we are lacking there. Think about docs covering: > * new scheduler behaviour and options > * task dependency engine > * api / kerberized api > * … > > Cheers > Bolke > > > On 11 Jan 2017, at 18:59, Arthur Wiedmer <[email protected]> > wrote: > > > > +1 > > > > We can always think about different ways of doing this later (fair share > > scheduling etc...) > > > > Best, > > Arthur > > > > On Wed, Jan 11, 2017 at 4:46 AM, Bolke de Bruin <[email protected]> > wrote: > > > >> Dear All, > >> > >> I would like to drop "Schedule all pending DAG runs in a single > scheduler > >> loop” from the 1.8.0 release (updated: https://github.com/apache/ > >> incubator-airflow/pull/1980 <https://github.com/apache/ > >> incubator-airflow/pull/1980>, original: https://github.com/apache/ > >> incubator-airflow/pull/1906 <https://github.com/apache/ > >> incubator-airflow/pull/1906>). The reason for this is that it, imho, > >> biases the scheduler towards a single DAG as it fills the queue with > tasks > >> from one DAG and then goes to the next DAG. Starving DAGs that come > after > >> the first for resources. As such it should be updated and that will take > >> time. > >> > >> Please let me know if I am incorrect. > >> > >> Thanks > >> Bolke > >> > >>> On 10 Jan 2017, at 09:25, Bolke de Bruin <[email protected]> wrote: > >>> > >>> Dear All, > >>> > >>> I have made Airflow 1.8.0 alpha 4 available at > >> https://people.apache.org/~bolke/ <https://people.apache.org/~bolke/> . > >> Again no Apache release yet - this is for testing purposes. I consider > this > >> Alpha to be a Beta if not for the pending features. If the pending > features > >> are merged within a reasonable time frame (except for **, as no progress > >> currently) then I am planning to mark the tarball as Beta and only allow > >> bug fixes and (very) minor features. This week hopefully. > >>> > >>> Blockers: > >>> > >>> * None > >>> > >>> Fixed issues > >>> * Regression in email > >>> * LDAP case sensitivity > >>> * one_failed task not being run: now seems to pass suddenly (so fixed?) > >> -> need to investigate why > >>> * Email attachments > >>> * Pinned jinja2 to < 2.9.0 (2.9.1 has a confirmed regression) > >>> * Improve time units for task performance charts > >>> * XCom throws an duplicate / locking error > >>> * Add execution_date to trigger_dag > >>> > >>> Pending features: > >>> * DAG.catchup : minor changes needed, documentation still required, > >> integration tests seem to pass flawlessly > >>> * Cgroups + impersonation: clean up of patches on going, more tests and > >> more elaborate documentation required. Integration tests not executed > yet > >>> * Schedule all pending DAG runs in a single scheduler loop: no progress > >> (**) > >>> > >>> Cheers! > >>> Bolke > >> > >> > >
