Thanks Ash for putting together the doc, somehow I cannot do anything on
confluence so I'll put my comments here.
+1 for using this opportunity to define how we want to do releases, e.g.
frequency, compatibility rules, etc.
If the DAG isolation is being worked on I would love to see it in 2.0.
Adding two other items I think are quite important:
- DB reliability/performance
- DB is a single point of failure just as the scheduler and per
experience operating a huge cluster in Airbnb( 6k+ DAGs/60k+
tasks), it is
a bigger treat on the stability of Airflow
- If the reason behind improving scheduler performance is
scalability then I think we can instead work on the DB, or something like
AIP-17
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+Airflow+sensor+optimization>
- Project baseline
- As we grow more mature doing releases, we should consider establish
the baseline for Airflow and thus create easier upgrade experience, e.g.
performance benchmarking, defining API( not the web endpoint but API like
how each operator params are used) and tests on them, etc.
- Not necessarily need to be fully included in 2.0 as I image this
would be a long incremental work but the earlier we start the earlier we
benefit
Cheers,
Kevin Y
On Wed, Oct 9, 2019 at 7:00 PM Chao-Han Tsai <milton0...@gmail.com> wrote:
> Although Airflow has the concept of task priority like Ash mentioned, it
> does not pre-empt running tasks.
>
> On Wed, Oct 9, 2019 at 12:42 AM Ash Berlin-Taylor <a...@apache.org> wrote:
>
> > There's already a concept called priority_weight on tasks
> > http://airflow.apache.org/concepts.html?highlight=priority_weight#pools
> > (the doc about it is in relation to pools, but everything is run in a
> pool
> > of "default_pool" if not specified.)
> >
> > Is that what you want?
> >
> > On 9 October 2019 07:38:38 BST, bharath palaksha <bharath...@gmail.com>
> > wrote:
> > >Hi,
> > >
> > >Is there any discussion thread on adding priority to tasks and
> > >cost-based
> > >optimization?
> > >priority and pre-emption as an option to the user. If priority is
> > >specified, scheduler has to schedule high priority tasks and if
> > >pre-emption
> > >is true, it can pre-empt current running task which is of lower
> > >priority
> > >
> > >Thanks,
> > >Bharath
> > >
> > >
> > >On Mon, Sep 30, 2019 at 11:19 PM James Meickle
> > ><jmeic...@quantopian.com.invalid> wrote:
> > >
> > >> For what I'm looking for out of a 2.0, as an operator/cluster admin
> > >> (separate from what I'd like to see as a DAG developer), I'd love to
> > >see:
> > >>
> > >> - Combine breaking changes into 2.0, and do as few as possible after
> > >> - A semver policy for 2.0 and onwards. (For instance we got bit hard
> > >by a
> > >> breaking API change in the k8s operator)
> > >> - Regularly scheduled releases (like: "minor every other month, major
> > >every
> > >> other year")
> > >> - A security backport policy
> > >> - Pinned deps for releases
> > >> - A way to get integration/cloud vendor operator updates out-of-tree,
> > >> without having to pull in unrelated Airflow updates
> > >>
> > >> For a lot of people, Airflow is an off-the-shelf app rather than a
> > >library,
> > >> but we don't actually ship or support it anything like most
> > >comparable
> > >> off-the-shelf apps. It makes it much harder to support than other
> > >> applications, unless you're a Python developer yourself.
> > >>
> > >> On Mon, Sep 30, 2019 at 11:18 AM Jarek Potiuk
> > ><jarek.pot...@polidea.com>
> > >> wrote:
> > >>
> > >> > All those are very important and we are going to work on some of
> > >them as
> > >> > well.
> > >> >
> > >> > I think if there are breaking changes, we should rather try to fit
> > >them
> > >> in
> > >> > 2.0 release - at least to the point that they can be base for
> > >extending
> > >> it
> > >> > in later versions in backwards-compatible way (maybe then we should
> > >adopt
> > >> > SemVer officially and follow it).
> > >> >
> > >> > J.
> > >> >
> > >> >
> > >> > On Tue, Sep 24, 2019 at 11:52 PM James Meickle
> > >> > <jmeic...@quantopian.com.invalid> wrote:
> > >> >
> > >> > > My question with that is, how often do we want to do major
> > >version
> > >> > > increments? There's a few API breaking changes I'd love to see,
> > >but
> > >> > > whether to propose them for 2.0 depends on what the wait until
> > >3.0
> > >> looks
> > >> > > like (or whether we'll allow more minor version breakages in the
> > >> future)
> > >> > >
> > >> > > On Tue, Sep 24, 2019, 11:44 Dan Davydov
> > ><ddavy...@twitter.com.invalid>
> > >> > > wrote:
> > >> > >
> > >> > > > I think along with "Improve Webserver Performance" we should
> > >solve
> > >> the
> > >> > > > serialization and task execution isolation problems a little
> > >bit more
> > >> > > > completely since I imagine there could be backwards
> > >compatibility
> > >> > issues.
> > >> > > > e.g. mapping each task JSON to a Docker image or other
> > >serialized
> > >> > > > representation that workers would then consume. See the
> > >attached PDF,
> > >> > > > AIP-24 is a subset of the DAG Definition Serialization work,
> > >but in
> > >> my
> > >> > > > opinion we should still work on DAG Isolation too. My only
> > >concern is
> > >> > > that
> > >> > > > the scope is too big for 2.0.
> > >> > > >
> > >> > > > cc @Sumit Maheshwari <smaheshw...@twitter.com> who is also
> > >looking
> > >> at
> > >> > > > tackling some of these problems.
> > >> > > >
> > >> > > > On Tue, Sep 24, 2019 at 9:47 AM Ash Berlin-Taylor
> > ><a...@apache.org>
> > >> > > wrote:
> > >> > > >
> > >> > > >> I'm also in favour of py-test (and it's what I use for most of
> > >my
> > >> > > >> development) which is why I created
> > >> > > >> https://issues.apache.org/jira/browse/AIRFLOW-4863, but I
> > >don't
> > >> think
> > >> > > >> non-user-facing/impacting changes need to go on the road map.
> > >> > > >>
> > >> > > >> -ash
> > >> > > >>
> > >> > > >> > On 24 Sep 2019, at 13:53, Tomasz Urbaszek <
> > >> > > tomasz.urbas...@polidea.com>
> > >> > > >> wrote:
> > >> > > >> >
> > >> > > >> > I am thinking about proposing migration from nosetest to
> > >pytest
> > >> > > because
> > >> > > >> > it's "more up to date". I have even a POC but a lot of test
> > >fails
> > >> > due
> > >> > > to
> > >> > > >> > probably side effects.
> > >> > > >> >
> > >> > > >> > Best,
> > >> > > >> > Tomek
> > >> > > >> >
> > >> > > >> > On Tue, Sep 24, 2019 at 2:38 PM Ash Berlin-Taylor
> > ><a...@apache.org
> > >> >
> > >> > > >> wrote:
> > >> > > >> >
> > >> > > >> >> That formatted very badly in plain text. The list was:
> > >> > > >> >>
> > >> > > >> >> • Knative Executor (AIP-25, currently draft. Being
> > >worked
> > >> on
> > >> > > by
> > >> > > >> >> Daniel Imberman )
> > >> > > >> >> • Improve Webserver performance (AIP-24, currently
> > >draft.
> > >> > > Being
> > >> > > >> >> worked on by myself, Kaxil Naik and Zhou Fang)
> > >> > > >> >> • Enhanced real-time UI
> > >> > > >> >> • Improve Scheduler performance
> > >> > > >> >> • Extend/finish the API (AIP-13 is part of this, but
> > >not
> > >> > all)
> > >> > > >> >> • Production Docker image + Helm chart
> > >> > > >> >>
> > >> > > >> >>> On 24 Sep 2019, at 13:36, Ash Berlin-Taylor
> > ><a...@apache.org>
> > >> > wrote:
> > >> > > >> >>>
> > >> > > >> >>> Hi everyone,
> > >> > > >> >>>
> > >> > > >> >>> I'd like to start working on a concrete plan to get
> > >Airflow 2.0
> > >> > out,
> > >> > > >> and
> > >> > > >> >> as a result I've started updating
> > >> > > >> >>
> > >https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0
> > >> > > >> >>>
> > >> > > >> >>> In addition to all the tidy up work ("spring cleaning",
> > >finish
> > >> > tidy
> > >> > > up
> > >> > > >> >> after dropping Py2 etc) I'd propose the following 6 high
> > >level
> > >> > items:
> > >> > > >> >>>
> > >> > > >> >>> Knative Executor (AIP-25, currently draft. Being worked on
> > >by
> > >> > Daniel
> > >> > > >> >> Imberman )
> > >> > > >> >>> Improve Webserver performance (AIP-24, currently draft.
> > >Being
> > >> > worked
> > >> > > >> on
> > >> > > >> >> by myself, Kaxil Naik and Zhou Fang)
> > >> > > >> >>> Enhanced real-time UI
> > >> > > >> >>> Improve Scheduler performance
> > >> > > >> >>> Extend/finish the API (AIP-13 is part of this, but not
> > >all)
> > >> > > >> >>> Production Docker image + Helm chart
> > >> > > >> >>> We at Astronomer are committing to work on these in
> > >roughly this
> > >> > > order
> > >> > > >> >> if no one else gets to them first. I also propose that we
> > >create
> > >> > SIGs
> > >> > > >> >> (Special Interest Groups) in slack with weekly/fortnightly
> > >(every
> > >> > 14
> > >> > > >> days)
> > >> > > >> >> "calls"/update sessions. We already have #sig-ui and
> > >> > > >> #sig-dag-serialisation.
> > >> > > >> >>>
> > >> > > >> >>> This roadmap is also not a promise that all of these will
> > >be
> > >> done
> > >> > > >> before
> > >> > > >> >> Airflow 2.0 - we may decide later to push something back to
> > >v2.1
> > >> > etc.
> > >> > > >> >>>
> > >> > > >> >>> Does anyone disagree strongly with these priorities, or
> > >have
> > >> > > anything
> > >> > > >> >> they want to add that you are willing to work on?
> > >> > > >> >>>
> > >> > > >> >>> Thanks,
> > >> > > >> >>> Ash
> > >> > > >> >>
> > >> > > >> >>
> > >> > > >> >
> > >> > > >> > --
> > >> > > >> >
> > >> > > >> > Tomasz Urbaszek
> > >> > > >> > Polidea <https://www.polidea.com/> | Junior Software
> > >Engineer
> > >> > > >> >
> > >> > > >> > M: +48 505 628 493 <+48505628493>
> > >> > > >> > E: tomasz.urbas...@polidea.com
> > ><tomasz.urbasz...@polidea.com>
> > >> > > >> >
> > >> > > >> > Unique Tech
> > >> > > >> > Check out our projects! <https://www.polidea.com/our-work>
> > >> > > >>
> > >> > > >>
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > Jarek Potiuk
> > >> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >> >
> > >> > M: +48 660 796 129 <+48660796129>
> > >> > [image: Polidea] <https://www.polidea.com/>
> > >> >
> > >>
> >
>
>
> --
>
> Chao-Han Tsai
>
