Hello Ludovic,
Thank you so much. Things are a lot clearer now. Looks like there is a way to
make this work for us.
Have a nice day,
Arvind
On 16/01/17, 22:30, "Ludovic Claude" wrote:
Hello Arvind,
DAGs in Airflow are simple Python objects that only need to be registered
in
hello,
I would like to prevent a DAG from scheduling / running concurrently
whether by scheduler or if triggered externally (via airflow trigger_dag or
UI).
I tried to set max_active_runs to 1 but i did not work for externally
triggered case - airflow would still go ahead and execute multiple dag
You could use a pool, if you are fine with the “second” dag run being run later.
> On 17 Jan 2017, at 16:18, Boris Tyukin wrote:
>
> hello,
>
> I would like to prevent a DAG from scheduling / running concurrently
> whether by scheduler or if triggered externally (via airflow trigger_dag or
> U
interesting but I have 200 DAGs like that - I generate them
programmatically. Does it mean I need a different pool for each one of
them? Can I define pools not from UI? that would be painful :)
On Tue, Jan 17, 2017 at 10:22 AM, Bolke de Bruin wrote:
> You could use a pool, if you are fine with t
Well yes I guess so. You can actually create pools from a DAG definition as
well (I personally don’t like that for operational security reasons), but yes
this works. So could verify if the pool exists for a certain DAG and create it
if it doesn’t.
Bolke
> On 17 Jan 2017, at 17:26, Boris Tyukin
Without looking at the latest code to confirm what I'm about to write,
`max_active_runs` really only prevents the scheduler from creating new
active DAG runs. For `max_active_runs` to apply to externally triggered
runs, we'd need to introduce handling of a new status of `scheduled` to DAG
runs. The
interesting, thanks Bolke for giving me this idea!
On Tue, Jan 17, 2017 at 11:29 AM, Bolke de Bruin wrote:
> Well yes I guess so. You can actually create pools from a DAG definition
> as well (I personally don’t like that for operational security reasons),
> but yes this works. So could verify i
Bolke, I couldn't thank you enough for driving the release process!
I'll coordinate with the Airbnb team around impersonation/CGROUPs and on
making sure we put this release in our staging ASAP. We have our employee
conference this week so things are slower, but we'll be back at full speed
Friday.
Would be good to cherrypick Arthur's fix into here if possible:
https://github.com/apache/incubator-airflow/pull/1973/files (commit
43bf89d)
The impersonation stuff should be wrapping up shortly pending Bolke's
comments.
Also agreed with Max on the thanks. Thanks Alex too for the change log!
On
Arthur's change is already in!
B.
Sent from my iPhone
> On 17 Jan 2017, at 22:20, Dan Davydov wrote:
>
> Would be good to cherrypick Arthur's fix into here if possible:
> https://github.com/apache/incubator-airflow/pull/1973/files (commit
> 43bf89d)
>
> The impersonation stuff should be wrap
So it is, my bad. Bad skills with ctrl-f :).
On Tue, Jan 17, 2017 at 3:31 PM, Bolke de Bruin wrote:
> Arthur's change is already in!
>
> B.
>
> Sent from my iPhone
>
> > On 17 Jan 2017, at 22:20, Dan Davydov
> wrote:
> >
> > Would be good to cherrypick Arthur's fix into here if possible:
> > ht
Alex, for the CHANGELOG.md, I've been using `github-changes`, a js app that
make changelog generation flexible and easy.
https://www.npmjs.com/package/github-changes
Command looks something like:
`github-changes -o apache -r incubator-airflow --token
--between-tags 1.7.2...1.8.0beta` (tags may d
12 matches
Mail list logo
