Scheduler not honouring non_pooled_task_slot_count config

2018-07-09 Thread ramandumcs
We are using airflow version 1.9 with celery executor. And we are observing that Airflow Scheduler is not honouring the "non_pooled_task_slot_count" config. We are using default setting which is set to 128. But we could schedule and run >128 tasks concurrently. >From code it seems that

Re: [VOTE] Airflow 1.10.0rc1

2018-07-09 Thread Bolke de Bruin
Kaxil, Please vote with binding / non-binding. This is important for the IPMC vote that won’t know all committers. B. Verstuurd vanaf mijn iPad > Op 10 jul. 2018 om 03:22 heeft Naik Kaxil het volgende > geschreven: > > +1 > > Regards, > Kaxil > > On 10/07/2018, 02:09, "Kengo Seki"

Re: Apache Airflow 1.10.0b3

2018-07-09 Thread Taylor Edmiston
We discussed the splitting out of hooks & operators from core internally at Astronomer again today. I should be able to allocate some time to splitting out hooks & operators for the purpose of speeding up the CI. A faster CI on core would be hugely beneficial for all contributors. I created a

Re: [VOTE] Airflow 1.10.0rc1

2018-07-09 Thread Naik Kaxil
+1 Regards, Kaxil On 10/07/2018, 02:09, "Kengo Seki" wrote: +1 (non-binding) - Checked signatures and checksums - Confirmed the source distribution corresponds to v1-10-test's HEAD (f1083cb). - Installed the binary distribution with cryptography package ran some

Re: [VOTE] Airflow 1.10.0rc1

2018-07-09 Thread Kengo Seki
+1 (non-binding) - Checked signatures and checksums - Confirmed the source distribution corresponds to v1-10-test's HEAD (f1083cb). - Installed the binary distribution with cryptography package ran some example dags with no error - Checked version info via WebUI and confirmed RBAC worked Kengo

Timezone support on Web UI

2018-07-09 Thread Manu Zhang
Hi all, We've been using Airflow for job scheduling for quite a while. It's been great except that everything is in UTC time (We are from China and in GMT+8) . We are pretty excited when timezone support is added on latest master and try upgrading. Nonetheless, as in the doc

Airflow's JS code (and dependencies) manageable via npm and webpack

2018-07-09 Thread Verdan Mahmood
​Hey Guys, ​ In an effort to simplify the JS dependencies of Airflow ​​ , ​I've introduce ​d​ npm and webpack for the package management. For now, it only implements this in the www_rbac version of the web server. ​ Pull Request: https://github.com/apache/incubator-airflow/pull/3572 The

Re: Timezone support on Web UI

2018-07-09 Thread Bolke de Bruin
Of course just didn’t get to it yet. PRs are also welcome btw! Verstuurd vanaf mijn iPad > Op 9 jul. 2018 om 08:05 heeft Manu Zhang het > volgende geschreven: > > Hi all, > > We've been using Airflow for job scheduling for quite a while. It's been > great except that everything is in UTC

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

2018-07-09 Thread Ash Berlin-Taylor
Great! Thanks for doing this. I've left some review comments on your PR. -ash > On 9 Jul 2018, at 11:45, Verdan Mahmood wrote: > > ​Hey Guys, ​ > > In an effort to simplify the JS dependencies of Airflow > ​​ > , > ​I've > introduce > ​d​ > npm and webpack for the package management. For now,

PR for refactoring Airflow SLAs

2018-07-09 Thread James Meickle
Hi folks, Based on my earlier email to the list, I have submitted a PR that splits `sla=` into three independent SLA parameters, as well as heavily restructuring other parts of the SLA feature: https://github.com/apache/incubator-airflow/pull/3584 This is my first Airflow PR and I'm still