Re: 1.10.1 Release?

2018-11-05 Thread Bolke de Bruin
The fix is in master and should work across all DST changes. It will be included in 1.10.1. B. Sent from my iPhone > On 5 Nov 2018, at 19:54, Dave Fisher wrote: > > > >> On 2018/10/28 00:09:05, Bolke de Bruin wrote: >> I wonder how to treat this: >> >> This is what I think happens (nee

Re: 1.10.1 Release?

2018-11-05 Thread Dave Fisher
On 2018/10/28 00:09:05, Bolke de Bruin wrote: > I wonder how to treat this: > > This is what I think happens (need to verify more, but I am pretty sure) the > specified DAG should run every 5 minutes. At DST change (3AM -> 2AM) FYI - In the US the DST change is 2AM -> 1AM. Yes, TZ is hard s

Re: REST API roadmap/plan?

2018-11-05 Thread Verdan Mahmood
Hi Mathew, As Fokko mentioned, we don't have any roadmap for the REST API at the moment, but we can definitely start planning this out, gathering the list of APIs, review them together and making a roadmap. I'd be happy to help with this :) Few things: - I started out with a list of possible APIs

Re: A Naive Multi-Scheduler Architecture Experiment of Airflow

2018-11-05 Thread Deng Xiaodong
Thanks Devjyoti for your reply. To elaborate based on your inputs: - *When to add one more shard*: We have designed some metrics, like "how long the scheduler instance takes to parse & schedule all DAGs (in the subdir it’s taking care of)". When the metric is higher than a given threshold for

Re: REST API roadmap/plan?

2018-11-05 Thread Driesprong, Fokko
Thanks Matthew, We're working on getting the experimental API. This API should be used for both the GUI and can be used for external systems to interface with Airflow (triggering dags for example). Verdan was working on this, but for now, this process is a bit stuck again. It turns out that data e

Feed Back on Remote Dag Fetching Proposal

2018-11-05 Thread Ian Davison
Hey Everyone, I just created a first draft of an Airflow Improvement Proposal for Remote Dag Fetching https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+DagFetcher The initial PR is located here: https://github.com/apache/incubator-airflow/pull/3138 I tried to go through and encapsulate