issue fetching master repo

2017-04-19 Thread Boris Tyukin
hey guys, want to submit my first tiny PR and once I fork airflow and clone my repo get this message below: I cannot commit / rebase and I cannot find a way to remove this file. Is it only my who has this issue? git status # On branch master # Changed but not updated: # (use "git add ..." to u

Accessing Presto over SSL from Airflow

2017-04-19 Thread Nicholas Hodgkinson
I'm curious is anyone accessing Presto over SSL? I'm about to enable LDAP authentication on Presto which requires connections with the coordinator to be over SSL and I wonder if anyone has setup Airflow to make that happen. If not I can dig in myself to figure it out, just don't want to dig in if i

Re: [VOTE] Release Airflow 1.8.1 based on Airflow 1.8.1 RC0

2017-04-19 Thread siddharth anand
https://issues.apache.org/jira/browse/AIRFLOW-1121 is merged to fix the webserver pid issue.. thx Kengo! -s On Tue, Apr 18, 2017 at 6:15 PM, Hitesh Shah wrote: > -1. > > Not sure if these have been called out earlier. > > For all the bundled files with different licenses (MIT, BSD, etc), the fu

Re: Too many task instances

2017-04-19 Thread Maxime Beauchemin
You can archive the `job` and `tasks_instance` table, the scheduler won't try to backfill them as their respective DagRuns are not in a `running` state. The scheduler only tries to schedule active DagRuns, and only creates new [active] DagRuns forward from the latest one. Note that the criteria to

Questions regarding externally submitted DAGs

2017-04-19 Thread jeeyoung kim
I am creating a DAG used for ad-hoc analysis and backfill. It has schedule of None and is manually triggered via airflow trigger_dag. I have few questions around it. - I pass parameters to the dag via --conf flag, which is available via dagrun object. I believe that this is the right way to

Re: 1.8.0 Backfill Clarification

2017-04-19 Thread Maxime Beauchemin
Nice! Thanks Bolke for being so quick! Arguably real progress isn't really achieved until it's protected by a unit test, and I'm most guilty for writing tons of untested features... The backfill logic has always been surprisingly intricate and complex, and increasingly so as we've added more feat

Re: 1.8.0 Backfill Clarification

2017-04-19 Thread Chris Fei
Thank you for the clarification, Max, and thanks for the quick turnaround, Bolke. I can confirm that your fix addresses my issue. Chris On Wed, Apr 19, 2017, at 07:51 AM, Bolke de Bruin wrote: > PR is out: https://github.com/apache/incubator-airflow/pull/2247 > > Includes tests. > > - Bolke >

Too many task instances

2017-04-19 Thread Yongjun Park
Hi folks. I have a question about task instances. Is it possible to delete old task instances that have run successfully? Isn't it trying to backfill missing tasks? I have about 1,500 dags and am getting more dags. There're about 300 thousand of task instances currently. 10,000 tasks instances a

Re: 1.8.0 Backfill Clarification

2017-04-19 Thread Bolke de Bruin
PR is out: https://github.com/apache/incubator-airflow/pull/2247 Includes tests. - Bolke > On 19 Apr 2017, at 05:33, Bolke de Bruin wrote: > > Agreed. This is a bug and imho a blocker for 1.8.1. > > My bad: re-implementation and lack of sufficient unit tests is what is > causing this. > >

SubdagOperator work with CeleryExecutor

2017-04-19 Thread Devjyoti Patra
I am not able to make SubdagOperator work with CeleryExecutor. The following code works well on my local setup (with LocalExecutor), but on production, when clicking on "Zoom into Sub Dag", Airflow complains that dag with the name "parent_name.child_name". is not found. Please tell me what did I m

Re: Welcome @saguziel as a committer and PMC member!

2017-04-19 Thread Sumit Maheshwari
Welcome Alex.. On Mon, Apr 17, 2017 at 10:33 PM, siddharth anand wrote: > Welcome Alex! > > On Fri, Apr 14, 2017 at 8:23 AM, Chris Riccomini > wrote: > > > Congrats, Alex! Welcome. :) > > > > On Thu, Apr 13, 2017 at 7:06 PM, Dan Davydov > invalid > > > wrote: > > > > > Alex (@saguziel - Air