[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15300086#comment-15300086
]
Jeremiah Lowin commented on AIRFLOW-52:
---
Should this be closed?
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: release
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> -2. The following kind of patterns fail:-
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> -3. Airflow losing queued tasks (see linked issue)-
> -4. Airflow putting dags in a stuck state (AIRFLOW-92)-
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291680#comment-15291680
]
ASF subversion and git services commented on AIRFLOW-52:
Commit 1d0d8681ddf89fabd5f3521d0846a758e47962b3 in incubator-airflow's branch
refs/heads/airbnb_rb1.7.1_3 from [~aoen]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=1d0d868 ]
[AIRFLOW-52] 1.7.1 version bump and changelog
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: release
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> -2. The following kind of patterns fail:-
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> -3. Airflow losing queued tasks (see linked issue)-
> -4. Airflow putting dags in a stuck state (AIRFLOW-92)-
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291677#comment-15291677
]
ASF subversion and git services commented on AIRFLOW-52:
Commit 0b3d101ff5aa430ff4e9e709523192fc6e6c6dee in incubator-airflow's branch
refs/heads/master from [~aoen]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0b3d101 ]
[AIRFLOW-52] 1.7.1 version bump and changelog
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: release
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> -2. The following kind of patterns fail:-
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> -3. Airflow losing queued tasks (see linked issue)-
> -4. Airflow putting dags in a stuck state (AIRFLOW-92)-
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291669#comment-15291669
]
Chris Riccomini commented on AIRFLOW-52:
+1 on https://github.com/apache/incubator-airflow/pull/1522
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: release
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> -2. The following kind of patterns fail:-
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> -3. Airflow losing queued tasks (see linked issue)-
> -4. Airflow putting dags in a stuck state (AIRFLOW-92)-
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287087#comment-15287087
]
Chris Riccomini commented on AIRFLOW-52:
Awesome, thanks for the update!
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: release
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> -2. The following kind of patterns fail:-
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> -3. Airflow losing queued tasks (see linked issue)-
> -4. Airflow putting dags in a stuck state (AIRFLOW-92)-
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15287075#comment-15287075
]
Dan Davydov commented on AIRFLOW-52:
Update:
I released to production yesterday and so far things have been stable. We are
on track to push the 1.7.1 tag on Thursday.
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: release
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> -2. The following kind of patterns fail:-
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> -3. Airflow losing queued tasks (see linked issue)-
> -4. Airflow putting dags in a stuck state (AIRFLOW-92)-
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15282915#comment-15282915
]
Dan Davydov commented on AIRFLOW-52:
Another update:
Sid found another blocker a few days ago and got a fix in. I have a new release
baking in staging today (airbnb_1.7.1rc6, keeping airbnb_ as the prefix for
this release to be consistent but we will drop it future releases). Will likely
release to our production on Monday and then push the official release out next
Thursday (May 19) if things are stable.
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: release
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> -2. The following kind of patterns fail:-
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> -3. Airflow losing queued tasks (see linked issue)-
> -4. Airflow putting dags in a stuck state (AIRFLOW-92)-
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15279211#comment-15279211
]
Chris Riccomini commented on AIRFLOW-52:
Thanks for the update!
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: release
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> -2. The following kind of patterns fail:-
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> -3. Airflow losing queued tasks (see linked issue)-
> -4. Airflow putting dags in a stuck state (AIRFLOW-92)-
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15279187#comment-15279187
]
Dan Davydov commented on AIRFLOW-52:
Update: I'm going to let the release bake in our staging cluster tomorrow, and
release on Thursday. If this is stable through Monday (May 16) in production I
will push the 1.7.1 tag.
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: release
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> -2. The following kind of patterns fail:-
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> -3. Airflow losing queued tasks (see linked issue)-
> -4. Airflow putting dags in a stuck state (AIRFLOW-92)-
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15278503#comment-15278503
]
Dan Davydov commented on AIRFLOW-52:
Roger.
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: release
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> -2. The following kind of patterns fail:-
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> -3. Airflow losing queued tasks (see linked issue)-
> 4. Airflow putting dags in a stuck state (AIRFLOW-92)
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15278025#comment-15278025
]
Bolke de Bruin commented on AIRFLOW-52:
---
[~aoen] Linked an additional issue that I also consider a blocker
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: ease
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> -2. The following kind of patterns fail:-
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> -3. Airflow losing queued tasks (see linked issue)-
> 4. Airflow putting dags in a stuck state (AIRFLOW-92)
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15277515#comment-15277515
]
Dan Davydov commented on AIRFLOW-52:
Update: I'm almost ready to stage the new version, right now I'm hitting one
last newly introduced exception (same issue as the previous exception of
calling add_tasks twice) and I've reached out to the owner to help fix it:
AirflowException: Tried to create relationships between tasks that don't have
DAGs yet. Set the DAG for at least one task and try again
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: ease
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> -2. The following kind of patterns fail:-
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> -3. Airflow losing queued tasks (see linked issue)-
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15277355#comment-15277355
]
ASF subversion and git services commented on AIRFLOW-52:
Commit ff3a8552501ac88821f3ce641471cd396ca93d6b in incubator-airflow's branch
refs/heads/airbnb_rb1.7.1_3 from jlowin
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=ff3a855 ]
AIRFLOW-52 Warn about overwriting tasks in a DAG
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: ease
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> 2. The following kind of patterns fail:
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> 3. Airflow losing queued tasks (see linked issue)
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15277353#comment-15277353
]
ASF subversion and git services commented on AIRFLOW-52:
Commit 0ed36a14d8047bbfac749c73a81581f293b43af6 in incubator-airflow's branch
refs/heads/airbnb_rb1.7.1_3 from [~jlowin]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0ed36a1 ]
[AIRFLOW-52] Fix bottlenecks when working with many tasks
Dag hash function tried (and failed) to hash the list of tasks, then fell back
on repr-ing the list, which took forever. Instead, hash
tuple(task_dict.keys()). In addition this replaces two slow list comprehensions
with much faster hash lookups (using the new task_dict).
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: ease
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> 2. The following kind of patterns fail:
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> 3. Airflow losing queued tasks (see linked issue)
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15277241#comment-15277241
]
ASF subversion and git services commented on AIRFLOW-52:
Commit 385add2bf3eb7cd6103cfec0e5516234eeb72443 in incubator-airflow's branch
refs/heads/master from jlowin
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=385add2 ]
AIRFLOW-52 Warn about overwriting tasks in a DAG
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: ease
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> -1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)-
> 2. The following kind of patterns fail:
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> 3. Airflow losing queued tasks (see linked issue)
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15276748#comment-15276748
]
Bolke de Bruin commented on AIRFLOW-52:
---
[~aoen] I think [~criccomini] refers to:
https://github.com/apache/incubator-airflow/pull/1378 (as is also linked in
AIRFLOW-56)
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: ease
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> 1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)
> 2. The following kind of patterns fail:
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds.
> 3. Airflow losing queued tasks (see linked issue)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15276659#comment-15276659
]
Dan Davydov commented on AIRFLOW-52:
This one right https://github.com/apache/incubator-airflow/pull/1473 ? In this
case yes.
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: ease
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> 1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)
> 2. The following kind of patterns fail:
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds.
> 3. Airflow losing queued tasks (see linked issue)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15276637#comment-15276637
]
Chris Riccomini commented on AIRFLOW-52:
[~aoen], are you planning to burn in [~jlowin]'s fix before release?
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: ease
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> 1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)
> 2. The following kind of patterns fail:
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds.
> 3. Airflow losing queued tasks (see linked issue)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275198#comment-15275198
]
Bolke de Bruin commented on AIRFLOW-52:
---
[~jlowin] hash fix has been committed.
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: ease
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> 1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)
> 2. The following kind of patterns fail:
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds.
> 3. Airflow losing queued tasks (see linked issue)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274161#comment-15274161
]
Chris Riccomini commented on AIRFLOW-52:
Note: Looks like we can do our old release style without blocking on the Apache
release criteria. So, we're good to go.
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: ease
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are three issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> 1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)
> 2. The following kind of patterns fail:
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds.
> 3. Airflow loosing queued tasks (see linked issue)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273778#comment-15273778
]
Bolke de Bruin commented on AIRFLOW-52:
---
Please note that we probably have another blocker. Airflow can "loose" queued
tasks, ie. tasks in queued state that will never be picked up. I have
reproduced this and will report on it shortly.
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: ease
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are two issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> 1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)
> 2. The following kind of patterns fail:
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[
https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273389#comment-15273389
]
Dan Davydov commented on AIRFLOW-52:
Updated the ticket with details on the blockers.
> Release airflow 1.7.1
> -
>
> Key: AIRFLOW-52
> URL: https://issues.apache.org/jira/browse/AIRFLOW-52
> Project: Apache Airflow
> Issue Type: Task
> Components: ease
>Reporter: Dan Davydov
>Assignee: Dan Davydov
> Labels: release
>
> Release the airflow 1.7.1 tag.
> Current status:
> There are two issues blocking this release caused by this commit:
> https://github.com/apache/incubator-airflow/commit/fb0c5775cda4f84c07d8d5c0e6277fc387c172e6
> 1. DAGs with a lot of tasks take much longer to parse (~25x slowdown)
> 2. The following kind of patterns fail:
> {code}
> email.set_upstream(dag.roots)
> dag.add_task(email)
> {code}
> This is because set_upstream now calls add_task and a task can't be added
> more than once.
> I'm working with the owner of the commit to resolve these issues.
> The way to catch (1) in the future is an integration test that asserts a
> given non-trivial DAG parses under X seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[ https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273239#comment-15273239 ] Chris Riccomini commented on AIRFLOW-52: Posted question: http://mail-archives.apache.org/mod_mbox/incubator-general/201605.mbox/%3CCABYbY7eFgR0ChxaQt6KGNsKT2bzMCRBySi4BS_Mjp_zpdEtZmw%40mail.gmail.com%3E > Release airflow 1.7.1 > - > > Key: AIRFLOW-52 > URL: https://issues.apache.org/jira/browse/AIRFLOW-52 > Project: Apache Airflow > Issue Type: Task > Components: ease >Reporter: Dan Davydov >Assignee: Dan Davydov > Labels: release > > Release the airflow 1.7.1 tag. > Current status: > I found the commit that caused the slowdown in 1.7.1rc3 yesterday (some DAGs > were timing out when they got parsed via collect_dags). > Working with the owner of the commit to decide whether it's better to roll > forward with a fix or revert the commit for now. > The way to catch these issues in the future is an integration test that > asserts a given non-trivial DAG parses under X seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[ https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273085#comment-15273085 ] Chris Riccomini commented on AIRFLOW-52: [~jghoman], the "release" is really just a git tag and a pypi push that's run from someone's laptop right now. So, IMO, it's barely even a release at all. I'll ping general@incubator, though. > Release airflow 1.7.1 > - > > Key: AIRFLOW-52 > URL: https://issues.apache.org/jira/browse/AIRFLOW-52 > Project: Apache Airflow > Issue Type: Task > Components: ease >Reporter: Dan Davydov >Assignee: Dan Davydov > Labels: release > > Release the airflow 1.7.1 tag. > Current status: > I found the commit that caused the slowdown in 1.7.1rc3 yesterday (some DAGs > were timing out when they got parsed via collect_dags). > Working with the owner of the commit to decide whether it's better to roll > forward with a fix or revert the commit for now. > The way to catch these issues in the future is an integration test that > asserts a given non-trivial DAG parses under X seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[ https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273026#comment-15273026 ] Jakob Homan commented on AIRFLOW-52: So, the release will be coordinated (and built from) github and github resources? I think Chris is right, but I'd be concerned about running the vote or whatever process was being used for the release on ASF resources. I'd want to avoid the appearance of trying to make an end-run around asf procedure (which I know isn't the case). I'm honestly not sure though; may be worth pinging general@incubator. > Release airflow 1.7.1 > - > > Key: AIRFLOW-52 > URL: https://issues.apache.org/jira/browse/AIRFLOW-52 > Project: Apache Airflow > Issue Type: Task > Components: ease >Reporter: Dan Davydov >Assignee: Dan Davydov > Labels: release > > Release the airflow 1.7.1 tag. > Current status: > I found the commit that caused the slowdown in 1.7.1rc3 yesterday (some DAGs > were timing out when they got parsed via collect_dags). > Working with the owner of the commit to decide whether it's better to roll > forward with a fix or revert the commit for now. > The way to catch these issues in the future is an integration test that > asserts a given non-trivial DAG parses under X seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[ https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272993#comment-15272993 ] Chris Riccomini commented on AIRFLOW-52: Awesome. We are currently working on fixing some issues that are going to block our release (e.g. forcing license headers, removing High Charts, etc). It's good to know we can release short-term while that work is in flight. > Release airflow 1.7.1 > - > > Key: AIRFLOW-52 > URL: https://issues.apache.org/jira/browse/AIRFLOW-52 > Project: Apache Airflow > Issue Type: Task > Components: ease >Reporter: Dan Davydov >Assignee: Dan Davydov > Labels: release > > Release the airflow 1.7.1 tag. > Current status: > I found the commit that caused the slowdown in 1.7.1rc3 yesterday (some DAGs > were timing out when they got parsed via collect_dags). > Working with the owner of the commit to decide whether it's better to roll > forward with a fix or revert the commit for now. > The way to catch these issues in the future is an integration test that > asserts a given non-trivial DAG parses under X seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[ https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272975#comment-15272975 ] Chris Nauroth commented on AIRFLOW-52: -- AFAIK, entrance into the incubator does not require that you halt any release from the existing airbnb GitHub repo, using whatever process you've always used there. It's definitely good to show progress towards a first Apache release, but that's often easier to do decoupled from any time pressures about trying to ship a particular feature or bug fix. That way, you can focus on things like the licensing requirements, which are tricky for the first release in their own right. > Release airflow 1.7.1 > - > > Key: AIRFLOW-52 > URL: https://issues.apache.org/jira/browse/AIRFLOW-52 > Project: Apache Airflow > Issue Type: Task > Components: ease >Reporter: Dan Davydov >Assignee: Dan Davydov > Labels: release > > Release the airflow 1.7.1 tag. > Current status: > I found the commit that caused the slowdown in 1.7.1rc3 yesterday (some DAGs > were timing out when they got parsed via collect_dags). > Working with the owner of the commit to decide whether it's better to roll > forward with a fix or revert the commit for now. > The way to catch these issues in the future is an integration test that > asserts a given non-trivial DAG parses under X seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[ https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272964#comment-15272964 ] Chris Riccomini commented on AIRFLOW-52: [~jghoman]/[~hitesh]/[~cnauroth] what's the best way to handle this? We basically had an RC in flight for like a month before Apache. > Release airflow 1.7.1 > - > > Key: AIRFLOW-52 > URL: https://issues.apache.org/jira/browse/AIRFLOW-52 > Project: Apache Airflow > Issue Type: Task > Components: ease >Reporter: Dan Davydov >Assignee: Dan Davydov > Labels: release > > Release the airflow 1.7.1 tag. > Current status: > I found the commit that caused the slowdown in 1.7.1rc3 yesterday (some DAGs > were timing out when they got parsed via collect_dags). > Working with the owner of the commit to decide whether it's better to roll > forward with a fix or revert the commit for now. > The way to catch these issues in the future is an integration test that > asserts a given non-trivial DAG parses under X seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[ https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272919#comment-15272919 ] Bolke de Bruin commented on AIRFLOW-52: --- [~criccomini] Can we do an 'airbnb' release for now then? Ie. not use the apache brand? Cause this release is long overdue and 1.7.0 is a bit buggy. Chasing licenses will cost at least another week or even more (with broken PRs to add) > Release airflow 1.7.1 > - > > Key: AIRFLOW-52 > URL: https://issues.apache.org/jira/browse/AIRFLOW-52 > Project: Apache Airflow > Issue Type: Task > Components: ease >Reporter: Dan Davydov >Assignee: Dan Davydov > Labels: release > > Release the airflow 1.7.1 tag. > Current status: > I found the commit that caused the slowdown in 1.7.1rc3 yesterday (some DAGs > were timing out when they got parsed via collect_dags). > Working with the owner of the commit to decide whether it's better to roll > forward with a fix or revert the commit for now. > The way to catch these issues in the future is an integration test that > asserts a given non-trivial DAG parses under X seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[ https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272778#comment-15272778 ] Chris Riccomini commented on AIRFLOW-52: No worries. > Release airflow 1.7.1 > - > > Key: AIRFLOW-52 > URL: https://issues.apache.org/jira/browse/AIRFLOW-52 > Project: Apache Airflow > Issue Type: Task > Components: ease >Reporter: Dan Davydov >Assignee: Dan Davydov > Labels: release > > Release the airflow 1.7.1 tag. > Current status: > I found the commit that caused the slowdown in 1.7.1rc3 yesterday (some DAGs > were timing out when they got parsed via collect_dags). > Working with the owner of the commit to decide whether it's better to roll > forward with a fix or revert the commit for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[ https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272776#comment-15272776 ] Dan Davydov commented on AIRFLOW-52: Gotcha I will coordinate with you on the status of those items once the fix for the bad commit lands. Thanks for letting me know! > Release airflow 1.7.1 > - > > Key: AIRFLOW-52 > URL: https://issues.apache.org/jira/browse/AIRFLOW-52 > Project: Apache Airflow > Issue Type: Task > Components: ease >Reporter: Dan Davydov >Assignee: Dan Davydov > Labels: release > > Release the airflow 1.7.1 tag. > Current status: > I found the commit that caused the slowdown in 1.7.1rc3 yesterday (some DAGs > were timing out when they got parsed via collect_dags). > Working with the owner of the commit to decide whether it's better to roll > forward with a fix or revert the commit for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-52) Release airflow 1.7.1
[ https://issues.apache.org/jira/browse/AIRFLOW-52?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272765#comment-15272765 ] Chris Riccomini commented on AIRFLOW-52: Before we can do an Apache release, I think we need to wrap up our licenses, and add a DISCLAIMER and NOTICE file, and what not: http://incubator.apache.org/guides/releasemanagement.html#check-list It's unclear to me if having High Charts in the source dist will be a problem or not. > Release airflow 1.7.1 > - > > Key: AIRFLOW-52 > URL: https://issues.apache.org/jira/browse/AIRFLOW-52 > Project: Apache Airflow > Issue Type: Task > Components: ease >Reporter: Dan Davydov >Assignee: Dan Davydov > Labels: release > > Release the airflow 1.7.1 tag. > Current status: > I found the regression yesterday that caused some DAGs to time out when they > got parsed (via the collect_dags call). > Working with the owner of the commit to decide whether it's better to roll > forward with a fix or revert the commit for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
