[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-05-09 Thread Bolke de Bruin (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15277096#comment-15277096 ] Bolke de Bruin commented on AIRFLOW-14: --- Agreed :). Just for the sake of this side n

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-05-09 Thread Jeremiah Lowin (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15276922#comment-15276922 ] Jeremiah Lowin commented on AIRFLOW-14: --- Here's how I'd characterize the current sys

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-05-09 Thread Chris Riccomini (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15276805#comment-15276805 ] Chris Riccomini commented on AIRFLOW-14: Yea, I agree with this. I guess the two t

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-05-09 Thread Bolke de Bruin (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15276785#comment-15276785 ] Bolke de Bruin commented on AIRFLOW-14: --- Now the scheduler is, in case of using the

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-05-09 Thread Chris Riccomini (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15276669#comment-15276669 ] Chris Riccomini commented on AIRFLOW-14: {quote} Now the scheduler and the executo

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-05-08 Thread Bolke de Bruin (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275716#comment-15275716 ] Bolke de Bruin commented on AIRFLOW-14: --- While I understand that for being truly dis

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-05-08 Thread Bolke de Bruin (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275711#comment-15275711 ] Bolke de Bruin commented on AIRFLOW-14: --- Please use with_for_update() for those oper

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-05-06 Thread Jeremiah Lowin (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274796#comment-15274796 ] Jeremiah Lowin commented on AIRFLOW-14: --- Good question. Scheduling and evaluating po

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-05-06 Thread Paul Yang (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274760#comment-15274760 ] Paul Yang commented on AIRFLOW-14: -- To clarify, would different DagRuns be evaluated in t

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-04-28 Thread Chris Riccomini (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262943#comment-15262943 ] Chris Riccomini commented on AIRFLOW-14: If there are two DRJs running simultaneou

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-04-28 Thread Jeremiah Lowin (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262693#comment-15262693 ] Jeremiah Lowin commented on AIRFLOW-14: --- The good news is that lock_id is already se

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-04-28 Thread Jeremiah Lowin (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262359#comment-15262359 ] Jeremiah Lowin commented on AIRFLOW-14: --- Yes, that situation could arise... I'm not

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-04-28 Thread Chris Riccomini (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262371#comment-15262371 ] Chris Riccomini commented on AIRFLOW-14: One way to do this is make the lock claim

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-04-28 Thread Jeremiah Lowin (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262357#comment-15262357 ] Jeremiah Lowin commented on AIRFLOW-14: --- Yes, that situation could arise... I'm not

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-04-28 Thread Chris Riccomini (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262337#comment-15262337 ] Chris Riccomini commented on AIRFLOW-14: > DRJ needs to make sure to refresh from

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-04-28 Thread Jeremiah Lowin (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262324#comment-15262324 ] Jeremiah Lowin commented on AIRFLOW-14: --- DagRuns are primary keyed by (dag_id, execu

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-04-28 Thread Chris Riccomini (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262315#comment-15262315 ] Chris Riccomini commented on AIRFLOW-14: How does a DRJ grab a Dag to execute? It

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-04-28 Thread Jeremiah Lowin (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262304#comment-15262304 ] Jeremiah Lowin commented on AIRFLOW-14: --- That is exactly correct. Multiple DRJ are n

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-04-28 Thread Jeremiah Lowin (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262303#comment-15262303 ] Jeremiah Lowin commented on AIRFLOW-14: --- Excellent question. I've tried to address t

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-04-28 Thread Chris Riccomini (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262300#comment-15262300 ] Chris Riccomini commented on AIRFLOW-14: If I'm not mistaken, I think this work sh

[jira] [Commented] (AIRFLOW-14) DagRun Refactor (Scheduler 2.0)

2016-04-28 Thread Chris Riccomini (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262235#comment-15262235 ] Chris Riccomini commented on AIRFLOW-14: Overall, major +1 on this. One thing tha