[jira] [Updated] (AIRFLOW-511) DagRun Failure: Retry, Email, Callbacks
[ https://issues.apache.org/jira/browse/AIRFLOW-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Froetscher updated AIRFLOW-511: --- Description: Are there any plans to have retry, email, or callbacks on failure of a DAG run? Would you guys be open to someone implementing that? Right now particularly with dagrun_timeout, there is not much insight that the dag actually stopped. Pseudocode: https://github.com/apache/incubator-airflow/compare/master...rfroetscher:dagrun_failure was: Are there any plans to have retry, email, or callbacks on failure of a DAG run? Would you guys be open to someone implementing that? Right now particularly with dagrun_timeout, there is not much insight that the dag actually stopped. Psuedocode: https://github.com/apache/incubator-airflow/compare/master...rfroetscher:dagrun_failure > DagRun Failure: Retry, Email, Callbacks > --- > > Key: AIRFLOW-511 > URL: https://issues.apache.org/jira/browse/AIRFLOW-511 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rob Froetscher >Priority: Minor > > Are there any plans to have retry, email, or callbacks on failure of a DAG > run? Would you guys be open to someone implementing that? Right now > particularly with dagrun_timeout, there is not much insight that the dag > actually stopped. > Pseudocode: > https://github.com/apache/incubator-airflow/compare/master...rfroetscher:dagrun_failure -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AIRFLOW-511) DagRun Failure Retry, Email, Callbacks
[ https://issues.apache.org/jira/browse/AIRFLOW-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Froetscher updated AIRFLOW-511: --- Description: Are there any plans to have retry, email, or callbacks on failure of a DAG run? Would you guys be open to someone implementing that? Right now particularly with dagrun_timeout, there is not much insight that the dag actually stopped. (was: Are there any plans to have retry, email, or callbacks on failure of a DAG run? Would you guys be open to someone implementing that. Right now particularly with dagrun_timeout, there is not much insight that the dag actually stopped.) > DagRun Failure Retry, Email, Callbacks > -- > > Key: AIRFLOW-511 > URL: https://issues.apache.org/jira/browse/AIRFLOW-511 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rob Froetscher >Priority: Minor > > Are there any plans to have retry, email, or callbacks on failure of a DAG > run? Would you guys be open to someone implementing that? Right now > particularly with dagrun_timeout, there is not much insight that the dag > actually stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (AIRFLOW-511) DagRun Failure: Retry, Email, Callbacks
[ https://issues.apache.org/jira/browse/AIRFLOW-511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Froetscher updated AIRFLOW-511: --- Summary: DagRun Failure: Retry, Email, Callbacks (was: DagRun Failure Retry, Email, Callbacks) > DagRun Failure: Retry, Email, Callbacks > --- > > Key: AIRFLOW-511 > URL: https://issues.apache.org/jira/browse/AIRFLOW-511 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rob Froetscher >Priority: Minor > > Are there any plans to have retry, email, or callbacks on failure of a DAG > run? Would you guys be open to someone implementing that? Right now > particularly with dagrun_timeout, there is not much insight that the dag > actually stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-511) DagRun Failure Retry, Email, Callbacks
[ https://issues.apache.org/jira/browse/AIRFLOW-511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15494568#comment-15494568 ] Rob Froetscher commented on AIRFLOW-511: [~criccomini] any thoughts on this? > DagRun Failure Retry, Email, Callbacks > -- > > Key: AIRFLOW-511 > URL: https://issues.apache.org/jira/browse/AIRFLOW-511 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rob Froetscher >Priority: Minor > > Are there any plans to have retry, email, or callbacks on failure of a DAG > run? Would you guys be open to someone implementing that. Right now > particularly with dagrun_timeout, there is not much insight that the dag > actually stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-346) ECS Hook, Operator, Sensor
Rob Froetscher created AIRFLOW-346: -- Summary: ECS Hook, Operator, Sensor Key: AIRFLOW-346 URL: https://issues.apache.org/jira/browse/AIRFLOW-346 Project: Apache Airflow Issue Type: New Feature Reporter: Rob Froetscher Assignee: Rob Froetscher Priority: Minor It would be good to have an ECS operator that descends from the AWS hook. ECS hook, descending from the AWS hook that allows general interaction with ECS ECS operator to submit a new task definition ECS operator to kick off a task ECS operator to update a service with a new task definition ECS sensor to check on the status of a task There is a sort of half done one here https://github.com/apache/incubator-airflow/pull/652. But the author says they have abandoned it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-303) EMR Operator job_flow_overrides allow for templating
[ https://issues.apache.org/jira/browse/AIRFLOW-303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358097#comment-15358097 ] Rob Froetscher commented on AIRFLOW-303: Ah no worries, figured it was a bot because it was on the github PR > EMR Operator job_flow_overrides allow for templating > > > Key: AIRFLOW-303 > URL: https://issues.apache.org/jira/browse/AIRFLOW-303 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rob Froetscher >Assignee: Rob Froetscher >Priority: Minor > > See discussion here: https://github.com/apache/incubator-airflow/pull/1630 > Idea is that we want to allow templating on a dictionary argument to an > operator. This is probably best done by introspecting what are the strings in > the dictionary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (AIRFLOW-303) EMR Operator job_flow_overrides allow for templating
[ https://issues.apache.org/jira/browse/AIRFLOW-303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Froetscher reopened AIRFLOW-303: > EMR Operator job_flow_overrides allow for templating > > > Key: AIRFLOW-303 > URL: https://issues.apache.org/jira/browse/AIRFLOW-303 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rob Froetscher >Assignee: Rob Froetscher >Priority: Minor > > See discussion here: https://github.com/apache/incubator-airflow/pull/1630 > Idea is that we want to allow templating on a dictionary argument to an > operator. This is probably best done by introspecting what are the strings in > the dictionary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-303) EMR Operator job_flow_overrides allow for templating
Rob Froetscher created AIRFLOW-303: -- Summary: EMR Operator job_flow_overrides allow for templating Key: AIRFLOW-303 URL: https://issues.apache.org/jira/browse/AIRFLOW-303 Project: Apache Airflow Issue Type: Improvement Reporter: Rob Froetscher Assignee: Rob Froetscher Priority: Minor See discussion here: https://github.com/apache/incubator-airflow/pull/1630 Idea is that we want to allow templating on a dictionary argument to an operator. This is probably best done by introspecting what are the strings in the dictionary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-247) EMR Hook, Operators, Sensor
[ https://issues.apache.org/jira/browse/AIRFLOW-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352017#comment-15352017 ] Rob Froetscher commented on AIRFLOW-247: https://github.com/apache/incubator-airflow/pull/1630 > EMR Hook, Operators, Sensor > --- > > Key: AIRFLOW-247 > URL: https://issues.apache.org/jira/browse/AIRFLOW-247 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Rob Froetscher >Assignee: Rob Froetscher >Priority: Minor > > Substory of https://issues.apache.org/jira/browse/AIRFLOW-115. It would be > nice to have an EMR hook and operators. > Hook to generally interact with EMR. > Operators to: > * setup and start a job flow > * add steps to an existing jobflow > A sensor to: > * monitor completion and status of EMR jobs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (AIRFLOW-247) EMR Hook, Operators, Sensor
[ https://issues.apache.org/jira/browse/AIRFLOW-247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-247 started by Rob Froetscher. -- > EMR Hook, Operators, Sensor > --- > > Key: AIRFLOW-247 > URL: https://issues.apache.org/jira/browse/AIRFLOW-247 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Rob Froetscher >Assignee: Rob Froetscher >Priority: Minor > > Substory of https://issues.apache.org/jira/browse/AIRFLOW-115. It would be > nice to have an EMR hook and operators. > Hook to generally interact with EMR. > Operators to: > * setup and start a job flow > * add steps to an existing jobflow > A sensor to: > * monitor completion and status of EMR jobs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-247) EMR Hook, Operators, Sensor
[ https://issues.apache.org/jira/browse/AIRFLOW-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332314#comment-15332314 ] Rob Froetscher commented on AIRFLOW-247: [~sanand], not sure what the protocol is for creating issues related to something I'd like to implement. Is what I've done here acceptable? > EMR Hook, Operators, Sensor > --- > > Key: AIRFLOW-247 > URL: https://issues.apache.org/jira/browse/AIRFLOW-247 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Rob Froetscher >Assignee: Rob Froetscher >Priority: Minor > > Substory of https://issues.apache.org/jira/browse/AIRFLOW-115. It would be > nice to have an EMR hook and operators. > Hook to generally interact with EMR. > Operators to: > * setup and start a job flow > * add steps to an existing jobflow > A sensor to: > * monitor completion and status of EMR jobs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (AIRFLOW-247) EMR Hook, Operators, Sensor
[ https://issues.apache.org/jira/browse/AIRFLOW-247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Froetscher reassigned AIRFLOW-247: -- Assignee: Rob Froetscher > EMR Hook, Operators, Sensor > --- > > Key: AIRFLOW-247 > URL: https://issues.apache.org/jira/browse/AIRFLOW-247 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Rob Froetscher >Assignee: Rob Froetscher >Priority: Minor > > Substory of https://issues.apache.org/jira/browse/AIRFLOW-115. It would be > nice to have an EMR hook and operators. > Hook to generally interact with EMR. > Operators to: > * setup and start a job flow > * add steps to an existing jobflow > A sensor to: > * monitor completion and status of EMR jobs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-247) EMR Hook, Operators, Sensor
Rob Froetscher created AIRFLOW-247: -- Summary: EMR Hook, Operators, Sensor Key: AIRFLOW-247 URL: https://issues.apache.org/jira/browse/AIRFLOW-247 Project: Apache Airflow Issue Type: New Feature Reporter: Rob Froetscher Priority: Minor Substory of https://issues.apache.org/jira/browse/AIRFLOW-115. It would be nice to have an EMR hook and operators. Hook to generally interact with EMR. Operators to: * setup and start a job flow * add steps to an existing jobflow A sensor to: * monitor completion and status of EMR jobs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
