[
https://issues.apache.org/jira/browse/AIRFLOW-30?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Mandich reassigned AIRFLOW-30:
------------------------------------
Assignee: Kevin Mandich
> Make preoperators part of the same transaction as the actual operation
> ----------------------------------------------------------------------
>
> Key: AIRFLOW-30
> URL: https://issues.apache.org/jira/browse/AIRFLOW-30
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
> Reporter: Bence Nagy
> Assignee: Kevin Mandich
>
> All my use cases would work better if each operator would execute everything
> in one transaction. Two examples:
> - I want to {{GenericTransfer}} a set of rows from one DB to another, and I
> have to create the table first in the destination DB. I feel like it'd be a
> lot more clean if I didn't have empty tables lying around if the insertion
> fails for some reason later on.
> - I want to {{GenericTransfer}} all rows from an entire table periodically to
> sync it from one DB to another. To do this correctly I want to clear the
> destination table first to make sure I end up with no duplicate rows, so I'd
> have a {{DELETE * FROM dst_table}} preoperator. If the insertions fail
> afterwards, I'd end up with no data (it would be better in most cases to fall
> back to the old data), and even if everything is working correctly, I'll have
> an empty table while the insertions as still executing.
> To fix this, the relevant {{DbApiHook}} methods could support a new kwarg to
> set whether it should commit at the end.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
