Robin Miller created AIRFLOW-657:
------------------------------------
Summary: MsSqlOperator Autocommit Option
Key: AIRFLOW-657
URL: https://issues.apache.org/jira/browse/AIRFLOW-657
Project: Apache Airflow
Issue Type: Improvement
Components: operators
Reporter: Robin Miller
Assignee: Robin Miller
Priority: Trivial
The MsSqlOperator is not exposing the option to enable autocommit, even though
the MsSqlHook has supports_autocommit to true.
This means that tasks running on MsSql are always wrapped in an additional
transaction, which means that parallel tasks using shared resources can cause
SQL Deadlocks.
We see no reason not to expose this option to the user via the operator and
have produced and tested the change, keeping the existing behaviour as the
default.
In our use case we have a number of identical parallel tasks to work through a
list of identical build tasks with an appropriate level of parallelism. This is
prevented from working as intended since the first of these tasks to start,
updates this task list and then holds a transaction lock on this table, meaning
the other threads cannot then access the table. (It would not be convenient to
have a task for each task in the list defined in the dag as there are upwards
of 7000 tasks and it is convenient for us to be able to manually add or remove
tasks from this list during execution)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
