[jira] [Commented] (AIRFLOW-2340) SQLalchemy pessimistic connection handling not working
[ https://issues.apache.org/jira/browse/AIRFLOW-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582329#comment-16582329 ] Mishika Singh commented on AIRFLOW-2340: similar issue was raised https://issues.apache.org/jira/browse/AIRFLOW-2703 and PR is also raised for this. > SQLalchemy pessimistic connection handling not working > -- > > Key: AIRFLOW-2340 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2340 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.9.0 >Reporter: John Arnold >Priority: Critical > Attachments: airflow_traceback.txt, webserver.txt > > > Our scheduler keeps crashing, about once a day. It seems to be triggered by > a failure to connect to the postgresql database, but then it doesn't recover > and crashes the scheduler over and over. > The scheduler runs in a container in our environment, so after several > container restarts, docker gives up and the container stays down. > Airflow should be able to recover from a connection failure without blowing > up the container altogether. Perhaps some exponential backoff is needed? > > See attached log from the scheduler. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2833) Delay in trigger of downstream tasks in DAG
Mishika Singh created AIRFLOW-2833: -- Summary: Delay in trigger of downstream tasks in DAG Key: AIRFLOW-2833 URL: https://issues.apache.org/jira/browse/AIRFLOW-2833 Project: Apache Airflow Issue Type: Bug Reporter: Mishika Singh Attachments: Screen Shot 2018-05-25 at 9.18.08 AM.png There is around 2 minutes of delay in triggering the downstream tasks on completion of upstream tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2703) Scheduler crashes if Mysql Connectivity is lost
[ https://issues.apache.org/jira/browse/AIRFLOW-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559422#comment-16559422 ] Mishika Singh commented on AIRFLOW-2703: raised PR for this: [https://github.com/apache/incubator-airflow/pull/3650] > Scheduler crashes if Mysql Connectivity is lost > --- > > Key: AIRFLOW-2703 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2703 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: Airflow 2.0, 1.9.0 >Reporter: raman >Priority: Major > > Airflow scheduler crashes if connectivity to Mysql is lost. > Below is the stack Trace > Traceback (most recent call last): File > "/usr/src/venv/local/lib/python2.7/site-packages/airflow/jobs.py", line 371, > in helper pickle_dags) File > "/usr/src/venv/local/lib/python2.7/site-packages/airflow/utils/db.py", line > 50, in wrapper result = func(*args, **kwargs) File > "/usr/src/venv/local/lib/python2.7/site-packages/airflow/jobs.py", line 1762, > in process_file dag.sync_to_db() File > "/usr/src/venv/local/lib/python2.7/site-packages/airflow/utils/db.py", line > 50, in wrapper result = func(*args, **kwargs) File > "/usr/src/venv/local/lib/python2.7/site-packages/airflow/models.py", line > 3816, in sync_to_db session.commit() File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", > line 943, in commit self.transaction.commit() File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", > line 471, in commit t[1].commit() File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", > line 1643, in commit self._do_commit() File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", > line 1674, in _do_commit self.connection._commit_impl() File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", > line 726, in _commit_impl self._handle_dbapi_exception(e, None, None, None, > None) File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", > line 1413, in _handle_dbapi_exception exc_info File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", > line 203, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, > cause=cause) File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", > line 724, in _commit_impl self.engine.dialect.do_commit(self.connection) File > "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", > line 1784, in do_commit dbapi_connection.commit() OperationalError: > (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server > during query') (Background on this error at: http://sqlalche.me/e/e3q8) > Process DagFileProcessor141318-Process: -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2614) Airflow trigger_run API is very slow
[ https://issues.apache.org/jira/browse/AIRFLOW-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16538486#comment-16538486 ] Mishika Singh commented on AIRFLOW-2614: raised PR for this : [https://github.com/apache/incubator-airflow/pull/3590] > Airflow trigger_run API is very slow > > > Key: AIRFLOW-2614 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2614 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: Airflow 2.0, 1.9.0 >Reporter: raman >Priority: Major > > Current implementation of trigger_dag processes all Local dag files > sequentially before creating a DAG run. Its done inside trigger_dag function > in trigger_dag.py > "def trigger_dag(dag_id, run_id=None, conf=None, execution_date=None): > dagbag = DagBag() > ." > Processing all the files to get the dagBag slows down the trigger_dag api and > increases latency. We have observed that it starts taking 10(s) of seconds as > number of Dag Files increase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2614) Airflow trigger_run API is very slow
[ https://issues.apache.org/jira/browse/AIRFLOW-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mishika Singh reassigned AIRFLOW-2614: -- Assignee: Mishika Singh > Airflow trigger_run API is very slow > > > Key: AIRFLOW-2614 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2614 > Project: Apache Airflow > Issue Type: Bug > Components: DagRun >Affects Versions: Airflow 2.0, 1.9.0 >Reporter: raman >Assignee: Mishika Singh >Priority: Major > > Current implementation of trigger_dag processes all Local dag files > sequentially before creating a DAG run. Its done inside trigger_dag function > in trigger_dag.py > "def trigger_dag(dag_id, run_id=None, conf=None, execution_date=None): > dagbag = DagBag() > ." > Processing all the files to get the dagBag slows down the trigger_dag api and > increases latency. We have observed that it starts taking 10(s) of seconds as > number of Dag Files increase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
