dstandish opened a new issue #12999: URL: https://github.com/apache/airflow/issues/12999
**Apache Airflow version**: 2.0.0rc1 **Environment**: - **Cloud provider or hardware configuration**: mac - **OS** (e.g. from /etc/os-release): macOS catalina - **Kernel** (e.g. `uname -a`): Darwin TFGELSOS3516375 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 i386 MacBookPro16,1 Darwin - **Install tools**: pip - **Others**: SequentialExecutor / sqlite **What happened**: Let some test dags run for a while in the background, seemed to be running fine, then got this error. I may have started webserver during this time -- it was close in time but i don't know if was concurrent. **What you expected to happen**: no error **How to reproduce it**: I just let it run for a while using mssql operator and odbc hook. Not sure why it is calling "sudo" **Other note** After this error, i left scheduler running, and it seems to be doing fine -- have not seen it again yet. When searching for issues i found this one https://github.com/apache/airflow/issues/12290 but that was 1.10.12. **Log** ``` [2020-12-10 12:30:11,277] {scheduler_job.py:1200} INFO - Executor reports execution of test_dag.mssqltask execution_date=2020-12-10 20:25:00+00:00 exited with status success for try_number 1 [2020-12-10 12:30:12,390] {dagrun.py:444} INFO - Marking run <DagRun test_dag @ 2020-12-10 20:25:00+00:00: scheduled__2020-12-10T20:25:00+00:00, externally triggered: False> successful Process ForkProcess-1: Traceback (most recent call last): File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context cursor, statement, parameters, context File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute cursor.execute(statement, parameters) sqlite3.OperationalError: database is locked The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/dstandish/.pyenv/versions/3.7.5/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/Users/dstandish/.pyenv/versions/3.7.5/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 365, in _run_processor_manager processor_manager.start() File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 596, in start return self._run_parsing_loop() File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 661, in _run_parsing_loop self._refresh_dag_dir() File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 744, in _refresh_dag_dir DagCode.remove_deleted_code(self._file_paths) File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/airflow/utils/session.py", line 65, in wrapper return func(*args, session=session, **kwargs) File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/airflow/models/dagcode.py", line 139, in remove_deleted_code ).delete(synchronize_session='fetch') File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3926, in delete delete_op.exec_() File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 1697, in exec_ self._do_exec() File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 1928, in _do_exec self._execute_stmt(delete_stmt) File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 1702, in _execute_stmt self.result = self.query._execute_crud(stmt, self.mapper) File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3568, in _execute_crud return conn.execute(stmt, self._params) File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1011, in execute return meth(self, multiparams, params) File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement distilled_params, File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context e, statement, parameters, cursor, context File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception sqlalchemy_exception, with_traceback=exc_info[2], from_=e File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_ raise exception File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context cursor, statement, parameters, context File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked [SQL: DELETE FROM dag_code WHERE dag_code.fileloc_hash NOT IN (?) AND dag_code.fileloc NOT IN (?)] [parameters: (65669301801004096, '/Users/dstandish/code/abctest/dags/testing.py')] (Background on this error at: http://sqlalche.me/e/13/e3q8) [2020-12-10 12:30:17,433] {base_job.py:222} ERROR - SchedulerJob heartbeat got an exception Traceback (most recent call last): File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 771, in _commit_impl self.engine.dialect.do_commit(self.connection) File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 546, in do_commit dbapi_connection.commit() sqlite3.OperationalError: database is locked The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 195, in heartbeat previous_heartbeat = self.latest_heartbeat File "/Users/dstandish/.pyenv/versions/3.7.5/lib/python3.7/contextlib.py", line 119, in __exit__ next(self.gen) File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/airflow/utils/session.py", line 32, in create_session session.commit() File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1042, in commit self.transaction.commit() File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 508, in commit t[1].commit() File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1762, in commit self._do_commit() File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1793, in _do_commit self.connection._commit_impl() File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 773, in _commit_impl self._handle_dbapi_exception(e, None, None, None, None) File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception sqlalchemy_exception, with_traceback=exc_info[2], from_=e File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_ raise exception File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 771, in _commit_impl self.engine.dialect.do_commit(self.connection) File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 546, in do_commit dbapi_connection.commit() sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked (Background on this error at: http://sqlalche.me/e/13/e3q8) [2020-12-10 12:30:18,510] {dag_processing.py:399} WARNING - DagFileProcessorManager (PID=5377) exited with exit code 1 - re-launching``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
