[
https://issues.apache.org/jira/browse/AIRFLOW-3372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16694315#comment-16694315
]
Jayce Li commented on AIRFLOW-3372:
-----------------------------------
How about just let 'airflow scheduler' pod run and see what's happen inside?
When I debug k8s pod, I will let the pod run for nothing. then I will '*kubectl
exec -it POD_NAME bash*' to insight pod.
{code:yaml}
// airflow_scheduler.yaml
containers:
- name: airflow-scheduler
image: airflow
imagePullPolicy: IfNotPresent
command: [ "/bin/bash", "-c", "--" ]
# args: ["airflow scheduler"]
args: [ "while true; do sleep 30; done;" ] # do nothing but running.
{code}
> Unable to start airflow scheduler
> ---------------------------------
>
> Key: AIRFLOW-3372
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3372
> Project: Apache Airflow
> Issue Type: Bug
> Components: docker, kubernetes, scheduler
> Affects Versions: 1.9.0
> Environment: Kubernetes,docker
> Reporter: MADHANKUMAR C
> Priority: Blocker
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> *I have installed airflow in kubernetes cluster.When i am installing airflow
> ,i am unable to start the scheduler.The below is the log of scheduler
> container.*
> [2018-11-20 12:02:40,860] {{__init__.py:51}} INFO - Using executor
> SequentialExecutor
> [2018-11-20 12:02:40,973] {{cli_action_loggers.py:69}} ERROR - Failed on
> pre-execution callback using <function default_action_log at 0x7f26b730b620>
> Traceback (most recent call last):
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py",
> line 1182, in _execute_context
> context)
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py",
> line 470, in do_execute
> cursor.execute(statement, parameters)
> sqlite3.OperationalError: no such table: log
> The above exception was the direct cause of the following exception:
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.5/dist-packages/airflow/utils/cli_action_loggers.py",
> line 67, in on_pre_execution
> cb(**kwargs)
> File
> "/usr/local/lib/python3.5/dist-packages/airflow/utils/cli_action_loggers.py",
> line 99, in default_action_log
> session.commit()
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py",
> line 927, in commit
> self.transaction.commit()
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py",
> line 467, in commit
> self._prepare_impl()
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py",
> line 447, in _prepare_impl
> self.session.flush()
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py",
> line 2209, in flush
> self._flush(objects)
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py",
> line 2329, in _flush
> transaction.rollback(_capture_exception=True)
> File
> "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/langhelpers.py", line
> 66, in __exit__
> compat.reraise(exc_type, exc_value, exc_tb)
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py",
> line 187, in reraise
> raise value
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py",
> line 2293, in _flush
> flush_context.execute()
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py",
> line 389, in execute
> rec.execute(self)
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/unitofwork.py",
> line 548, in execute
> uow
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/persistence.py",
> line 181, in save_obj
> mapper, table, insert)
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/persistence.py",
> line 835, in _emit_insert_statements
> execute(statement, params)
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py",
> line 945, in execute
> return meth(self, multiparams, params)
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/sql/elements.py",
> line 263, in _execute_on_connection
> return connection._execute_clauseelement(self, multiparams, params)
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py",
> line 1053, in _execute_clauseelement
> compiled_sql, distilled_params
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py",
> line 1189, in _execute_context
> context)
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py",
> line 1402, in _handle_dbapi_exception
> exc_info
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py",
> line 203, in raise_from_cause
> reraise(type(exception), exception, tb=exc_tb, cause=cause)
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/util/compat.py",
> line 186, in reraise
> raise value.with_traceback(tb)
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/base.py",
> line 1182, in _execute_context
> context)
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/engine/default.py",
> line 470, in do_execute
> cursor.execute(statement, parameters)
> sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table:
> log [SQL: 'INSERT INTO log (dttm, dag_id, task_id, event, execution_date,
> owner, extra) VALUES (?, ?, ?, ?, ?, ?, ?)'] [parameters: ('2018-11-20
> 12:02:40.969353', None, None, 'cli_scheduler', None, 'airflow', '
> {"host_name": "airflow-airflow-scheduler-5b5f8b9549-89dmn", "full_command":
> "[\'/usr/local/bin/airflow\', \'scheduler\', \'-n\', \'-1\', \'-p\']"}
> ')]
> ____________ _____________
> ____ |__( )_________ __/__ /________ __
> ____ /| |_ /__ ___/_ /_ __ /_ __ _ | /| / /
> ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
> _/_/ |_/_/ /_/ /_/ /_/ ____/____/|__/
> [2018-11-20 12:02:40,977] {{jobs.py:580}} *{color:#FF0000}ERROR - Cannot use
> more than 1 thread when using sqlite. Setting max_threads to 1{color}*
> /usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py:513:
> SAWarning: Session's state has been changed on a non-active transaction -
> this state will be discarded.
> "Session's state has been changed on "
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 32, in <module>
> args.func(args)
> File "/usr/local/lib/python3.5/dist-packages/airflow/utils/cli.py", line 74,
> in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 925,
> in scheduler
> job.run()
> File "/usr/local/lib/python3.5/dist-packages/airflow/jobs.py", line 196, in
> run
> session.commit()
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py",
> line 927, in commit
> self.transaction.commit()
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py",
> line 465, in commit
> self._assert_active(prepared_ok=True)
> File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/orm/session.py",
> line 276, in _assert_active
> % self._rollback_exception
> sqlalchemy.exc.InvalidRequestError: This Session's transaction has been
> rolled back due to a previous exception during flush. To begin a new
> transaction with this Session, first issue Session.rollback(). Original
> exception was: (sqlite3.OperationalError) no such table: log [SQL: 'INSERT
> INTO log (dttm, dag_id, task_id, event, execution_date, owner, extra) VALUES
> (?, ?, ?, ?, ?, ?, ?)'] [parameters: ('2018-11-20 12:02:40.969353', None,
> None, 'cli_scheduler', None, 'airflow', '
> {"host_name": "airflow-airflow-scheduler-5b5f8b9549-89dmn", "full_command":
> "[\'/usr/local/bin/airflow\', \'scheduler\', \'-n\', \'-1\', \'-p\']"}
> ')]
> {color:#FF0000}*The pod status is like below*{color}
> [root@kubernetes-cpal-master-0 kube-airflow]# *kubectl get pod*
> NAME READY STATUS RESTARTS AGE
> airflow-airflow-flower-6668559cf7-ll5bn 1/1 Running 0 19m
> ~{color:#FF0000}*airflow-airflow-scheduler-5b5f8b9549-89dmn 0/1
> CrashLoopBackOff 8 19m*{color}~
> airflow-airflow-web-89d8fb554-9ztss 1/1 Running 0 19m
> airflow-airflow-web-89d8fb554-f4mbm 1/1 Running 0 19m
> airflow-airflow-worker-0 1/1 Running 0 19m
> airflow-postgresql-9df799579-swg8q 1/1 Running 0 19m
> airflow-redis-7d75b85f7-26lsx 1/1 Running 0 19m
> centos 0/1 CrashLoopBackOff 1278 29d
> logger-deepinsights-fluentd-759ffcfc5d-fjmml 1/1 Running 8 33d
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
