[jira] [Resolved] (AIRFLOW-1040) Fix typos in comments/docstrings in models.py
[ https://issues.apache.org/jira/browse/AIRFLOW-1040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremiah Lowin resolved AIRFLOW-1040. - Resolution: Fixed Fix Version/s: 1.9.0 Issue resolved by pull request #2174 [https://github.com/apache/incubator-airflow/pull/2174] > Fix typos in comments/docstrings in models.py > - > > Key: AIRFLOW-1040 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1040 > Project: Apache Airflow > Issue Type: Task > Components: docs >Reporter: Matthew Schmoyer >Assignee: Matthew Schmoyer >Priority: Trivial > Fix For: 1.9.0 > > > There are several small spelling typos in > https://github.com/apache/incubator-airflow/blob/master/airflow/models.py > Also some doc string formatting needs to be fixed, such as colons being in > the incorrect spot in things like `:param: default` and the function > `clean_dirty()` has doc string params that don't exist in the actual function. > This issue is being addressed by PR: > https://github.com/apache/incubator-airflow/pull/2174 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1040) Fix typos in comments/docstrings in models.py
[ https://issues.apache.org/jira/browse/AIRFLOW-1040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941383#comment-15941383 ] ASF subversion and git services commented on AIRFLOW-1040: -- Commit d8c0f59d5d627efb65b47264d5169e3626195839 in incubator-airflow's branch refs/heads/master from [~mschmoyer] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d8c0f59 ] [AIRFLOW-1040] Fix some small typos in comments and docstrings Closes #2174 from mschmo/fix-some-typos-models > Fix typos in comments/docstrings in models.py > - > > Key: AIRFLOW-1040 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1040 > Project: Apache Airflow > Issue Type: Task > Components: docs >Reporter: Matthew Schmoyer >Assignee: Matthew Schmoyer >Priority: Trivial > Fix For: 1.9.0 > > > There are several small spelling typos in > https://github.com/apache/incubator-airflow/blob/master/airflow/models.py > Also some doc string formatting needs to be fixed, such as colons being in > the incorrect spot in things like `:param: default` and the function > `clean_dirty()` has doc string params that don't exist in the actual function. > This issue is being addressed by PR: > https://github.com/apache/incubator-airflow/pull/2174 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1017) get_task_instance should return None instead of throw an exception for non-existent TIs
[
https://issues.apache.org/jira/browse/AIRFLOW-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941325#comment-15941325
]
Jeeyoung Kim commented on AIRFLOW-1017:
---
I just ran into this issue. Thanks for resolving!
> get_task_instance should return None instead of throw an exception for
> non-existent TIs
> ---
>
> Key: AIRFLOW-1017
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1017
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Dan Davydov
>Assignee: Dan Davydov
>Priority: Critical
>
> We were seeing errors in our scheduler like the following due to this issue:
> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line
> 2796, in one
> raise orm_exc.NoResultFound("No row was found for one()")
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (AIRFLOW-676) Do not allow Pools with 0 slots
[ https://issues.apache.org/jira/browse/AIRFLOW-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Anand closed AIRFLOW-676. --- Resolution: Fixed Not enough community interest in this! > Do not allow Pools with 0 slots > --- > > Key: AIRFLOW-676 > URL: https://issues.apache.org/jira/browse/AIRFLOW-676 > Project: Apache Airflow > Issue Type: Bug > Components: pools >Affects Versions: Airflow 1.7.1 >Reporter: Jon Chase >Assignee: Siddharth Anand >Priority: Minor > > It's possible in the UI to create a pool with 0 slots. A pool with 0 slots > is unable to execute any tasks. > Additionally, there are no UI indicators such as queued tasks? With 0 sized > pools on 1.7.1.3, we did not see queued tasks per my recollection. This made > it harder to root cause why a dag was not being scheduled. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AIRFLOW-1041) DockerOperator replaces its xcom_push method with a boolean
Ludovic Claude created AIRFLOW-1041: --- Summary: DockerOperator replaces its xcom_push method with a boolean Key: AIRFLOW-1041 URL: https://issues.apache.org/jira/browse/AIRFLOW-1041 Project: Apache Airflow Issue Type: Bug Components: docker Affects Versions: 1.8.0 Reporter: Ludovic Claude xcom_push method is not usable from DockerOperator as it is replaced by a boolean value. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AIRFLOW-1040) Fix typos in comments/docstrings in models.py
Matthew Schmoyer created AIRFLOW-1040: - Summary: Fix typos in comments/docstrings in models.py Key: AIRFLOW-1040 URL: https://issues.apache.org/jira/browse/AIRFLOW-1040 Project: Apache Airflow Issue Type: Task Components: docs Reporter: Matthew Schmoyer Assignee: Matthew Schmoyer Priority: Trivial There are several small spelling typos in https://github.com/apache/incubator-airflow/blob/master/airflow/models.py Also some doc string formatting needs to be fixed, such as colons being in the incorrect spot in things like `:param: default` and the function `clean_dirty()` has doc string params that don't exist in the actual function. This issue is being addressed by PR: https://github.com/apache/incubator-airflow/pull/2174 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1038) Specify celery serializers explicitly and pin version
[ https://issues.apache.org/jira/browse/AIRFLOW-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Guziel updated AIRFLOW-1038: - Summary: Specify celery serializers explicitly and pin version (was: Specify celery serializers explicitly) > Specify celery serializers explicitly and pin version > - > > Key: AIRFLOW-1038 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1038 > Project: Apache Airflow > Issue Type: Bug >Reporter: Alex Guziel >Assignee: Alex Guziel > > Celery 3->4 upgrade changes the default task and result serializer from > pickle to json. Pickle is faster and supports more types > http://docs.celeryproject.org/en/latest/userguide/calling.html > This also causes issues when different versions of celery are running on > different hosts. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1039) Airflow is raising IntegrityError when during parallel DAG trigger
[
https://issues.apache.org/jira/browse/AIRFLOW-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941074#comment-15941074
]
Matus Valo commented on AIRFLOW-1039:
-
PR created on github: https://github.com/apache/incubator-airflow/pull/2186
> Airflow is raising IntegrityError when during parallel DAG trigger
> --
>
> Key: AIRFLOW-1039
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1039
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
>Affects Versions: Airflow 1.8
>Reporter: Matus Valo
>Priority: Minor
>
> When Two concurrent processes are trying to trigger the same dag with the
> same execution date at the same time, the IntegrityError is thrown by
> SQLAlchemy:
> uwsgi[15887]: [2017-03-24 12:51:38,074] {app.py:1587} ERROR - Exception on /
> [POST]
> uwsgi[15887]: Traceback (most recent call last):
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/flask/app.py", line
> 1988, in wsgi_app
> uwsgi[15887]: response = self.full_dispatch_request()
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/flask/app.py", line
> 1641, in full_dispatch_request
> uwsgi[15887]: rv = self.handle_user_exception(e)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/flask/app.py", line
> 1544, in handle_user_exception
> uwsgi[15887]: reraise(exc_type, exc_value, tb)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/flask/app.py", line
> 1639, in full_dispatch_request
> uwsgi[15887]: rv = self.dispatch_request()
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/flask/app.py", line
> 1625, in dispatch_request
> uwsgi[15887]: return self.view_functions[rule.endpoint](**req.view_args)
> uwsgi[15887]: File "./ws.py", line 21, in hello
> uwsgi[15887]: trigger_dag('poc_dag2', run_id=str(uuid1()),
> conf=json.dumps({'input_files': input_files}), execution_date=datetime.now())
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/airflow/api/common/experimental/trigger_dag.py",
> line 56, in trigger_dag
> uwsgi[15887]: external_trigger=True
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/airflow/utils/db.py",
> line 53, in wrapper
> uwsgi[15887]: result = func(*args, **kwargs)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/airflow/models.py",
> line 3377, in create_dagrun
> uwsgi[15887]: session.commit()
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
> line 874, in commit
> uwsgi[15887]: self.transaction.commit()
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
> line 461, in commit
> uwsgi[15887]: self._prepare_impl()
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
> line 441, in _prepare_impl
> uwsgi[15887]: self.session.flush()
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
> line 2139, in flush
> uwsgi[15887]: self._flush(objects)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
> line 2259, in _flush
> uwsgi[15887]: transaction.rollback(_capture_exception=True)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py",
> line 60, in __exit__
> uwsgi[15887]: compat.reraise(exc_type, exc_value, exc_tb)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
> line 2223, in _flush
> uwsgi[15887]: flush_context.execute()
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py",
> line 389, in execute
> uwsgi[15887]: rec.execute(self)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py",
> line 548, in execute
> uwsgi[15887]: uow
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py",
> line 181, in save_obj
> uwsgi[15887]: mapper, table, insert)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py",
> line 835, in _emit_insert_statements
> uwsgi[15887]: execute(statement, params)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 945, in execute
> uwsgi[15887]: return meth(self, multiparams, params)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/sql/elements.py",
> line 263, in _execute_on_connection
> uwsgi[15887]: return
[jira] [Updated] (AIRFLOW-1039) Airflow is raising IntegrityError when during parallel DAG trigger
[
https://issues.apache.org/jira/browse/AIRFLOW-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matus Valo updated AIRFLOW-1039:
Flags: Patch
> Airflow is raising IntegrityError when during parallel DAG trigger
> --
>
> Key: AIRFLOW-1039
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1039
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
>Affects Versions: Airflow 1.8
>Reporter: Matus Valo
>Priority: Minor
>
> When Two concurrent processes are trying to trigger the same dag with the
> same execution date at the same time, the IntegrityError is thrown by
> SQLAlchemy:
> uwsgi[15887]: [2017-03-24 12:51:38,074] {app.py:1587} ERROR - Exception on /
> [POST]
> uwsgi[15887]: Traceback (most recent call last):
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/flask/app.py", line
> 1988, in wsgi_app
> uwsgi[15887]: response = self.full_dispatch_request()
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/flask/app.py", line
> 1641, in full_dispatch_request
> uwsgi[15887]: rv = self.handle_user_exception(e)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/flask/app.py", line
> 1544, in handle_user_exception
> uwsgi[15887]: reraise(exc_type, exc_value, tb)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/flask/app.py", line
> 1639, in full_dispatch_request
> uwsgi[15887]: rv = self.dispatch_request()
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/flask/app.py", line
> 1625, in dispatch_request
> uwsgi[15887]: return self.view_functions[rule.endpoint](**req.view_args)
> uwsgi[15887]: File "./ws.py", line 21, in hello
> uwsgi[15887]: trigger_dag('poc_dag2', run_id=str(uuid1()),
> conf=json.dumps({'input_files': input_files}), execution_date=datetime.now())
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/airflow/api/common/experimental/trigger_dag.py",
> line 56, in trigger_dag
> uwsgi[15887]: external_trigger=True
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/airflow/utils/db.py",
> line 53, in wrapper
> uwsgi[15887]: result = func(*args, **kwargs)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/airflow/models.py",
> line 3377, in create_dagrun
> uwsgi[15887]: session.commit()
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
> line 874, in commit
> uwsgi[15887]: self.transaction.commit()
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
> line 461, in commit
> uwsgi[15887]: self._prepare_impl()
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
> line 441, in _prepare_impl
> uwsgi[15887]: self.session.flush()
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
> line 2139, in flush
> uwsgi[15887]: self._flush(objects)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
> line 2259, in _flush
> uwsgi[15887]: transaction.rollback(_capture_exception=True)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py",
> line 60, in __exit__
> uwsgi[15887]: compat.reraise(exc_type, exc_value, exc_tb)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
> line 2223, in _flush
> uwsgi[15887]: flush_context.execute()
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py",
> line 389, in execute
> uwsgi[15887]: rec.execute(self)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py",
> line 548, in execute
> uwsgi[15887]: uow
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py",
> line 181, in save_obj
> uwsgi[15887]: mapper, table, insert)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py",
> line 835, in _emit_insert_statements
> uwsgi[15887]: execute(statement, params)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
> line 945, in execute
> uwsgi[15887]: return meth(self, multiparams, params)
> uwsgi[15887]: File
> "/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/sql/elements.py",
> line 263, in _execute_on_connection
> uwsgi[15887]: return connection._execute_clauseelement(self, multiparams,
> params)
> uwsgi[15887]: File
>
[jira] [Created] (AIRFLOW-1039) Airflow is raising IntegrityError when during parallel DAG trigger
Matus Valo created AIRFLOW-1039:
---
Summary: Airflow is raising IntegrityError when during parallel
DAG trigger
Key: AIRFLOW-1039
URL: https://issues.apache.org/jira/browse/AIRFLOW-1039
Project: Apache Airflow
Issue Type: Bug
Components: DagRun
Affects Versions: Airflow 1.8
Reporter: Matus Valo
Priority: Minor
When Two concurrent processes are trying to trigger the same dag with the same
execution date at the same time, the IntegrityError is thrown by SQLAlchemy:
uwsgi[15887]: [2017-03-24 12:51:38,074] {app.py:1587} ERROR - Exception on /
[POST]
uwsgi[15887]: Traceback (most recent call last):
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/flask/app.py", line 1988,
in wsgi_app
uwsgi[15887]: response = self.full_dispatch_request()
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/flask/app.py", line 1641,
in full_dispatch_request
uwsgi[15887]: rv = self.handle_user_exception(e)
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/flask/app.py", line 1544,
in handle_user_exception
uwsgi[15887]: reraise(exc_type, exc_value, tb)
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/flask/app.py", line 1639,
in full_dispatch_request
uwsgi[15887]: rv = self.dispatch_request()
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/flask/app.py", line 1625,
in dispatch_request
uwsgi[15887]: return self.view_functions[rule.endpoint](**req.view_args)
uwsgi[15887]: File "./ws.py", line 21, in hello
uwsgi[15887]: trigger_dag('poc_dag2', run_id=str(uuid1()),
conf=json.dumps({'input_files': input_files}), execution_date=datetime.now())
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/airflow/api/common/experimental/trigger_dag.py",
line 56, in trigger_dag
uwsgi[15887]: external_trigger=True
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/airflow/utils/db.py",
line 53, in wrapper
uwsgi[15887]: result = func(*args, **kwargs)
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/airflow/models.py", line
3377, in create_dagrun
uwsgi[15887]: session.commit()
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 874, in commit
uwsgi[15887]: self.transaction.commit()
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 461, in commit
uwsgi[15887]: self._prepare_impl()
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 441, in _prepare_impl
uwsgi[15887]: self.session.flush()
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 2139, in flush
uwsgi[15887]: self._flush(objects)
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 2259, in _flush
uwsgi[15887]: transaction.rollback(_capture_exception=True)
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py",
line 60, in __exit__
uwsgi[15887]: compat.reraise(exc_type, exc_value, exc_tb)
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 2223, in _flush
uwsgi[15887]: flush_context.execute()
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py",
line 389, in execute
uwsgi[15887]: rec.execute(self)
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py",
line 548, in execute
uwsgi[15887]: uow
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py",
line 181, in save_obj
uwsgi[15887]: mapper, table, insert)
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py",
line 835, in _emit_insert_statements
uwsgi[15887]: execute(statement, params)
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 945, in execute
uwsgi[15887]: return meth(self, multiparams, params)
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/sql/elements.py",
line 263, in _execute_on_connection
uwsgi[15887]: return connection._execute_clauseelement(self, multiparams,
params)
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1053, in _execute_clauseelement
uwsgi[15887]: compiled_sql, distilled_params
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1189, in _execute_context
uwsgi[15887]: context)
uwsgi[15887]: File
"/home/matus/envs/airflow/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line
[jira] [Commented] (AIRFLOW-1003) DAG status flips erraticly
[ https://issues.apache.org/jira/browse/AIRFLOW-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940981#comment-15940981 ] Bolke de Bruin commented on AIRFLOW-1003: - I do consider it a bug but not something critical. It should be fixed eventually though. > DAG status flips erraticly > -- > > Key: AIRFLOW-1003 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1003 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: Airflow 1.8, 1.8.0rc5 >Reporter: Ruslan Dautkhanov > > Created a flow based on sample tutorial > https://airflow.incubator.apache.org/tutorial.html > (just changed dag-id to 'turorial-RD'). > See a 10 seconds video on this behavior: > http://screencast-o-matic.com/watch/cbebrn6kBw > Notice last DAG 'turorial-RD' keeps changing state > (it loses link and all the buttons and icons to show status > of the dag runs / tasks etc), then links and icons show back up. > In fact, during that short 10 seconds period it transitioned > that state 4 times (from "disabled" to a normal DAG). > There were no changes happenning in the system - I was just clicking > refresh in browser from time to time. All DAGs were disabled while > this was happening. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AIRFLOW-1038) Specify celery serializers explicitly
Alex Guziel created AIRFLOW-1038: Summary: Specify celery serializers explicitly Key: AIRFLOW-1038 URL: https://issues.apache.org/jira/browse/AIRFLOW-1038 Project: Apache Airflow Issue Type: Bug Reporter: Alex Guziel Assignee: Alex Guziel Celery 3->4 upgrade changes the default task and result serializer from pickle to json. Pickle is faster and supports more types http://docs.celeryproject.org/en/latest/userguide/calling.html This also causes issues when different versions of celery are running on different hosts. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1003) DAG status flips erraticly
[ https://issues.apache.org/jira/browse/AIRFLOW-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940779#comment-15940779 ] Ruslan Dautkhanov commented on AIRFLOW-1003: Thank you [~bolke]. Got it. If you consider this is expected behavior, I am okay to close this jira. > DAG status flips erraticly > -- > > Key: AIRFLOW-1003 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1003 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: Airflow 1.8, 1.8.0rc5 >Reporter: Ruslan Dautkhanov > > Created a flow based on sample tutorial > https://airflow.incubator.apache.org/tutorial.html > (just changed dag-id to 'turorial-RD'). > See a 10 seconds video on this behavior: > http://screencast-o-matic.com/watch/cbebrn6kBw > Notice last DAG 'turorial-RD' keeps changing state > (it loses link and all the buttons and icons to show status > of the dag runs / tasks etc), then links and icons show back up. > In fact, during that short 10 seconds period it transitioned > that state 4 times (from "disabled" to a normal DAG). > There were no changes happenning in the system - I was just clicking > refresh in browser from time to time. All DAGs were disabled while > this was happening. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-462) Concurrent Scheduler Jobs pushing the same task to queue
[ https://issues.apache.org/jira/browse/AIRFLOW-462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated AIRFLOW-462: --- Priority: Major (was: Blocker) > Concurrent Scheduler Jobs pushing the same task to queue > > > Key: AIRFLOW-462 > URL: https://issues.apache.org/jira/browse/AIRFLOW-462 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: Airflow 1.7.0 >Reporter: Yogesh > > Hi, > We are using airflow version 1.7.0 and we tried to implement high > availability for airflow daemons in our production environment. > Detailed high availability approach: > - Airflow running on two different machines with all the > daemons(webserver, scheduler, execueor) > - Single mysql db repository pointed by two schedulers > - Replicated dag files in both the machines > - Running Single Rabbitmq Instance as message broker > While doing so we came across below problem: > - A particular task was sent to executor twice (two entries in message > queue) by two different schedulers. But, we see only single entry for the > task instance in database which is correct. > We just checked out the code and found below fact: > - before sending the task to executor it checks for task state in > database and if its not already QUEUED it pushes that task to queue > issue: > As there is no locking implemented on the task instance in the database and > both the Scheduler jobs are running so close that the second one might check > for the status in the db before the first one updates that to QUEUED. > We are not sure if in recent release this issue have been taken care of. > Would you please help with some appropriate approach so that the high > availability can be achieved. > Thanks > Yogesh -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AIRFLOW-727) try_number is not increased
[ https://issues.apache.org/jira/browse/AIRFLOW-727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin resolved AIRFLOW-727. Resolution: Fixed Fix Version/s: 1.8.0 > try_number is not increased > --- > > Key: AIRFLOW-727 > URL: https://issues.apache.org/jira/browse/AIRFLOW-727 > Project: Apache Airflow > Issue Type: Bug >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.8.0 > > > A dag that has retries enabled will retry indefinitely as try_number gets > reset to 0 in LocalTaskJob as task_instance is not fully populated, but > nevertheless saved to the databases. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-462) Concurrent Scheduler Jobs pushing the same task to queue
[ https://issues.apache.org/jira/browse/AIRFLOW-462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated AIRFLOW-462: --- Issue Type: Wish (was: Bug) > Concurrent Scheduler Jobs pushing the same task to queue > > > Key: AIRFLOW-462 > URL: https://issues.apache.org/jira/browse/AIRFLOW-462 > Project: Apache Airflow > Issue Type: Wish > Components: scheduler >Affects Versions: Airflow 1.7.0 >Reporter: Yogesh > > Hi, > We are using airflow version 1.7.0 and we tried to implement high > availability for airflow daemons in our production environment. > Detailed high availability approach: > - Airflow running on two different machines with all the > daemons(webserver, scheduler, execueor) > - Single mysql db repository pointed by two schedulers > - Replicated dag files in both the machines > - Running Single Rabbitmq Instance as message broker > While doing so we came across below problem: > - A particular task was sent to executor twice (two entries in message > queue) by two different schedulers. But, we see only single entry for the > task instance in database which is correct. > We just checked out the code and found below fact: > - before sending the task to executor it checks for task state in > database and if its not already QUEUED it pushes that task to queue > issue: > As there is no locking implemented on the task instance in the database and > both the Scheduler jobs are running so close that the second one might check > for the status in the db before the first one updates that to QUEUED. > We are not sure if in recent release this issue have been taken care of. > Would you please help with some appropriate approach so that the high > availability can be achieved. > Thanks > Yogesh -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AIRFLOW-607) Cannot initdb on Oracle because of String(5000) fields in model
[ https://issues.apache.org/jira/browse/AIRFLOW-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin resolved AIRFLOW-607. Resolution: Won't Fix Oracle is not a supported backend DB. We will accept a PR for this, but it is not a blocker/bug to us. > Cannot initdb on Oracle because of String(5000) fields in model > --- > > Key: AIRFLOW-607 > URL: https://issues.apache.org/jira/browse/AIRFLOW-607 > Project: Apache Airflow > Issue Type: Bug >Affects Versions: Airflow 1.7.1.3 >Reporter: Paul Luong >Assignee: Paul Luong >Priority: Blocker > > Unfortunately Oracle cannot be used to host the Airflow application DB due to > the fact that Oracle limits varchar data types to 4000 bytes. > If you try to run the airflow initdb command to install the DB onto Oracle > you will end up with the following error: > sqlalchemy.exc.DatabaseError: (cx_Oracle.DatabaseError) ORA-00910: specified > length too long for its datatype > [SQL: '\nCREATE TABLE connection (\n\tid INTEGER NOT NULL, \n\tconn_id > VARCHAR2(250 CHAR), \n\tconn_type VARCHAR2(500 CHAR), \n\thost VARCHAR2(500 > CHAR), \n\tschema VARCHAR2(500 CHAR), \n\tlogin VARCHAR2(500 CHAR), > \n\tpassword VARCHAR2(500 CHAR), \n\tport INTEGER, \n\textra VARCHAR2(5000 > CHAR), \n\tPRIMARY KEY (id)\n)\n\n'] > Request that we change String(5000) types in the Chart and Connection models > to 4000. > Thanks! -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (AIRFLOW-660) Impossible to record second task failure
[
https://issues.apache.org/jira/browse/AIRFLOW-660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bolke de Bruin resolved AIRFLOW-660.
Resolution: Cannot Reproduce
> Impossible to record second task failure
>
>
> Key: AIRFLOW-660
> URL: https://issues.apache.org/jira/browse/AIRFLOW-660
> Project: Apache Airflow
> Issue Type: Bug
> Components: db
>Affects Versions: Airflow 1.8
>Reporter: Alexander Shorin
>Priority: Blocker
>
> {code}
> /var/log/airflow/airflow_scheduler_err.log.10: [SQL: 'INSERT INTO task_fail
> (task_id, dag_id, execution_date, start_date, end_date, duration) VALUES
> (%(task_id)s, %(dag_id)s, %(execution_date)s, %(start_date)s, %(end_date)s,
> %(duration)s)'] [parameters: {'task_id': 'test_task', 'end_date':
> datetime.datetime(2016, 11, 30, 14, 38, 39, 197485), 'execution_date':
> datetime.datetime(2016, 11, 30, 0, 0), 'duration': 331.723087, 'start_date':
> datetime.datetime(2016, 11, 30, 14, 33, 7, 474398), 'dag_id': 'test_dag'}]
> /var/log/airflow/airflow_scheduler_err.log.10-Process
> DagFileProcessor314-Process:
> /var/log/airflow/airflow_scheduler_err.log.10-Traceback (most recent call
> last):
> /var/log/airflow/airflow_scheduler_err.log.10- File
> "/usr/local/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
> /var/log/airflow/airflow_scheduler_err.log.10-self.run()
> /var/log/airflow/airflow_scheduler_err.log.10- File
> "/usr/local/lib/python2.7/multiprocessing/process.py", line 114, in run
> /var/log/airflow/airflow_scheduler_err.log.10-self._target(*self._args,
> **self._kwargs)
> /var/log/airflow/airflow_scheduler_err.log.10- File
> "/usr/local/lib/python2.7/site-packages/airflow/jobs.py", line 318, in helper
> /var/log/airflow/airflow_scheduler_err.log.10-pickle_dags)
> /var/log/airflow/airflow_scheduler_err.log.10- File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 56, in
> wrapper
> /var/log/airflow/airflow_scheduler_err.log.10-session.commit()
> /var/log/airflow/airflow_scheduler_err.log.10- File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 813,
> in commit
> /var/log/airflow/airflow_scheduler_err.log.10-self.transaction.commit()
> /var/log/airflow/airflow_scheduler_err.log.10- File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 390,
> in commit
> /var/log/airflow/airflow_scheduler_err.log.10-
> self._assert_active(prepared_ok=True)
> /var/log/airflow/airflow_scheduler_err.log.10- File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 214,
> in _assert_active
> /var/log/airflow/airflow_scheduler_err.log.10-% self._rollback_exception
> /var/log/airflow/airflow_scheduler_err.log.10:InvalidRequestError: This
> Session's transaction has been rolled back due to a previous exception during
> flush. To begin a new transaction with this Session, first issue
> Session.rollback(). Original exception was: (psycopg2.IntegrityError)
> duplicate key value violates unique constraint "task_fail_pkey"
> /var/log/airflow/airflow_scheduler_err.log.10-DETAIL: Key (task_id, dag_id,
> execution_date)=(test_dag, test_task, 2016-11-30 00:00:00) already exists.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1001) Landing Time shows "unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'" on example_subdag_operator
[
https://issues.apache.org/jira/browse/AIRFLOW-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bolke de Bruin updated AIRFLOW-1001:
Priority: Blocker (was: Major)
> Landing Time shows "unsupported operand type(s) for -: 'datetime.datetime'
> and 'NoneType'" on example_subdag_operator
> -
>
> Key: AIRFLOW-1001
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1001
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, ui
>Affects Versions: 1.8.0, 1.8.0rc5
>Reporter: Ruslan Dautkhanov
>Priority: Blocker
>
> Sample DAG example_subdag_operator shows below Oops page
> on "Landing Times" tab
> {noformat}
> ---
> Traceback (most recent call last):
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/flask/app.py",
> line 1988, in wsgi_app
> response = self.full_dispatch_request()
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/flask/app.py",
> line 1641, in full_dispatch_request
> rv = self.handle_user_exception(e)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/flask/app.py",
> line 1544, in handle_user_exception
> reraise(exc_type, exc_value, tb)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/flask/app.py",
> line 1639, in full_dispatch_request
> rv = self.dispatch_request()
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/flask/app.py",
> line 1625, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/flask_admin/base.py",
> line 69, in inner
> return self._run_view(f, *args, **kwargs)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/flask_admin/base.py",
> line 368, in _run_view
> return fn(self, *args, **kwargs)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/flask_login.py",
> line 755, in decorated_view
> return func(*args, **kwargs)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/www/utils.py",
> line 125, in wrapper
> return f(*args, **kwargs)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/www/views.py",
> line 1560, in landing_times
> secs = (ti.end_date - ts).total_seconds()
> TypeError: unsupported operand type(s) for -: 'datetime.datetime' and
> 'NoneType'
> {noformat}
> http://somehost:18111/admin/airflow/landing_times?root==30_id=example_subdag_operator
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1003) DAG status flips erraticly
[ https://issues.apache.org/jira/browse/AIRFLOW-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated AIRFLOW-1003: Component/s: ui > DAG status flips erraticly > -- > > Key: AIRFLOW-1003 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1003 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: Airflow 1.8, 1.8.0rc5 >Reporter: Ruslan Dautkhanov > > Created a flow based on sample tutorial > https://airflow.incubator.apache.org/tutorial.html > (just changed dag-id to 'turorial-RD'). > See a 10 seconds video on this behavior: > http://screencast-o-matic.com/watch/cbebrn6kBw > Notice last DAG 'turorial-RD' keeps changing state > (it loses link and all the buttons and icons to show status > of the dag runs / tasks etc), then links and icons show back up. > In fact, during that short 10 seconds period it transitioned > that state 4 times (from "disabled" to a normal DAG). > There were no changes happenning in the system - I was just clicking > refresh in browser from time to time. All DAGs were disabled while > this was happening. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-1003) DAG status flips erraticly
[ https://issues.apache.org/jira/browse/AIRFLOW-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940674#comment-15940674 ] Bolke de Bruin commented on AIRFLOW-1003: - this is due to the fact how the webservers load the dag bag and are multi process. it will go away after a while by itself > DAG status flips erraticly > -- > > Key: AIRFLOW-1003 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1003 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: Airflow 1.8, 1.8.0rc5 >Reporter: Ruslan Dautkhanov > > Created a flow based on sample tutorial > https://airflow.incubator.apache.org/tutorial.html > (just changed dag-id to 'turorial-RD'). > See a 10 seconds video on this behavior: > http://screencast-o-matic.com/watch/cbebrn6kBw > Notice last DAG 'turorial-RD' keeps changing state > (it loses link and all the buttons and icons to show status > of the dag runs / tasks etc), then links and icons show back up. > In fact, during that short 10 seconds period it transitioned > that state 4 times (from "disabled" to a normal DAG). > There were no changes happenning in the system - I was just clicking > refresh in browser from time to time. All DAGs were disabled while > this was happening. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1013) airflow/jobs.py:manage_slas() exception for @once dag
[
https://issues.apache.org/jira/browse/AIRFLOW-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bolke de Bruin updated AIRFLOW-1013:
Priority: Blocker (was: Major)
> airflow/jobs.py:manage_slas() exception for @once dag
> -
>
> Key: AIRFLOW-1013
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1013
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: Airflow 1.8
>Reporter: Ruslan Dautkhanov
>Assignee: Siddharth Anand
>Priority: Blocker
> Labels: dagrun, once, scheduler, sla
>
> Getting following exception
> {noformat}
> [2017-03-19 20:16:25,786] {jobs.py:354} DagFileProcessor2638 ERROR - Got an
> exception! Propagating...
> Traceback (most recent call last):
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/jobs.py",
> line 346, in helper
> pickle_dags)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/utils/db.py",
> line 53, in wrapper
> result = func(*args, **kwargs)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/jobs.py",
> line 1581, in process_file
> self._process_dags(dagbag, dags, ti_keys_to_schedule)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/jobs.py",
> line 1175, in _process_dags
> self.manage_slas(dag)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/utils/db.py",
> line 53, in wrapper
> result = func(*args, **kwargs)
> File
> "/opt/cloudera/parcels/Anaconda/lib/python2.7/site-packages/airflow/jobs.py",
> line 595, in manage_slas
> while dttm < datetime.now():
> TypeError: can't compare datetime.datetime to NoneType
> {noformat}
> Exception is in airflow/jobs.py:manage_slas() :
> https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/jobs.py#L595
> {code}
> ts = datetime.now()
> SlaMiss = models.SlaMiss
> for ti in max_tis:
> task = dag.get_task(ti.task_id)
> dttm = ti.execution_date
> if task.sla:
> dttm = dag.following_schedule(dttm)
> >>> while dttm < datetime.now(): <<< here
> following_schedule = dag.following_schedule(dttm)
> if following_schedule + task.sla < datetime.now():
> session.merge(models.SlaMiss(
> task_id=ti.task_id,
> {code}
> It seems that dag.following_schedule() returns None for @once dag?
> Here's how dag is defined:
> {code}
> main_dag = DAG(
> dag_id = 'DISCOVER-Oracle-Load',
> default_args = default_args,
> user_defined_macros= dag_macros,
> start_date = datetime.now(),
> catchup= False,
> schedule_interval = '@once',
> concurrency= 2,
> max_active_runs= 1,
> dagrun_timeout = timedelta(days=4),
> )
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1015) TreeView displayed over task instances
[ https://issues.apache.org/jira/browse/AIRFLOW-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated AIRFLOW-1015: Priority: Minor (was: Major) > TreeView displayed over task instances > -- > > Key: AIRFLOW-1015 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1015 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Affects Versions: Airflow 1.8 >Reporter: Ruslan Dautkhanov >Priority: Minor > Labels: ui > Attachments: TreeView-bug.png > > > See screnshot: > !TreeView-bug.png! > It would be nice if first TI horizontal offset would automatically shift to > the right, depending on how many tasks are in longest branch to the left. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1015) TreeView displayed over task instances
[ https://issues.apache.org/jira/browse/AIRFLOW-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated AIRFLOW-1015: Issue Type: Improvement (was: Bug) > TreeView displayed over task instances > -- > > Key: AIRFLOW-1015 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1015 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Affects Versions: Airflow 1.8 >Reporter: Ruslan Dautkhanov > Labels: ui > Attachments: TreeView-bug.png > > > See screnshot: > !TreeView-bug.png! > It would be nice if first TI horizontal offset would automatically shift to > the right, depending on how many tasks are in longest branch to the left. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AIRFLOW-1037) Webserver can't fetch large log files from Remote worker
Jorge Alpedrinha Ramos created AIRFLOW-1037:
---
Summary: Webserver can't fetch large log files from Remote worker
Key: AIRFLOW-1037
URL: https://issues.apache.org/jira/browse/AIRFLOW-1037
Project: Apache Airflow
Issue Type: Bug
Components: celery, logging, webserver
Affects Versions: Airflow 1.8
Environment: Running airflow with docker using CeleryExecutor
Reporter: Jorge Alpedrinha Ramos
When tasks logs are too big, the webserver isn't able to fetch them. This may
be related with https://github.com/kennethreitz/requests/issues/2422.
I've recovered a stacktrace from the worker:
{code}
worker_1 | [2017-03-24 10:29:39,574] {_internal.py:87} INFO -
172.18.0.8 - - [24/Mar/2017 10:29:39] "GET
/log/rates.ticker-to-analytics/parse-syslog/2017-01-12T07:45:00 HTTP/1.1" 200 -
worker_1 | Traceback (most recent call last):
worker_1 | File "/usr/local/bin/airflow", line 28, in
worker_1 | args.func(args)
worker_1 | File
"/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 858, in
serve_logs
worker_1 | host='0.0.0.0', port=WORKER_LOG_SERVER_PORT)
worker_1 | File
"/usr/local/lib/python2.7/site-packages/flask/app.py", line 843, in run
worker_1 | run_simple(host, port, self, **options)
worker_1 | File
"/usr/local/lib/python2.7/site-packages/werkzeug/serving.py", line 736, in
run_simple
worker_1 | inner()
worker_1 | File
"/usr/local/lib/python2.7/site-packages/werkzeug/serving.py", line 699, in inner
worker_1 | srv.serve_forever()
worker_1 | File
"/usr/local/lib/python2.7/site-packages/werkzeug/serving.py", line 536, in
serve_forever
worker_1 | HTTPServer.serve_forever(self)
worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line 233,
in serve_forever
worker_1 | self._handle_request_noblock()
worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line 292,
in _handle_request_noblock
worker_1 | self.handle_error(request, client_address)
worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line 290,
in _handle_request_noblock
worker_1 | self.process_request(request, client_address)
worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line 318,
in process_request
worker_1 | self.finish_request(request, client_address)
worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line 331,
in finish_request
worker_1 | self.RequestHandlerClass(request, client_address, self)
worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line 654,
in __init__
worker_1 | self.finish()
worker_1 | File "/usr/local/lib/python2.7/SocketServer.py", line 713,
in finish
worker_1 | self.wfile.close()
worker_1 | File "/usr/local/lib/python2.7/socket.py", line 283, in
close
worker_1 | self.flush()
worker_1 | File "/usr/local/lib/python2.7/socket.py", line 307, in
flush
worker_1 |
self._sock.sendall(view[write_offset:write_offset+buffer_size])
worker_1 | socket.error: [Errno 32] Broken pipe
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
