[GitHub] [airflow] ashb commented on issue #7363: [AIRFLOW-6730] Use total_seconds instead of seconds
ashb commented on issue #7363: [AIRFLOW-6730] Use total_seconds instead of seconds URL: https://github.com/apache/airflow/pull/7363#issuecomment-582528294 :man_facepalming: Guess we've just been lucky and the default timeouts don't fire. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
ashb commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582527830 @mik-laj has a simiarl PR (but it keeps it as a GET which doesn't fix this problem) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] davlum commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S worker pod to be configured from JSON/YAML file
davlum commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S
worker pod to be configured from JSON/YAML file
URL: https://github.com/apache/airflow/pull/6230#discussion_r375409435
##
File path: airflow/example_dags/example_kubernetes_executor_config.py
##
@@ -87,8 +87,8 @@ def test_volume_mount():
python_callable=print_stuff,
executor_config={
"KubernetesExecutor": {
-"securityContext": {
-"runAsUser": 1000
+"labels": {
+"runAsUser": "1000"
Review comment:
Exactly, if you're in support of removing `securityContext` I'll change [the
comment](https://github.com/apache/airflow/blob/0499bab86e5045fd18683e64ae471d06ad6c806f/airflow/example_dags/example_kubernetes_executor_config.py#L84)
as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7363: [AIRFLOW-6730] Use total_seconds instead of seconds
ashb commented on a change in pull request #7363: [AIRFLOW-6730] Use total_seconds instead of seconds URL: https://github.com/apache/airflow/pull/7363#discussion_r375409336 ## File path: airflow/providers/google/cloud/operators/postgres_to_gcs.py ## @@ -94,7 +94,7 @@ def convert_type(self, value, schema_type): return datetime.timedelta( hours=formated_time.tm_hour, minutes=formated_time.tm_min, -seconds=formated_time.tm_sec).seconds +seconds=formated_time.tm_sec).total_seconds() Review comment: This one is I think wrong. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor
codecov-io edited a comment on issue #7163: [AIRFLOW-6542] add spark-on-k8s operator/hook/sensor URL: https://github.com/apache/airflow/pull/7163#issuecomment-574641714 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7163?src=pr=h1) Report > Merging [#7163](https://codecov.io/gh/apache/airflow/pull/7163?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4bf967cacb5d756a4d4815ae11acf9ce5b59d77e?src=pr=desc) will **decrease** coverage by `53.6%`. > The diff coverage is `2.54%`. [](https://codecov.io/gh/apache/airflow/pull/7163?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7163 +/- ## === - Coverage 86.35% 32.74% -53.61% === Files 871 874+3 Lines 4062740731 +104 === - Hits3508313339-21744 - Misses 554427392+21848 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7163?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/7163/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `53.52% <ø> (-41.55%)` | :arrow_down: | | [...providers/cncf/kubernetes/hooks/kubernetes\_hook.py](https://codecov.io/gh/apache/airflow/pull/7163/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL2hvb2tzL2t1YmVybmV0ZXNfaG9vay5weQ==) | `0% <0%> (ø)` | | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7163/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `25.53% <0%> (-50.54%)` | :arrow_down: | | [.../kubernetes/operators/spark\_kubernetes\_operator.py](https://codecov.io/gh/apache/airflow/pull/7163/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9zcGFya19rdWJlcm5ldGVzX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | | | [.../example\_dags/example\_spark\_kubernetes\_operator.py](https://codecov.io/gh/apache/airflow/pull/7163/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL2V4YW1wbGVfZGFncy9leGFtcGxlX3NwYXJrX2t1YmVybmV0ZXNfb3BlcmF0b3IucHk=) | `0% <0%> (ø)` | | | [...cncf/kubernetes/sensors/spark\_kubernetes\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7163/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL3NlbnNvcnMvc3Bhcmtfa3ViZXJuZXRlc19zZW5zb3IucHk=) | `0% <0%> (ø)` | | | [airflow/www/forms.py](https://codecov.io/gh/apache/airflow/pull/7163/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvZm9ybXMucHk=) | `92.45% <100%> (-7.55%)` | :arrow_down: | | [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/7163/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [...ing\_platform/example\_dags/example\_display\_video.py](https://codecov.io/gh/apache/airflow/pull/7163/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kaXNwbGF5X3ZpZGVvLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/hooks/vertica\_hook.py](https://codecov.io/gh/apache/airflow/pull/7163/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3ZlcnRpY2FfaG9vay5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | ... and [745 more](https://codecov.io/gh/apache/airflow/pull/7163/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7163?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7163?src=pr=footer). Last update [4bf967c...a6255e6](https://codecov.io/gh/apache/airflow/pull/7163?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-6736) Kubernetes Git test uses always apache/airflow:master as source of files
[ https://issues.apache.org/jira/browse/AIRFLOW-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-6736. --- Fix Version/s: 2.0.0 Resolution: Fixed > Kubernetes Git test uses always apache/airflow:master as source of files > > > Key: AIRFLOW-6736 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6736 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 2.0.0 > > > This makes it impossible to test how the Kubernetes GitSync test will behave > when merged. The git sync should always be done with the original repo/branch > or from the TRAVIS_BRANCH if this is a push build -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6736) Kubernetes Git test uses always apache/airflow:master as source of files
[ https://issues.apache.org/jira/browse/AIRFLOW-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030966#comment-17030966 ] ASF subversion and git services commented on AIRFLOW-6736: -- Commit 675cc6a83a3877573f4f4a5afef2d0673be9dfbd in airflow's branch refs/heads/master from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=675cc6a ] [AIRFLOW-6736] Fix the repo/branch that is used in PRs for git sync (#7368) Kubernetes Git test uses always apache/airflow:master as source of files This made it impossible to test how the Kubernetes GitSync test will behave when merged. The git sync should always be done with the original repo/branch or from the TRAVIS_BRANCH if this is a push build > Kubernetes Git test uses always apache/airflow:master as source of files > > > Key: AIRFLOW-6736 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6736 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 2.0.0 > > > This makes it impossible to test how the Kubernetes GitSync test will behave > when merged. The git sync should always be done with the original repo/branch > or from the TRAVIS_BRANCH if this is a push build -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031064#comment-17031064
]
Chinmay Joshi commented on AIRFLOW-2516:
Any progress on this? We are also facing this issue with stacktrace as below.
{code:java}
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/airflow/models/taskinstance.py",
line 930, in _run_raw_task
result = task_copy.execute(context=context)
File
"/usr/local/lib/python3.6/dist-packages/airflow/operators/subdag_operator.py",
line 102, in execute
executor=self.executor)
File "/usr/local/lib/python3.6/dist-packages/airflow/models/dag.py", line
1284, in run
job.run()
File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/base_job.py", line
222, in run
self._execute()
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/db.py", line 74,
in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/backfill_job.py",
line 769, in _execute
session=session)
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/db.py", line 70,
in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/backfill_job.py",
line 699, in _execute_for_run_dates
session=session)
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/db.py", line 70,
in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/backfill_job.py",
line 586, in _process_backfill_task_instances
_per_task_process(task, key, ti)
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/db.py", line 74,
in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/backfill_job.py",
line 508, in _per_task_process
session.commit()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
1036, in commit
self.transaction.commit()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
503, in commit
self._prepare_impl()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
482, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
2479, in flush
self._flush(objects)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
2617, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/langhelpers.py",
line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line
153, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
2577, in _flush
flush_context.execute()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/unitofwork.py",
line 422, in execute
rec.execute(self)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/unitofwork.py",
line 589, in execute
uow,
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/persistence.py",
line 236, in save_obj
update,
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/persistence.py",
line 996, in _emit_update_statements
statement, multiparams
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line
982, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/sql/elements.py",
line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line
1101, in _execute_clauseelement
distilled_params,
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line
1250, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line
1476, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line
398, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line
152, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line
1246, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py",
line 581, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.6/dist-packages/MySQLdb/cursors.py", line 209,
in execute
res = self._query(query)
File
[GitHub] [airflow] ashb commented on a change in pull request #7232: [AIRFLOW-6569] Flush pending Sentry exceptions before exiting forked process
ashb commented on a change in pull request #7232: [AIRFLOW-6569] Flush pending Sentry exceptions before exiting forked process URL: https://github.com/apache/airflow/pull/7232#discussion_r375402282 ## File path: airflow/sentry.py ## @@ -149,6 +154,10 @@ def wrapper(task_instance, *args, session=None, **kwargs): return wrapper +def flush(self): +from sentry_sdk import Hub +Hub.current.flush() Review comment: Should we be calling https://github.com/getsentry/sentry-python/blob/68ddcf459078f892ee25fbc354c9044057fbd0ef/sentry_sdk/api.py#L221-L229 instead? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7232: [AIRFLOW-6569] Flush pending Sentry exceptions before exiting forked process
ashb commented on a change in pull request #7232: [AIRFLOW-6569] Flush pending Sentry exceptions before exiting forked process URL: https://github.com/apache/airflow/pull/7232#discussion_r375402889 ## File path: airflow/sentry.py ## @@ -149,6 +154,10 @@ def wrapper(task_instance, *args, session=None, **kwargs): return wrapper +def flush(self): +from sentry_sdk import Hub +Hub.current.flush() Review comment: Looks like that was added in 0.7.3 and we already depend on >=0.8.0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
ashb commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582527590 Rather than a JSON body more "resty" would be to have this be a "form post" -- i.e. application/x-www-form-urlencoded body. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] madison-ookla commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
madison-ookla commented on issue #7364: [AIRFLOW-6728] Change various DAG info
methods to POST
URL: https://github.com/apache/airflow/pull/7364#issuecomment-582555070
Although it looks like I could do `request.form.getlist('dag_ids')` inside
the actual request handlers for a regular form POST, which is straightforward.
I'd just have to add the form encoding onto all of the requests. I could go
either way - let me know if JSON POST or form POST is preferred.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk merged pull request #7368: [AIRFLOW-6736] Fix the repo/branch that is used in PRs for git sync
potiuk merged pull request #7368: [AIRFLOW-6736] Fix the repo/branch that is used in PRs for git sync URL: https://github.com/apache/airflow/pull/7368 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6736) Kubernetes Git test uses always apache/airflow:master as source of files
[ https://issues.apache.org/jira/browse/AIRFLOW-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030964#comment-17030964 ] ASF GitHub Bot commented on AIRFLOW-6736: - potiuk commented on pull request #7368: [AIRFLOW-6736] Fix the repo/branch that is used in PRs for git sync URL: https://github.com/apache/airflow/pull/7368 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Kubernetes Git test uses always apache/airflow:master as source of files > > > Key: AIRFLOW-6736 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6736 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > > This makes it impossible to test how the Kubernetes GitSync test will behave > when merged. The git sync should always be done with the original repo/branch > or from the TRAVIS_BRANCH if this is a push build -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #7353: [AIRFLOW-6685] Data Quality Check operators
codecov-io edited a comment on issue #7353: [AIRFLOW-6685] Data Quality Check operators URL: https://github.com/apache/airflow/pull/7353#issuecomment-582190961 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7353?src=pr=h1) Report > Merging [#7353](https://codecov.io/gh/apache/airflow/pull/7353?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/2195bb45be00f0bdaadf4b7009db40f360f5455b?src=pr=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `84.61%`. [](https://codecov.io/gh/apache/airflow/pull/7353?src=pr=tree) ```diff @@Coverage Diff@@ ## master#7353 +/- ## = - Coverage 86.35% 86.06% -0.3% = Files 871 872 +1 Lines 4062740692 +65 = - Hits3508335021 -62 - Misses 5544 5671+127 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7353?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/data\_quality\_operator.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZGF0YV9xdWFsaXR5X29wZXJhdG9yLnB5) | `84.61% <84.61%> (ø)` | | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `76.08% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `100% <0%> (ø)` | :arrow_up: | | [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `85.29% <0%> (ø)` | :arrow_up: | | ... and [10 more](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7353?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7353?src=pr=footer). Last update [2195bb4...f6c003b](https://codecov.io/gh/apache/airflow/pull/7353?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6740) Setting webserver.proxy_fix_num_proxies raises exception
Sebastian Brandt created AIRFLOW-6740:
-
Summary: Setting webserver.proxy_fix_num_proxies raises exception
Key: AIRFLOW-6740
URL: https://issues.apache.org/jira/browse/AIRFLOW-6740
Project: Apache Airflow
Issue Type: Bug
Components: webserver
Affects Versions: 1.10.7
Reporter: Sebastian Brandt
Assignee: Sebastian Brandt
Following configuration variable always leads to an exception:
{code}
[webserver]
proxy_fix_num_proxies =
{code}
Exception:
{code}
Error handling request /home
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/sync.py", line
135, in handle
self.handle_request(listener, req, client, addr)
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/sync.py", line
176, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File
"/usr/local/lib/python3.7/site-packages/werkzeug/middleware/dispatcher.py",
line 66, in __call__
return app(environ, start_response)
File
"/usr/local/lib/python3.7/site-packages/sentry_sdk/integrations/flask.py", line
70, in sentry_patched_wsgi_app
environ, start_response
File
"/usr/local/lib/python3.7/site-packages/sentry_sdk/integrations/wsgi.py", line
126, in __call__
reraise(*_capture_exception(hub))
File "/usr/local/lib/python3.7/site-packages/sentry_sdk/_compat.py", line 57,
in reraise
raise value
File
"/usr/local/lib/python3.7/site-packages/sentry_sdk/integrations/wsgi.py", line
122, in __call__
_sentry_start_response, start_response, span
File
"/usr/local/lib/python3.7/site-packages/sentry_sdk/integrations/flask.py", line
69, in
return SentryWsgiMiddleware(lambda *a, **kw: old_app(self, *a, **kw))(
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2463, in
__call__
return self.wsgi_app(environ, start_response)
File
"/usr/local/lib/python3.7/site-packages/werkzeug/middleware/proxy_fix.py", line
195, in __call__
x_for = self._get_trusted_comma(self.x_for,
environ_get("HTTP_X_FORWARDED_FOR"))
File
"/usr/local/lib/python3.7/site-packages/werkzeug/middleware/proxy_fix.py", line
166, in _get_trusted_comma
if len(values) >= trusted:
TypeError: '>=' not supported between instances of 'int' and 'str'
{code}
This is because the variable is retrieved as string from the config, which then
causes the above TypeError. It was almost fixed in
https://github.com/apache/airflow/pull/6901, but forgotten.
Since {{num_proxies}} is set last,
[jira] [Updated] (AIRFLOW-6741) Cant run DAG using REST api POST call
[
https://issues.apache.org/jira/browse/AIRFLOW-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
raj ram updated AIRFLOW-6741:
-
Description:
Per [https://airflow.apache.org/docs/1.10.1/api.html], I should be able to
start a DAG through a POST call
I am not able to POST a request to airflow 1.10.1 to start a DAG. When I did a
POST using CURL
curl -X POST [http://localhost:8080/api/experimental/dags/]/dag_runs -H
'Cache-Control: no-cache' -H 'Content-Type: application/json' -d {}
I got this error. curl: (52) Empty reply from server
I was able to POST to other REST endpoints. Also, I am able to POST to airflow
version 1.10.5 using the exact same command.
Here is the airflow.cfg API setting
[cli]
# In what way should the cli access the API. The LocalClient will use the
# database directly, while the json_client will use the api running on the
# webserver
api_client = airflow.api.client.local_client
# If you set web_server_url_prefix, do NOT forget to append it here, ex:
# endpoint_url = [http://localhost:8080/myroot]
# So api will look like: [http://localhost:8080/myroot/api/experimental/]...
endpoint_url = [http://localhost:$AIRFLOW_PORT|http://localhost:$AIRFLOW_PORT/]
[api]
# How to authenticate users of the API
auth_backend = airflow.api.auth.backend.default
was:
I am not able to POST a request to airflow 1.10.1 to start a DAG. When I did a
POST using CURL
curl -X POST http://localhost:8080/api/experimental/dags//dag_runs -H
'Cache-Control: no-cache' -H 'Content-Type: application/json' -d {}
I got this error. curl: (52) Empty reply from server
I was able to POST to other REST endpoints. Also, I am able to POST to airflow
version 1.10.5 using the exact same command.
Here is the airflow.cfg API setting
[cli]
# In what way should the cli access the API. The LocalClient will use the
# database directly, while the json_client will use the api running on the
# webserver
api_client = airflow.api.client.local_client
# If you set web_server_url_prefix, do NOT forget to append it here, ex:
# endpoint_url = http://localhost:8080/myroot
# So api will look like: http://localhost:8080/myroot/api/experimental/...
endpoint_url = http://localhost:$AIRFLOW_PORT
[api]
# How to authenticate users of the API
auth_backend = airflow.api.auth.backend.default
> Cant run DAG using REST api POST call
> -
>
> Key: AIRFLOW-6741
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6741
> Project: Apache Airflow
> Issue Type: Bug
> Components: api
>Affects Versions: 1.10.1
> Environment: AWS Linux 1
>Reporter: raj ram
>Priority: Major
>
> Per [https://airflow.apache.org/docs/1.10.1/api.html], I should be able to
> start a DAG through a POST call
> I am not able to POST a request to airflow 1.10.1 to start a DAG. When I did
> a POST using CURL
> curl -X POST [http://localhost:8080/api/experimental/dags/]/dag_runs
> -H 'Cache-Control: no-cache' -H 'Content-Type: application/json' -d {}
>
> I got this error. curl: (52) Empty reply from server
>
> I was able to POST to other REST endpoints. Also, I am able to POST to
> airflow version 1.10.5 using the exact same command.
>
> Here is the airflow.cfg API setting
>
> [cli]
> # In what way should the cli access the API. The LocalClient will use the
> # database directly, while the json_client will use the api running on the
> # webserver
> api_client = airflow.api.client.local_client
> # If you set web_server_url_prefix, do NOT forget to append it here, ex:
> # endpoint_url = [http://localhost:8080/myroot]
> # So api will look like: [http://localhost:8080/myroot/api/experimental/]...
> endpoint_url =
> [http://localhost:$AIRFLOW_PORT|http://localhost:$AIRFLOW_PORT/]
> [api]
> # How to authenticate users of the API
> auth_backend = airflow.api.auth.backend.default
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031064#comment-17031064
]
Chinmay Joshi edited comment on AIRFLOW-2516 at 2/5/20 9:48 PM:
Any progress on this? We are also facing this issue with quite frequently with
consistent stacktrace as below.
{code:java}
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/airflow/models/taskinstance.py",
line 930, in _run_raw_task
result = task_copy.execute(context=context)
File
"/usr/local/lib/python3.6/dist-packages/airflow/operators/subdag_operator.py",
line 102, in execute
executor=self.executor)
File "/usr/local/lib/python3.6/dist-packages/airflow/models/dag.py", line
1284, in run
job.run()
File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/base_job.py", line
222, in run
self._execute()
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/db.py", line 74,
in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/backfill_job.py",
line 769, in _execute
session=session)
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/db.py", line 70,
in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/backfill_job.py",
line 699, in _execute_for_run_dates
session=session)
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/db.py", line 70,
in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/backfill_job.py",
line 586, in _process_backfill_task_instances
_per_task_process(task, key, ti)
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/db.py", line 74,
in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/backfill_job.py",
line 508, in _per_task_process
session.commit()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
1036, in commit
self.transaction.commit()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
503, in commit
self._prepare_impl()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
482, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
2479, in flush
self._flush(objects)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
2617, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/langhelpers.py",
line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line
153, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
2577, in _flush
flush_context.execute()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/unitofwork.py",
line 422, in execute
rec.execute(self)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/unitofwork.py",
line 589, in execute
uow,
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/persistence.py",
line 236, in save_obj
update,
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/persistence.py",
line 996, in _emit_update_statements
statement, multiparams
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line
982, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/sql/elements.py",
line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line
1101, in _execute_clauseelement
distilled_params,
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line
1250, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line
1476, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line
398, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line
152, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line
1246, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py",
line 581, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.6/dist-packages/MySQLdb/cursors.py", line 209,
in execute
[GitHub] [airflow] ashb commented on issue #7232: [AIRFLOW-6569] Flush pending Sentry exceptions before exiting forked process
ashb commented on issue #7232: [AIRFLOW-6569] Flush pending Sentry exceptions before exiting forked process URL: https://github.com/apache/airflow/pull/7232#issuecomment-582522050 Does this need a specific version of the sentry library to work? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7372: [AIRFLOW-6738] Upload container diagnostics on failure
codecov-io commented on issue #7372: [AIRFLOW-6738] Upload container diagnostics on failure URL: https://github.com/apache/airflow/pull/7372#issuecomment-582524422 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=h1) Report > Merging [#7372](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/2195bb45be00f0bdaadf4b7009db40f360f5455b?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7372 +/- ## == - Coverage 86.35% 86.35% -0.01% == Files 871 871 Lines 4062740627 == - Hits3508335082 -1 - Misses 5544 5545 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7372/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=footer). Last update [2195bb4...3425505](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
mik-laj commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582548229 I would like to point out that this endpoint belongs to the internal API so it didn't have to be pretty. This should work well. Only public APIs must be beautiful, standardized and backward compatible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk edited a comment on issue #7369: [AIRFLOW-6739] Update tutorial.rst
potiuk edited a comment on issue #7369: [AIRFLOW-6739] Update tutorial.rst URL: https://github.com/apache/airflow/pull/7369#issuecomment-582586950 For the future - doc only change can have [AIRFLOW-] and no JIRA issue :). The "[AIRFLOW-]" literal with X's is necessary :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] sbrandtb commented on issue #7359: [AIRFLOW-6740] Remove Undocumented, Deprecated, Dysfunctional PROXY_FIX_NUM_PROXIES
sbrandtb commented on issue #7359: [AIRFLOW-6740] Remove Undocumented, Deprecated, Dysfunctional PROXY_FIX_NUM_PROXIES URL: https://github.com/apache/airflow/pull/7359#issuecomment-582614074 @ashb weird. One CI job failed with "fatal: unable to access 'https://github.com/apache/airflow.git/': Failed to connect to github.com port 443: Connection timed out". Looks like it could be a temporary issue. How to retrigger the CI? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6230: [AIRFLOW-5413] Allow K8S worker pod to be configured from JSON/YAML file
codecov-io edited a comment on issue #6230: [AIRFLOW-5413] Allow K8S worker pod to be configured from JSON/YAML file URL: https://github.com/apache/airflow/pull/6230#issuecomment-537654940 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@96f8343`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `97.95%`. [](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=tree) ```diff @@Coverage Diff@@ ## master#6230 +/- ## = Coverage ? 85.99% = Files ? 871 Lines ?40660 Branches ?0 = Hits ?34964 Misses? 5696 Partials ?0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=tree) | Coverage Δ | | |---|---|---| | [...example\_dags/example\_kubernetes\_executor\_config.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9rdWJlcm5ldGVzX2V4ZWN1dG9yX2NvbmZpZy5weQ==) | `72.22% <ø> (ø)` | | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `94.94% <100%> (ø)` | | | [airflow/kubernetes/pod\_generator.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9nZW5lcmF0b3IucHk=) | `96.5% <100%> (ø)` | | | [airflow/kubernetes/worker\_configuration.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3dvcmtlcl9jb25maWd1cmF0aW9uLnB5) | `99.29% <100%> (ø)` | | | [airflow/executors/kubernetes\_executor.py](https://codecov.io/gh/apache/airflow/pull/6230/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMva3ViZXJuZXRlc19leGVjdXRvci5weQ==) | `56.8% <80%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=footer). Last update [96f8343...8b51627](https://codecov.io/gh/apache/airflow/pull/6230?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] madison-ookla commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
madison-ookla commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582536934 > Rather than a JSON body more "resty" would be to have this be a "form post" -- i.e. application/x-www-form-urlencoded body. I can change the request to use a form post instead of JSON This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] robinedwards commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
robinedwards commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582550474 Fair point :-) Cheers for fixing @madison-ookla and apologies for any breakage my original change caused. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7312: [AIRFLOW-6695] Add DAG run configuration to UI
codecov-io edited a comment on issue #7312: [AIRFLOW-6695] Add DAG run configuration to UI URL: https://github.com/apache/airflow/pull/7312#issuecomment-582590185 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7312?src=pr=h1) Report > Merging [#7312](https://codecov.io/gh/apache/airflow/pull/7312?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/92c72f40df24aa9bec8a5bf5159bcd29c85698cf?src=pr=desc) will **increase** coverage by `0.13%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7312?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7312 +/- ## == + Coverage 86.21% 86.35% +0.13% == Files 871 871 Lines 4057040636 +66 == + Hits3497935092 +113 + Misses 5591 5544 -47 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7312?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.22% <100%> (+0.14%)` | :arrow_up: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `76.08% <0%> (ø)` | :arrow_up: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `96.66% <0%> (ø)` | :arrow_up: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `93.61% <0%> (-4.04%)` | :arrow_down: | | [...providers/google/cloud/example\_dags/example\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2djcy5weQ==) | `96.29% <0%> (-3.71%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.12% <0%> (+0.19%)` | :arrow_up: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `91.73% <0%> (+1.65%)` | :arrow_up: | | ... and [13 more](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7312?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7312?src=pr=footer). Last update [92c72f4...3da126e](https://codecov.io/gh/apache/airflow/pull/7312?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7312: [AIRFLOW-6695] Add DAG run configuration to UI
codecov-io commented on issue #7312: [AIRFLOW-6695] Add DAG run configuration to UI URL: https://github.com/apache/airflow/pull/7312#issuecomment-582590185 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7312?src=pr=h1) Report > Merging [#7312](https://codecov.io/gh/apache/airflow/pull/7312?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/92c72f40df24aa9bec8a5bf5159bcd29c85698cf?src=pr=desc) will **decrease** coverage by `0.23%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7312?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7312 +/- ## == - Coverage 86.21% 85.98% -0.24% == Files 871 871 Lines 4057040636 +66 == - Hits3497934942 -37 - Misses 5591 5694 +103 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7312?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.22% <100%> (+0.14%)` | :arrow_up: | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0%> (-64.16%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `39.28% <0%> (-60.72%)` | :arrow_down: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0%> (-45.66%)` | :arrow_down: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55% <0%> (-45%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `91.66% <0%> (-5%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `93.61% <0%> (-4.04%)` | :arrow_down: | | [...providers/google/cloud/example\_dags/example\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2djcy5weQ==) | `96.29% <0%> (-3.71%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `85.63% <0%> (-2.3%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `88.42% <0%> (-1.66%)` | :arrow_down: | | ... and [13 more](https://codecov.io/gh/apache/airflow/pull/7312/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7312?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7312?src=pr=footer). Last update [92c72f4...3da126e](https://codecov.io/gh/apache/airflow/pull/7312?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mikeclarke commented on a change in pull request #7232: [AIRFLOW-6569] Flush pending Sentry exceptions before exiting forked process
mikeclarke commented on a change in pull request #7232: [AIRFLOW-6569] Flush pending Sentry exceptions before exiting forked process URL: https://github.com/apache/airflow/pull/7232#discussion_r375482492 ## File path: airflow/sentry.py ## @@ -149,6 +154,10 @@ def wrapper(task_instance, *args, session=None, **kwargs): return wrapper +def flush(self): +from sentry_sdk import Hub +Hub.current.flush() Review comment: Good point, I'll update - thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] robinedwards commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
robinedwards commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582555954 There the FormData API for doing the encoding if your interested https://developer.mozilla.org/en-US/docs/Web/API/FormData/Using_FormData_Objects This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-6739) Include initdb step in tutorial
[ https://issues.apache.org/jira/browse/AIRFLOW-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-6739. --- Fix Version/s: 2.0.0 Resolution: Fixed > Include initdb step in tutorial > > > Key: AIRFLOW-6739 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6739 > Project: Apache Airflow > Issue Type: Improvement > Components: documentation >Affects Versions: master >Reporter: Clint Dow >Assignee: Clint Dow >Priority: Trivial > Labels: tutorial > Fix For: 2.0.0 > > Attachments: Screenshot 2020-02-05 at 10.43.51 AM.png > > Original Estimate: 2m > Remaining Estimate: 2m > > In the tutorial under the Command Line Metadata Validation heading, if the > user has not yet initialized the database tables, sqlite3 will throw an ugly > traceback about the missing slot_pool table. > Update the documentation to include the airflow initdb command prior to > listing dags and tasks, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6739) Include initdb step in tutorial
[ https://issues.apache.org/jira/browse/AIRFLOW-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030980#comment-17030980 ] ASF GitHub Bot commented on AIRFLOW-6739: - potiuk commented on pull request #7369: [AIRFLOW-6739] Update tutorial.rst URL: https://github.com/apache/airflow/pull/7369 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Include initdb step in tutorial > > > Key: AIRFLOW-6739 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6739 > Project: Apache Airflow > Issue Type: Improvement > Components: documentation >Affects Versions: master >Reporter: Clint Dow >Assignee: Clint Dow >Priority: Trivial > Labels: tutorial > Attachments: Screenshot 2020-02-05 at 10.43.51 AM.png > > Original Estimate: 2m > Remaining Estimate: 2m > > In the tutorial under the Command Line Metadata Validation heading, if the > user has not yet initialized the database tables, sqlite3 will throw an ugly > traceback about the missing slot_pool table. > Update the documentation to include the airflow initdb command prior to > listing dags and tasks, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk commented on issue #7369: [AIRFLOW-6739] Update tutorial.rst
potiuk commented on issue #7369: [AIRFLOW-6739] Update tutorial.rst URL: https://github.com/apache/airflow/pull/7369#issuecomment-582586950 For the future - doc only change can have [AIRFLOW-] and no JIRA issue :). The "[AIRFLOW-]" is needed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk merged pull request #7369: [AIRFLOW-6739] Update tutorial.rst
potiuk merged pull request #7369: [AIRFLOW-6739] Update tutorial.rst URL: https://github.com/apache/airflow/pull/7369 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #7369: [AIRFLOW-6739] Update tutorial.rst
boring-cyborg[bot] commented on issue #7369: [AIRFLOW-6739] Update tutorial.rst URL: https://github.com/apache/airflow/pull/7369#issuecomment-582586535 Awesome work, congrats on your first merged pull request! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6739) Include initdb step in tutorial
[ https://issues.apache.org/jira/browse/AIRFLOW-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030981#comment-17030981 ] ASF subversion and git services commented on AIRFLOW-6739: -- Commit 61455c69ddb5677ce02af626fbbeaf4bc29c15d3 in airflow's branch refs/heads/master from Clinton Dow [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=61455c6 ] [AIRFLOW-6739] Update tutorial.rst (#7369) Traceback from sqlite3.OperationalError: no such table: slot_pool when tutorial being followed on a new installation, include the initdb step > Include initdb step in tutorial > > > Key: AIRFLOW-6739 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6739 > Project: Apache Airflow > Issue Type: Improvement > Components: documentation >Affects Versions: master >Reporter: Clint Dow >Assignee: Clint Dow >Priority: Trivial > Labels: tutorial > Fix For: 2.0.0 > > Attachments: Screenshot 2020-02-05 at 10.43.51 AM.png > > Original Estimate: 2m > Remaining Estimate: 2m > > In the tutorial under the Command Line Metadata Validation heading, if the > user has not yet initialized the database tables, sqlite3 will throw an ugly > traceback about the missing slot_pool table. > Update the documentation to include the airflow initdb command prior to > listing dags and tasks, -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6741) Cant run DAG using REST api POST call
raj ram created AIRFLOW-6741:
Summary: Cant run DAG using REST api POST call
Key: AIRFLOW-6741
URL: https://issues.apache.org/jira/browse/AIRFLOW-6741
Project: Apache Airflow
Issue Type: Bug
Components: api
Affects Versions: 1.10.1
Environment: AWS Linux 1
Reporter: raj ram
I am not able to POST a request to airflow 1.10.1 to start a DAG. When I did a
POST using CURL
curl -X POST http://localhost:8080/api/experimental/dags//dag_runs -H
'Cache-Control: no-cache' -H 'Content-Type: application/json' -d {}
I got this error. curl: (52) Empty reply from server
I was able to POST to other REST endpoints. Also, I am able to POST to airflow
version 1.10.5 using the exact same command.
Here is the airflow.cfg API setting
[cli]
# In what way should the cli access the API. The LocalClient will use the
# database directly, while the json_client will use the api running on the
# webserver
api_client = airflow.api.client.local_client
# If you set web_server_url_prefix, do NOT forget to append it here, ex:
# endpoint_url = http://localhost:8080/myroot
# So api will look like: http://localhost:8080/myroot/api/experimental/...
endpoint_url = http://localhost:$AIRFLOW_PORT
[api]
# How to authenticate users of the API
auth_backend = airflow.api.auth.backend.default
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] potiuk opened a new pull request #7373: [AIRFLOW-XXXX] Fix location of kubernetes tests
potiuk opened a new pull request #7373: [AIRFLOW-] Fix location of kubernetes tests URL: https://github.com/apache/airflow/pull/7373 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk merged pull request #7373: [AIRFLOW-XXXX] Fix location of kubernetes tests
potiuk merged pull request #7373: [AIRFLOW-] Fix location of kubernetes tests URL: https://github.com/apache/airflow/pull/7373 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] madison-ookla commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
madison-ookla commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582550979 Thanks all for the feedback :) I would prefer to keep it as JSON just because it makes processing processing the request body a bit easier (rather than having to split a comma separated string). That said I don't do a lot of web-dev work so that practice may be commonplace. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] sbrandtb commented on issue #7359: [AIRFLOW-6740] Remove Undocumented, Deprecated, Dysfunctional PROXY_FIX_NUM_PROXIES
sbrandtb commented on issue #7359: [AIRFLOW-6740] Remove Undocumented, Deprecated, Dysfunctional PROXY_FIX_NUM_PROXIES URL: https://github.com/apache/airflow/pull/7359#issuecomment-582590951 @ashb Done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #7359: [AIRFLOW-6740] Remove Undocumented, Deprecated, Dysfunctional PROXY_FIX_NUM_PROXIES
mik-laj commented on issue #7359: [AIRFLOW-6740] Remove Undocumented, Deprecated, Dysfunctional PROXY_FIX_NUM_PROXIES URL: https://github.com/apache/airflow/pull/7359#issuecomment-582614894 I restarted job. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031064#comment-17031064
]
Chinmay Joshi edited comment on AIRFLOW-2516 at 2/5/20 9:51 PM:
Any progress on this? We are also facing this issue in SubDAGs quite frequently
with consistent stacktrace as below.
{code:java}
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/airflow/models/taskinstance.py",
line 930, in _run_raw_task
result = task_copy.execute(context=context)
File
"/usr/local/lib/python3.6/dist-packages/airflow/operators/subdag_operator.py",
line 102, in execute
executor=self.executor)
File "/usr/local/lib/python3.6/dist-packages/airflow/models/dag.py", line
1284, in run
job.run()
File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/base_job.py", line
222, in run
self._execute()
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/db.py", line 74,
in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/backfill_job.py",
line 769, in _execute
session=session)
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/db.py", line 70,
in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/backfill_job.py",
line 699, in _execute_for_run_dates
session=session)
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/db.py", line 70,
in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/backfill_job.py",
line 586, in _process_backfill_task_instances
_per_task_process(task, key, ti)
File "/usr/local/lib/python3.6/dist-packages/airflow/utils/db.py", line 74,
in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/backfill_job.py",
line 508, in _per_task_process
session.commit()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
1036, in commit
self.transaction.commit()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
503, in commit
self._prepare_impl()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
482, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
2479, in flush
self._flush(objects)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
2617, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/langhelpers.py",
line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line
153, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/session.py", line
2577, in _flush
flush_context.execute()
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/unitofwork.py",
line 422, in execute
rec.execute(self)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/unitofwork.py",
line 589, in execute
uow,
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/persistence.py",
line 236, in save_obj
update,
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/orm/persistence.py",
line 996, in _emit_update_statements
statement, multiparams
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line
982, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/sql/elements.py",
line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line
1101, in _execute_clauseelement
distilled_params,
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line
1250, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line
1476, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line
398, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line
152, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line
1246, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py",
line 581, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.6/dist-packages/MySQLdb/cursors.py", line 209,
in
[GitHub] [airflow] mik-laj commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
mik-laj commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582538062 Here is my PR: https://github.com/apache/airflow/pulls/mik-laj https://github.com/apache/airflow/pull/6855 But this PR is better. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj closed pull request #6855: [AIRFLOW-6307] Improve support for a special characters in DAG's name in stats view
mik-laj closed pull request #6855: [AIRFLOW-6307] Improve support for a special characters in DAG's name in stats view URL: https://github.com/apache/airflow/pull/6855 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] data-navigator commented on issue #7369: [AIRFLOW-6739] Update tutorial.rst
data-navigator commented on issue #7369: [AIRFLOW-6739] Update tutorial.rst URL: https://github.com/apache/airflow/pull/7369#issuecomment-582555689 Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] robinedwards edited a comment on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
robinedwards edited a comment on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582555954 There is the FormData API for doing the encoding if your interested https://developer.mozilla.org/en-US/docs/Web/API/FormData/Using_FormData_Objects This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-6741) Cant run DAG using REST api POST call
[
https://issues.apache.org/jira/browse/AIRFLOW-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
raj ram updated AIRFLOW-6741:
-
Description:
Per [https://airflow.apache.org/docs/1.10.1/api.html], I should be able to
start a DAG through a POST call. But I am not able to POST a request to
airflow 1.10.1 to start a DAG. When I did a POST using CURL
curl -X POST [http://localhost:8080/api/experimental/dags/]/dag_runs -H
'Cache-Control: no-cache' -H 'Content-Type: application/json' -d {}
I got this error. curl: (52) Empty reply from server
I was able to POST to other REST endpoints. Also, I am able to POST to airflow
version 1.10.5 using the exact same command.
Here is the airflow.cfg API setting
[cli]
# In what way should the cli access the API. The LocalClient will use the
# database directly, while the json_client will use the api running on the
# webserver
api_client = airflow.api.client.local_client
# If you set web_server_url_prefix, do NOT forget to append it here, ex:
# endpoint_url = [http://localhost:8080/myroot]
# So api will look like: [http://localhost:8080/myroot/api/experimental/]...
endpoint_url = [http://localhost:$AIRFLOW_PORT|http://localhost:$AIRFLOW_PORT/]
[api]
# How to authenticate users of the API
auth_backend = airflow.api.auth.backend.default
was:
Per [https://airflow.apache.org/docs/1.10.1/api.html], I should be able to
start a DAG through a POST call
I am not able to POST a request to airflow 1.10.1 to start a DAG. When I did a
POST using CURL
curl -X POST [http://localhost:8080/api/experimental/dags/]/dag_runs -H
'Cache-Control: no-cache' -H 'Content-Type: application/json' -d {}
I got this error. curl: (52) Empty reply from server
I was able to POST to other REST endpoints. Also, I am able to POST to airflow
version 1.10.5 using the exact same command.
Here is the airflow.cfg API setting
[cli]
# In what way should the cli access the API. The LocalClient will use the
# database directly, while the json_client will use the api running on the
# webserver
api_client = airflow.api.client.local_client
# If you set web_server_url_prefix, do NOT forget to append it here, ex:
# endpoint_url = [http://localhost:8080/myroot]
# So api will look like: [http://localhost:8080/myroot/api/experimental/]...
endpoint_url = [http://localhost:$AIRFLOW_PORT|http://localhost:$AIRFLOW_PORT/]
[api]
# How to authenticate users of the API
auth_backend = airflow.api.auth.backend.default
> Cant run DAG using REST api POST call
> -
>
> Key: AIRFLOW-6741
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6741
> Project: Apache Airflow
> Issue Type: Bug
> Components: api
>Affects Versions: 1.10.1
> Environment: AWS Linux 1
>Reporter: raj ram
>Priority: Major
>
> Per [https://airflow.apache.org/docs/1.10.1/api.html], I should be able to
> start a DAG through a POST call. But I am not able to POST a request to
> airflow 1.10.1 to start a DAG. When I did a POST using CURL
> curl -X POST [http://localhost:8080/api/experimental/dags/]/dag_runs
> -H 'Cache-Control: no-cache' -H 'Content-Type: application/json' -d {}
>
> I got this error. curl: (52) Empty reply from server
>
> I was able to POST to other REST endpoints. Also, I am able to POST to
> airflow version 1.10.5 using the exact same command.
>
> Here is the airflow.cfg API setting
>
> [cli]
> # In what way should the cli access the API. The LocalClient will use the
> # database directly, while the json_client will use the api running on the
> # webserver
> api_client = airflow.api.client.local_client
> # If you set web_server_url_prefix, do NOT forget to append it here, ex:
> # endpoint_url = [http://localhost:8080/myroot]
> # So api will look like: [http://localhost:8080/myroot/api/experimental/]...
> endpoint_url =
> [http://localhost:$AIRFLOW_PORT|http://localhost:$AIRFLOW_PORT/]
> [api]
> # How to authenticate users of the API
> auth_backend = airflow.api.auth.backend.default
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] ashb commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S worker pod to be configured from JSON/YAML file
ashb commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S
worker pod to be configured from JSON/YAML file
URL: https://github.com/apache/airflow/pull/6230#discussion_r375410637
##
File path: airflow/example_dags/example_kubernetes_executor_config.py
##
@@ -87,8 +87,8 @@ def test_volume_mount():
python_callable=print_stuff,
executor_config={
"KubernetesExecutor": {
-"securityContext": {
-"runAsUser": 1000
+"labels": {
+"runAsUser": "1000"
Review comment:
Not really in favour of removing the feature -- that seems like a massive
bug that we need to fix.
And if it doesn't work there's no point having that example task.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] davlum commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S worker pod to be configured from JSON/YAML file
davlum commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S
worker pod to be configured from JSON/YAML file
URL: https://github.com/apache/airflow/pull/6230#discussion_r375416151
##
File path: airflow/example_dags/example_kubernetes_executor_config.py
##
@@ -87,8 +87,8 @@ def test_volume_mount():
python_callable=print_stuff,
executor_config={
"KubernetesExecutor": {
-"securityContext": {
-"runAsUser": 1000
+"labels": {
+"runAsUser": "1000"
Review comment:
Ok, so continue supporting passing the argument even though it doesn't set
the user? I don't know if it's a bug exactly. You can pass `security_context`
and that will work to set user.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-6307) Improve support for a special characters in DAG's name in stats view
[ https://issues.apache.org/jira/browse/AIRFLOW-6307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030889#comment-17030889 ] ASF GitHub Bot commented on AIRFLOW-6307: - mik-laj commented on pull request #6855: [AIRFLOW-6307] Improve support for a special characters in DAG's name in stats view URL: https://github.com/apache/airflow/pull/6855 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve support for a special characters in DAG's name in stats view > > > Key: AIRFLOW-6307 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6307 > Project: Apache Airflow > Issue Type: New Feature > Components: webserver >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] robinedwards commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
robinedwards commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582538572 I guess another option is doing multiple GET requests... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7353: [AIRFLOW-6685] Data Quality Check operators
codecov-io edited a comment on issue #7353: [AIRFLOW-6685] Data Quality Check operators URL: https://github.com/apache/airflow/pull/7353#issuecomment-582190961 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7353?src=pr=h1) Report > Merging [#7353](https://codecov.io/gh/apache/airflow/pull/7353?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/2195bb45be00f0bdaadf4b7009db40f360f5455b?src=pr=desc) will **decrease** coverage by `0.65%`. > The diff coverage is `84.61%`. [](https://codecov.io/gh/apache/airflow/pull/7353?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7353 +/- ## == - Coverage 86.35% 85.69% -0.66% == Files 871 872 +1 Lines 4062740692 +65 == - Hits3508334873 -210 - Misses 5544 5819 +275 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7353?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/data\_quality\_operator.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZGF0YV9xdWFsaXR5X29wZXJhdG9yLnB5) | `84.61% <84.61%> (ø)` | | | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0%> (-64.16%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `39.28% <0%> (-60.72%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `50% <0%> (-50%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0%> (-45.66%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0%> (-45.08%)` | :arrow_down: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55% <0%> (-45%)` | :arrow_down: | | [...roviders/google/cloud/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9wb3N0Z3Jlc190b19nY3MucHk=) | `52.94% <0%> (-32.36%)` | :arrow_down: | | ... and [10 more](https://codecov.io/gh/apache/airflow/pull/7353/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7353?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7353?src=pr=footer). Last update [2195bb4...f6c003b](https://codecov.io/gh/apache/airflow/pull/7353?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
ashb commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582664082 There's also a `postAsForm` helper method already in Airflow https://github.com/apache/airflow/blob/9e150cf17c4f279df7d9290aec3c7c726d61c32e/airflow/www/static/js/base.js#L75-L103 if that's of use. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
ashb commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582664239 And yes, using `request.form.getlist()` is the way to go over json This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7372: [AIRFLOW-6738] Upload container diagnostics always
codecov-io edited a comment on issue #7372: [AIRFLOW-6738] Upload container diagnostics always URL: https://github.com/apache/airflow/pull/7372#issuecomment-582524422 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=h1) Report > Merging [#7372](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/2195bb45be00f0bdaadf4b7009db40f360f5455b?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7372 +/- ## === Coverage 86.35% 86.35% === Files 871 871 Lines 4062740627 === Hits3508335083 Misses 5544 5544 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=footer). Last update [2195bb4...75fcc87](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6744) Make the Conn Type field in Connection's Tab UI dynamic
Felipe Lolas created AIRFLOW-6744: - Summary: Make the Conn Type field in Connection's Tab UI dynamic Key: AIRFLOW-6744 URL: https://issues.apache.org/jira/browse/AIRFLOW-6744 Project: Apache Airflow Issue Type: Improvement Components: hooks Affects Versions: 1.10.7 Reporter: Felipe Lolas Hi! Currently in our team we had been developing a lots of customs hooks and operators, sadly we can't setup easily using Connection's Tab UI because the Conn Type field is hardcoded in the frontend(please correct me if i'm wrong). Maybe the Conn Type field should be dynamic generated by the hook implementation itself (idk if it's possibly this solution) or make a custom list in the Admin's Tab with add/update/delete actions. Cheers Felipe -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031106#comment-17031106
]
Jarek Potiuk commented on AIRFLOW-2516:
---
Sorry I've been quite busy and did not look at the last one yet. Which version
are you on [~cjsh5794]? The upcoming 1.10.8 should have the most frequent
problem solved. It should happen far more frequently if you upgrade (after it
gets released)
> Deadlock found when trying to update task_instance table
>
>
> Key: AIRFLOW-2516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2516
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
>Affects Versions: 1.8.0, 1.9.0, 1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4,
> 1.10.5, 1.10.6, 1.10.7
>Reporter: Jeff Liu
>Assignee: Jarek Potiuk
>Priority: Major
> Fix For: 1.10.8
>
> Attachments: Screenshot 2019-12-30 at 10.42.52.png,
> image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png,
> jobs.py, jobs_fixed_deadlock_possibly_1.9.py,
> scheduler_job_fixed_deadlock_possibly_1.10.6.py
>
>
>
>
> {code:java}
> [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask:
> [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue:
> airflow run production_wipeout_wipe_manager.Carat Carat_20180227
> 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py
> [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback
> (most recent call last):
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/bin/airflow", line 27, in
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> args.func(args)
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> pool=args.pool,
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in
> wrapper
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result =
> func(*args, **kwargs)
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in
> _run_raw_task
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> self.handle_failure(e, test_mode, context)
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in
> handle_failure
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> session.merge(self)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1920, in merge
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask:
> _resolve_conflict_map=_resolve_conflict_map)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1974, in _merge
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged =
> self.query(mapper.class_).get(key[1])
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882,
> in get
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident,
> loading.load_on_pk_identity)
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952,
> in _get_impl
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> db_load_fn(self, primary_key_identity)
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247,
> in load_on_pk_i
> dentity
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> q.one()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884,
> in one
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> self.one_or_none()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854,
> in one_or_none
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> list(self)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File
>
[GitHub] [airflow] madison-ookla commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
madison-ookla commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582706477 I've converted the endpoints from JSON to form POST. It was easier than I thought! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031253#comment-17031253
]
Chinmay Joshi commented on AIRFLOW-2516:
Hi [~potiuk], the stacktrace is from environment using 1.10.5, but we are also
seeing issues on 1.10.7, with similar stacktrace.
I think the fixes are in scheduler_job.py, geared towards fix for deadlock in
Main DAG.
But I believe we use backfill_job.py for SubDAG, where no change was made for
the deadlock issue.
Also, here there is a session.merge() followed by a session.commit(), which
seems to be generating the FOR UPDATE implicitly, perhaps due to
*ti.refresh_from_db(lock_for_update=True, session=session)* few lines back.
Maybe I should change _lock_for_update=False_.
I was thinking if I should backport
[this|https://github.com/apache/airflow/commit/0be39219cd058ba7d50cdf34b2cc46513f4f5ab3#diff-e69c537c7e60bc4ba289fce64e53981a]
change which makes use of scheduler.py instead of backfill_job.py for SubDAGs,
but since it is not released, I ain't sure.
Any guidance would be greatly appreciated. Thank you.
> Deadlock found when trying to update task_instance table
>
>
> Key: AIRFLOW-2516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2516
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
>Affects Versions: 1.8.0, 1.9.0, 1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4,
> 1.10.5, 1.10.6, 1.10.7
>Reporter: Jeff Liu
>Assignee: Jarek Potiuk
>Priority: Major
> Fix For: 1.10.8
>
> Attachments: Screenshot 2019-12-30 at 10.42.52.png,
> image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png,
> jobs.py, jobs_fixed_deadlock_possibly_1.9.py,
> scheduler_job_fixed_deadlock_possibly_1.10.6.py
>
>
>
>
> {code:java}
> [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask:
> [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue:
> airflow run production_wipeout_wipe_manager.Carat Carat_20180227
> 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py
> [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback
> (most recent call last):
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/bin/airflow", line 27, in
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> args.func(args)
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> pool=args.pool,
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in
> wrapper
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result =
> func(*args, **kwargs)
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in
> _run_raw_task
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> self.handle_failure(e, test_mode, context)
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in
> handle_failure
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> session.merge(self)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1920, in merge
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask:
> _resolve_conflict_map=_resolve_conflict_map)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1974, in _merge
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged =
> self.query(mapper.class_).get(key[1])
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882,
> in get
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident,
> loading.load_on_pk_identity)
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952,
> in _get_impl
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> db_load_fn(self, primary_key_identity)
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247,
> in load_on_pk_i
> dentity
> [2018-05-23 17:59:57,237]
[jira] [Created] (AIRFLOW-6742) Task instance state set to failed even though the Pod succeeded when using KubernetesExecutor
Andrew Cleland created AIRFLOW-6742: --- Summary: Task instance state set to failed even though the Pod succeeded when using KubernetesExecutor Key: AIRFLOW-6742 URL: https://issues.apache.org/jira/browse/AIRFLOW-6742 Project: Apache Airflow Issue Type: Bug Components: executor-kubernetes Affects Versions: 1.10.7, 1.10.8 Environment: Kubernetes (1.16), Airflow (1.10.7, 10.10.8rc1) Reporter: Andrew Cleland Assignee: Daniel Imberman Attachments: airflow_scheduler_logs.txt, failed_dag_run.png, failed_task_instance.png, k8s_pods.png, kubernetes_executor_logs.txt When running a KubernetesPodOperator task with the KubernetesExecutor, the Pod succeeds but Airflow sets the task instance state to Failed. Attached files: * k8s_pods.png - KubernetesExecutor pod and KubernetesPodOperator pod both succeeded * kubernetes_executor_logs - Launched the KubernetesPodOperator successfully * airflow_scheduler_logs - "Found matching task with current state failed" * failed_task_instance - The failed task instance in the airflow UI * failed_dag_run - The failed dag run in the airflow UI It seems that the database is being updated with task state of failed, but I'm not sure whereabouts this state is being changed. [Here|https://github.com/apache/airflow/blob/1.10.7/airflow/contrib/executors/kubernetes_executor.py#L628] is the line where the KubernetesExecutor queries the database and finds a failed task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
ashb commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582667085 That helper method won't accept multiple fields of the same name just yet, so you might have to extend that or use something else This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] madison-ookla commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
madison-ookla commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582668507 > That helper method won't accept multiple fields of the same name just yet, so you might have to extend that or use something else Since the request is done with `d3.json` I'm not sure I'd be able to use that anyway This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7372: [AIRFLOW-6738] Upload container diagnostics on failure
codecov-io edited a comment on issue #7372: [AIRFLOW-6738] Upload container diagnostics on failure URL: https://github.com/apache/airflow/pull/7372#issuecomment-582524422 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=h1) Report > Merging [#7372](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/2195bb45be00f0bdaadf4b7009db40f360f5455b?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7372 +/- ## === Coverage 86.35% 86.35% === Files 871 871 Lines 4062740627 === Hits3508335083 Misses 5544 5544 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=footer). Last update [2195bb4...1f147f0](https://codecov.io/gh/apache/airflow/pull/7372?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-6738) Diagnostics from all containers should be uploaded to file.io
[ https://issues.apache.org/jira/browse/AIRFLOW-6738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-6738: -- Summary: Diagnostics from all containers should be uploaded to file.io (was: Diagnostics from all containers should be uploaded to file.io on failure) > Diagnostics from all containers should be uploaded to file.io > - > > Key: AIRFLOW-6738 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6738 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > > Uploading logs from all containers should help when there is a test failure -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6743) Implement SSL for Hive2ServerHook
Felipe Lolas created AIRFLOW-6743: - Summary: Implement SSL for Hive2ServerHook Key: AIRFLOW-6743 URL: https://issues.apache.org/jira/browse/AIRFLOW-6743 Project: Apache Airflow Issue Type: Improvement Components: hooks Affects Versions: 1.10.7 Reporter: Felipe Lolas Hi! It would be nice to have a implementation for a working Hive2ServerHook with SASL(GSSAPI) + SSL. Currently only works with SASL without SSL support. Sadly PyHive library doesn't support this out-of-the-box, but its manageable in the connect method having the following python's deps python 3.7 pyhive[hive,kerberos]>=0.6.1 thrift-sasl>=0.4 thrift==0.10.0 sasl>=0.2.1 I can make a PR in the next days! Cheers, Felipe -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk commented on a change in pull request #7371: [AIRFLOW-6737] Enable kubernetes diagnostics for all kubernetes tests
potiuk commented on a change in pull request #7371: [AIRFLOW-6737] Enable kubernetes diagnostics for all kubernetes tests URL: https://github.com/apache/airflow/pull/7371#discussion_r375545117 ## File path: tests/runtime/kubernetes/test_kubernetes_executor.py ## @@ -66,6 +64,10 @@ def setUp(self): def tearDown(self): self.session.close() +@classmethod +def tearDownClass(cls) -> None: +dump_kubernetes_logs() Review comment: I chaged it completely now. I moved all the extract/upload to the entrypoint_ci.sh bash script - it makes much more sense there (we have guarante that the output containing files.io link) will be printed in the log from Travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] madison-ookla commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
madison-ookla commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582665122 Gotcha, thanks @ashb. I'll move this over to a form POST then. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
codecov-io edited a comment on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582268234 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7364?src=pr=h1) Report > Merging [#7364](https://codecov.io/gh/apache/airflow/pull/7364?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/2bfd7f28441ba331841e79196ef80df8c788?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7364?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7364 +/- ## == + Coverage 86.35% 86.35% +<.01% == Files 871 871 Lines 4062740627 == + Hits3508235084 +2 + Misses 5545 5543 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7364?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7364/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `88.12% <0%> (+0.19%)` | :arrow_up: | | [airflow/providers/ssh/operators/ssh.py](https://codecov.io/gh/apache/airflow/pull/7364/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvc3NoL29wZXJhdG9ycy9zc2gucHk=) | `84.61% <0%> (+1.28%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7364?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7364?src=pr=footer). Last update [2bfd7f2...8b72c8c](https://codecov.io/gh/apache/airflow/pull/7364?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] KKcorps commented on issue #6875: [AIRFLOW-6076] Fix dag.cli() Key error
KKcorps commented on issue #6875: [AIRFLOW-6076] Fix dag.cli() Key error URL: https://github.com/apache/airflow/pull/6875#issuecomment-582744947 @mik-laj These PR changes are already there in the master. I think I should close it then? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4509) SubDagOperator using scheduler instead of backfill
[ https://issues.apache.org/jira/browse/AIRFLOW-4509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031310#comment-17031310 ] Chinmay Joshi commented on AIRFLOW-4509: Hi [~milton0825], I see that today the code in master is very different from the code when you had made this commit. We are currently on latest released version (1.10.7). Will this commit be portable to our current code base? We are actually facing deadlocks in SubDAGs ([AIRFLOW-2516|https://issues.apache.org/jira/browse/AIRFLOW-2516]), and thinking if we could change the way it is scheduled. Could you please let us know? Thank you. > SubDagOperator using scheduler instead of backfill > -- > > Key: AIRFLOW-4509 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4509 > Project: Apache Airflow > Issue Type: New Feature > Components: scheduler >Affects Versions: 1.10.3 >Reporter: Chao-Han Tsai >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 2.0.0 > > > Make SubDagOperator use Airflow scheduler instead of backfill to schedule > tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb commented on a change in pull request #7312: [AIRFLOW-6695] Add DAG run configuration to UI
ashb commented on a change in pull request #7312: [AIRFLOW-6695] Add DAG run
configuration to UI
URL: https://github.com/apache/airflow/pull/7312#discussion_r375380927
##
File path: airflow/www/templates/airflow/trigger.html
##
@@ -0,0 +1,36 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends "airflow/master.html" %}
Review comment:
```suggestion
{% extends base_template %}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7312: [AIRFLOW-6695] Add DAG run configuration to UI
ashb commented on a change in pull request #7312: [AIRFLOW-6695] Add DAG run
configuration to UI
URL: https://github.com/apache/airflow/pull/7312#discussion_r375381604
##
File path: airflow/www/views.py
##
@@ -972,42 +972,58 @@ def delete(self):
# Upon success return to origin.
return redirect(origin)
-@expose('/trigger', methods=['POST'])
+@expose('/trigger', methods=['POST', 'GET'])
@has_dag_access(can_dag_edit=True)
@has_access
@action_logging
@provide_session
def trigger(self, session=None):
+
dag_id = request.values.get('dag_id')
origin = request.values.get('origin') or url_for('Airflow.index')
-dag = session.query(models.DagModel).filter(models.DagModel.dag_id ==
dag_id).first()
-if not dag:
-flash("Cannot find dag {}".format(dag_id))
-return redirect(origin)
-execution_date = timezone.utcnow()
-run_id = "manual__{0}".format(execution_date.isoformat())
+if request.method == 'GET':
+return self.render_template(
+'airflow/trigger.html',
+dag_id=dag_id,
+origin=origin
+)
+else:
Review comment:
Since the GET path returns, you can drop the `else:` (and not have to touch
the rest of this method (making the diff easier to review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7312: [AIRFLOW-6695] Add DAG run configuration to UI
ashb commented on a change in pull request #7312: [AIRFLOW-6695] Add DAG run
configuration to UI
URL: https://github.com/apache/airflow/pull/7312#discussion_r375382143
##
File path: airflow/www/templates/airflow/trigger.html
##
@@ -0,0 +1,36 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends "airflow/master.html" %}
+
+{% block content %}
+{{ super() }}
+Trigger DAG
Review comment:
Which dag? We should show that on the page.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7312: [AIRFLOW-6695] Add DAG run configuration to UI
ashb commented on a change in pull request #7312: [AIRFLOW-6695] Add DAG run
configuration to UI
URL: https://github.com/apache/airflow/pull/7312#discussion_r375382623
##
File path: airflow/www/templates/airflow/trigger.html
##
@@ -0,0 +1,36 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+#}
+
+{% extends "airflow/master.html" %}
+
+{% block content %}
+{{ super() }}
+Trigger DAG
+
+
+
+
+
+Configuration JSON (Optional)
+
+
+
+ bail.
Review comment:
If we are passing `origin` to the template we should use that rather than
`history.back` --(that or drop the origin param alltogther.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] robinedwards commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
robinedwards commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582511175 Have you actually encountered issues with the request body size? POST feels a bit weird here not very RESTy This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S worker pod to be configured from JSON/YAML file
ashb commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S
worker pod to be configured from JSON/YAML file
URL: https://github.com/apache/airflow/pull/6230#discussion_r375392235
##
File path: airflow/example_dags/example_kubernetes_executor_config.py
##
@@ -87,8 +87,8 @@ def test_volume_mount():
python_callable=print_stuff,
executor_config={
"KubernetesExecutor": {
-"securityContext": {
-"runAsUser": 1000
+"labels": {
+"runAsUser": "1000"
Review comment:
Are you saying per your other comment that this does not set runAsUser to
1000?
(For this please pick a different label that doesn't look like it might do
something :slightly_smiling_face: )
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7133: [AIRFLOW-6535] add exception to fail without retry
ashb commented on a change in pull request #7133: [AIRFLOW-6535] add exception to fail without retry URL: https://github.com/apache/airflow/pull/7133#discussion_r375393610 ## File path: airflow/models/taskinstance.py ## @@ -1124,64 +1129,56 @@ def handle_failure(self, error, test_mode=None, context=None, session=None): if context is not None: context['exception'] = error -# Let's go deeper -try: Review comment: Thanks (I was reviewing from my phone, so wasn't easy for me to check) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7133: [AIRFLOW-6535] add exception to fail without retry
ashb commented on a change in pull request #7133: [AIRFLOW-6535] add exception
to fail without retry
URL: https://github.com/apache/airflow/pull/7133#discussion_r375394352
##
File path: docs/conf.py
##
@@ -207,9 +207,12 @@
ROOT_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir))
# Generate top-level
+
+allowed_top_level = ("exceptions.py",)
Review comment:
@mik-laj @potiuk Any thoughts on this specific change?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #7133: [AIRFLOW-6535] add exception to fail without retry
ashb commented on a change in pull request #7133: [AIRFLOW-6535] add exception
to fail without retry
URL: https://github.com/apache/airflow/pull/7133#discussion_r375395180
##
File path: docs/conf.py
##
@@ -207,9 +207,12 @@
ROOT_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir))
# Generate top-level
+
+allowed_top_level = ("exceptions.py",)
Review comment:
Looking at it a bit more I think you are right though @jstern
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Assigned] (AIRFLOW-6732) Add Google Ads operators
[ https://issues.apache.org/jira/browse/AIRFLOW-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd de Quincey reassigned AIRFLOW-6732: Assignee: Todd de Quincey > Add Google Ads operators > > > Key: AIRFLOW-6732 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Assignee: Todd de Quincey >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb commented on a change in pull request #7133: [AIRFLOW-6535] add exception to fail without retry
ashb commented on a change in pull request #7133: [AIRFLOW-6535] add exception to fail without retry URL: https://github.com/apache/airflow/pull/7133#discussion_r375395844 ## File path: tests/models/test_taskinstance.py ## @@ -1475,6 +1476,8 @@ def test_handle_failure(self): dag=dag) ti1 = TI(task=task1, execution_date=start_date) ti1.state = State.FAILED +ti1.start_date = start_date +ti1.end_date = end_date Review comment: With that PR is this change now not needed? (Does it make sense to keep it anyway?) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7371: [AIRFLOW-6737] Enable kubernetes diagnostics for all kubernetes tests
codecov-io commented on issue #7371: [AIRFLOW-6737] Enable kubernetes diagnostics for all kubernetes tests URL: https://github.com/apache/airflow/pull/7371#issuecomment-582517139 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7371?src=pr=h1) Report > Merging [#7371](https://codecov.io/gh/apache/airflow/pull/7371?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/2195bb45be00f0bdaadf4b7009db40f360f5455b?src=pr=desc) will **decrease** coverage by `54.26%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7371?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7371 +/- ## === - Coverage 86.35% 32.08% -54.27% === Files 871 870-1 Lines 4062740614 -13 === - Hits3508313031-22052 - Misses 554427583+22039 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7371?src=pr=tree) | Coverage Δ | | |---|---|---| | [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/7371/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [...ing\_platform/example\_dags/example\_display\_video.py](https://codecov.io/gh/apache/airflow/pull/7371/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kaXNwbGF5X3ZpZGVvLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/hooks/vertica\_hook.py](https://codecov.io/gh/apache/airflow/pull/7371/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3ZlcnRpY2FfaG9vay5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/sensors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7371/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvX19pbml0X18ucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/7371/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/hooks/webhdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7371/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy93ZWJoZGZzX2hvb2sucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/sensors/emr\_base\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7371/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZW1yX2Jhc2Vfc2Vuc29yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [...irflow/contrib/operators/slack\_webhook\_operator.py](https://codecov.io/gh/apache/airflow/pull/7371/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9zbGFja193ZWJob29rX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/hooks/jdbc\_hook.py](https://codecov.io/gh/apache/airflow/pull/7371/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9qZGJjX2hvb2sucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/python\_operator.py](https://codecov.io/gh/apache/airflow/pull/7371/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcHl0aG9uX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | ... and [740 more](https://codecov.io/gh/apache/airflow/pull/7371/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7371?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7371?src=pr=footer). Last update [2195bb4...06b22d7](https://codecov.io/gh/apache/airflow/pull/7371?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #7359: [AIRFLOW-XXXX] Remove Undocumented, Deprecated, Dysfunctional PROXY_FIX_NUM_PROXIES
ashb commented on issue #7359: [AIRFLOW-] Remove Undocumented, Deprecated, Dysfunctional PROXY_FIX_NUM_PROXIES URL: https://github.com/apache/airflow/pull/7359#issuecomment-582519895 Change looks good, but this needs a jira ticket please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] madison-ookla commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST
madison-ookla commented on issue #7364: [AIRFLOW-6728] Change various DAG info methods to POST URL: https://github.com/apache/airflow/pull/7364#issuecomment-582519423 > Have you actually encountered issues with the request body size? POST feels a bit weird here not very RESTy I brought this issue forward because the `Recent Tasks` column in our production instance wasn't displaying any information. After investigation I found that it was because of the request URL size. Since the limit is 4096, to query 100 DAGs those DAGs only need to be 41 characters long to reach that limit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read time out error
ashb commented on issue #6643: [AIRFLOW-6040] Fix KubernetesJobWatcher Read time out error URL: https://github.com/apache/airflow/pull/6643#issuecomment-582520438 The kube tests in particular are the hardest to test, yes :( This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-5231) S3Hook delete fails with over 1000 keys
[
https://issues.apache.org/jira/browse/AIRFLOW-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cyril Shcherbin reassigned AIRFLOW-5231:
Assignee: Cyril Shcherbin
> S3Hook delete fails with over 1000 keys
> ---
>
> Key: AIRFLOW-5231
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5231
> Project: Apache Airflow
> Issue Type: Bug
> Components: aws, hooks
>Affects Versions: 2.0.0
>Reporter: Silviu Tantos
>Assignee: Cyril Shcherbin
>Priority: Major
>
> Error raised:
> {noformat}
> botocore.exceptions.ClientError: An error occurred (MalformedXML) when
> calling the DeleteObjects operation: The XML you provided was not well-formed
> or did not validate against our published schema{noformat}
> See also: https://github.com/spotify/luigi/issues/2511
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6664) Introduce implicit packages
[ https://issues.apache.org/jira/browse/AIRFLOW-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-6664: -- Summary: Introduce implicit packages (was: Remove empty __init__ packages ) > Introduce implicit packages > --- > > Key: AIRFLOW-6664 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6664 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci, packages >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6731) kubernetes pod_generator: SIDECAR_CONTAINER image name should be configurable
[ https://issues.apache.org/jira/browse/AIRFLOW-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030546#comment-17030546 ] Kamil Bregula commented on AIRFLOW-6731: For ad-hoc solution, you could use pod_mutation_hook [https://airflow.readthedocs.io/en/latest/kubernetes.html#pod-mutation-hook] I think, we should support configuration from yaml file: [https://github.com/apache/airflow/pull/6230/files] > kubernetes pod_generator: SIDECAR_CONTAINER image name should be configurable > - > > Key: AIRFLOW-6731 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6731 > Project: Apache Airflow > Issue Type: Improvement > Components: executor-kubernetes >Affects Versions: 1.10.5, 1.10.6, 1.10.7 >Reporter: thierry accart >Assignee: Daniel Imberman >Priority: Major > > Hi > Since v1.10.5, as side car container is added to KubernetesPodExecutor with a > hardcoded image name (initially "python:3.5-alpine", updated to "alpine" with > no version later.) > Unfortunately, when running on a private cloud infrastructure not connected > to internet, using such image name will fail. Moreover, some security > policies could forbid usage of uncontrolled images. > I think image name for sidecar containers should be part of airflow > configuration file. > ( Source code is airflow/kubernetes/pod_generator.py, see SIDECAR_CONTAINER > object creation) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6731) kubernetes pod_generator: SIDECAR_CONTAINER image name should be configurable
thierry accart created AIRFLOW-6731: --- Summary: kubernetes pod_generator: SIDECAR_CONTAINER image name should be configurable Key: AIRFLOW-6731 URL: https://issues.apache.org/jira/browse/AIRFLOW-6731 Project: Apache Airflow Issue Type: Improvement Components: executor-kubernetes Affects Versions: 1.10.7, 1.10.6, 1.10.5 Reporter: thierry accart Assignee: Daniel Imberman Hi Since v1.10.5, as side car container is added to KubernetesPodExecutor with a hardcoded image name (initially "python:3.5-alpine", updated to "alpine" with no version later.) Unfortunately, when running on a private cloud infrastructure not connected to internet, using such image name will fail. Moreover, some security policies could forbid usage of uncontrolled images. I think image name for sidecar containers should be part of airflow configuration file. ( Source code is airflow/kubernetes/pod_generator.py, see SIDECAR_CONTAINER object creation) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6732) Add Google Ads operators
Tomasz Urbaszek created AIRFLOW-6732: Summary: Add Google Ads operators Key: AIRFLOW-6732 URL: https://issues.apache.org/jira/browse/AIRFLOW-6732 Project: Apache Airflow Issue Type: New Feature Components: gcp, operators Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6733) Extend, don't replace FAB layout template
Ash Berlin-Taylor created AIRFLOW-6733: -- Summary: Extend, don't replace FAB layout template Key: AIRFLOW-6733 URL: https://issues.apache.org/jira/browse/AIRFLOW-6733 Project: Apache Airflow Issue Type: Improvement Components: webserver Affects Versions: 1.10.9 Reporter: Ash Berlin-Taylor This is a minor tidy-up of the templates. Right now we don't set the base_template of the appbuilder, but instead we replace the whole file earlier in the search path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?
[
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030635#comment-17030635
]
Assaf Liebstein commented on AIRFLOW-5071:
--
We experience the same behavior.
Basically we get multiple retries after the task is successfully completes in
the executor.
We use 10.0.6 and this happens in a dag py file that generates multiple dags
and multiple steps according to variables.
> Thousand os Executor reports task instance X finished (success) although the
> task says its queued. Was the task killed externally?
> --
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, scheduler
>Affects Versions: 1.10.3
>Reporter: msempere
>Priority: Critical
> Attachments: image-2020-01-27-18-10-29-124.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands
> of daily messages like the following in the logs:
>
> ```
> {{__init__.py:1580}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mik-laj merged pull request #7360: [AIRFLOW-XXXX] Add notes about airflow.providers and docs
mik-laj merged pull request #7360: [AIRFLOW-] Add notes about airflow.providers and docs URL: https://github.com/apache/airflow/pull/7360 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj merged pull request #7361: [AIRFLOW-XXXX] Remove "Core and community package" section
mik-laj merged pull request #7361: [AIRFLOW-] Remove "Core and community package" section URL: https://github.com/apache/airflow/pull/7361 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6734) Some templates ignore base_template setting
Ash Berlin-Taylor created AIRFLOW-6734: -- Summary: Some templates ignore base_template setting Key: AIRFLOW-6734 URL: https://issues.apache.org/jira/browse/AIRFLOW-6734 Project: Apache Airflow Issue Type: Bug Components: ui Affects Versions: 1.10.8 Reporter: Ash Berlin-Taylor Assignee: Ash Berlin-Taylor Some of the templates are hard-coded to extend airflow/master.html, thus ignoring the base_template setting. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-6734) Some templates ignore base_template setting
[ https://issues.apache.org/jira/browse/AIRFLOW-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-6734 started by Ash Berlin-Taylor. -- > Some templates ignore base_template setting > --- > > Key: AIRFLOW-6734 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6734 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.8 >Reporter: Ash Berlin-Taylor >Assignee: Ash Berlin-Taylor >Priority: Minor > > Some of the templates are hard-coded to extend airflow/master.html, thus > ignoring the base_template setting. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6735) Include path-based-args in action audit logs
Ash Berlin-Taylor created AIRFLOW-6735:
--
Summary: Include path-based-args in action audit logs
Key: AIRFLOW-6735
URL: https://issues.apache.org/jira/browse/AIRFLOW-6735
Project: Apache Airflow
Issue Type: Improvement
Components: webserver
Affects Versions: 1.10.8
Reporter: Ash Berlin-Taylor
Assignee: Ash Berlin-Taylor
It is possible to define arguments to Flask views as part of the UR:
{code:python}
@expose("/check", methods=["POST"])
@action_logging
def check(self, version):
...
{code}
Flask automatically pulls the component out of the URL for this. Right now the
action_logging decorator only shows query/post parameters, but it would be nice
to include these path arguments too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6733) Extend, don't replace FAB layout template
[ https://issues.apache.org/jira/browse/AIRFLOW-6733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030619#comment-17030619 ] ASF GitHub Bot commented on AIRFLOW-6733: - ashb commented on pull request #7366: [AIRFLOW-6733] Extend not replace template URL: https://github.com/apache/airflow/pull/7366 We had copied whole-sale the baselayout.html from FAB and customized it, but pre-existing mechanism in flask/jinja of template inheritance means we don't have to do this. We can instead extend the template from FAB and change the few blocks we need to. I have written this PR as two commits -- one to move the file without any changes (we can't extend `appbuilder/baselayout.html` from FAB while our file exists, as that is "earlier" in the template search path) and then a second commit that keeps just the blocks we have extended/changed. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Extend, don't replace FAB layout template > - > > Key: AIRFLOW-6733 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6733 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Affects Versions: 1.10.9 >Reporter: Ash Berlin-Taylor >Priority: Minor > > This is a minor tidy-up of the templates. Right now we don't set the > base_template of the appbuilder, but instead we replace the whole file > earlier in the search path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb opened a new pull request #7366: [AIRFLOW-6733] Extend not replace template
ashb opened a new pull request #7366: [AIRFLOW-6733] Extend not replace template URL: https://github.com/apache/airflow/pull/7366 We had copied whole-sale the baselayout.html from FAB and customized it, but pre-existing mechanism in flask/jinja of template inheritance means we don't have to do this. We can instead extend the template from FAB and change the few blocks we need to. I have written this PR as two commits -- one to move the file without any changes (we can't extend `appbuilder/baselayout.html` from FAB while our file exists, as that is "earlier" in the template search path) and then a second commit that keeps just the blocks we have extended/changed. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
