[GitHub] [airflow] HaloKo4 commented on issue #5785: [AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator
HaloKo4 commented on issue #5785: [AIRFLOW-5176] Add Azure Data Explorer (Kusto) operator URL: https://github.com/apache/airflow/pull/5785#issuecomment-575026647 I'm looking forward to this PR! Airflow has almost none support for Azure. this addition is very important! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7141: [AIRFLOW-6544] add log_id to end-of-file mark and also add an index config for logs
codecov-io edited a comment on issue #7141: [AIRFLOW-6544] add log_id to end-of-file mark and also add an index config for logs URL: https://github.com/apache/airflow/pull/7141#issuecomment-573571156 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=h1) Report > Merging [#7141](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d654d69d7794a57c5c51685a8a96f1d7c38c2302?src=pr=desc) will **decrease** coverage by `0.14%`. > The diff coverage is `66.66%`. [](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7141 +/- ## == - Coverage 85.24% 85.09% -0.15% == Files 683 723 +40 Lines 3915539550 +395 == + Hits3337833656 +278 - Misses 5777 5894 +117 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/config\_templates/airflow\_local\_settings.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWdfdGVtcGxhdGVzL2FpcmZsb3dfbG9jYWxfc2V0dGluZ3MucHk=) | `70.21% <0%> (-1.53%)` | :arrow_down: | | [airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5) | `93.57% <80%> (+0.18%)` | :arrow_up: | | [airflow/contrib/hooks/azure\_data\_lake\_hook.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F6dXJlX2RhdGFfbGFrZV9ob29rLnB5) | `0% <0%> (-93.11%)` | :arrow_down: | | [airflow/contrib/sensors/azure\_cosmos\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvYXp1cmVfY29zbW9zX3NlbnNvci5weQ==) | `0% <0%> (-81.25%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/security/kerberos.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9zZWN1cml0eS9rZXJiZXJvcy5weQ==) | `30.43% <0%> (-45.66%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.49%)` | :arrow_down: | | ... and [163 more](https://codecov.io/gh/apache/airflow/pull/7141/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=footer). Last update [d654d69...fb27397](https://codecov.io/gh/apache/airflow/pull/7141?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6469: [AIRFLOW-5816] S3 to snowflake operator
codecov-io edited a comment on issue #6469: [AIRFLOW-5816] S3 to snowflake operator URL: https://github.com/apache/airflow/pull/6469#issuecomment-547868728 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6469?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@bbe2f61`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `96.11%`. [](https://codecov.io/gh/apache/airflow/pull/6469?src=pr=tree) ```diff @@Coverage Diff@@ ## master#6469 +/- ## = Coverage ? 85.13% = Files ? 726 Lines ?39581 Branches ?0 = Hits ?33697 Misses? 5884 Partials ?0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6469?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/providers/snowflake/operators/snowflake.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvc25vd2ZsYWtlL29wZXJhdG9ycy9zbm93Zmxha2UucHk=) | `0% <ø> (ø)` | | | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.81% <ø> (ø)` | | | [airflow/providers/snowflake/hooks/snowflake.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvc25vd2ZsYWtlL2hvb2tzL3Nub3dmbGFrZS5weQ==) | `0% <ø> (ø)` | | | [airflow/utils/log/gcs\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZ2NzX3Rhc2tfaGFuZGxlci5weQ==) | `0% <0%> (ø)` | | | [...ders/google/cloud/example\_dags/example\_dataproc.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2RhdGFwcm9jLnB5) | `0% <0%> (ø)` | | | [airflow/operators/cassandra\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvY2Fzc2FuZHJhX3RvX2djcy5weQ==) | `64.91% <100%> (ø)` | | | [airflow/contrib/hooks/gcs\_hook.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djc19ob29rLnB5) | `100% <100%> (ø)` | | | [airflow/gcp/hooks/gcs.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZ2NzLnB5) | `84.98% <100%> (ø)` | | | [airflow/operators/gcs\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2NzX3RvX2djcy5weQ==) | `95.65% <100%> (ø)` | | | ... and [39 more](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6469?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6469?src=pr=footer). Last update [bbe2f61...26ac0b0](https://codecov.io/gh/apache/airflow/pull/6469?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6469: [AIRFLOW-5816] S3 to snowflake operator
codecov-io edited a comment on issue #6469: [AIRFLOW-5816] S3 to snowflake operator URL: https://github.com/apache/airflow/pull/6469#issuecomment-547868728 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6469?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@bbe2f61`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `96.11%`. [](https://codecov.io/gh/apache/airflow/pull/6469?src=pr=tree) ```diff @@Coverage Diff@@ ## master#6469 +/- ## = Coverage ? 85.13% = Files ? 726 Lines ?39581 Branches ?0 = Hits ?33697 Misses? 5884 Partials ?0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6469?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/providers/snowflake/operators/snowflake.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvc25vd2ZsYWtlL29wZXJhdG9ycy9zbm93Zmxha2UucHk=) | `0% <ø> (ø)` | | | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.81% <ø> (ø)` | | | [airflow/providers/snowflake/hooks/snowflake.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvc25vd2ZsYWtlL2hvb2tzL3Nub3dmbGFrZS5weQ==) | `0% <ø> (ø)` | | | [airflow/utils/log/gcs\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZ2NzX3Rhc2tfaGFuZGxlci5weQ==) | `0% <0%> (ø)` | | | [...ders/google/cloud/example\_dags/example\_dataproc.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2RhdGFwcm9jLnB5) | `0% <0%> (ø)` | | | [airflow/operators/cassandra\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvY2Fzc2FuZHJhX3RvX2djcy5weQ==) | `64.91% <100%> (ø)` | | | [airflow/contrib/hooks/gcs\_hook.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djc19ob29rLnB5) | `100% <100%> (ø)` | | | [airflow/gcp/hooks/gcs.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZ2NzLnB5) | `84.98% <100%> (ø)` | | | [airflow/operators/gcs\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2NzX3RvX2djcy5weQ==) | `95.65% <100%> (ø)` | | | ... and [39 more](https://codecov.io/gh/apache/airflow/pull/6469/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6469?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6469?src=pr=footer). Last update [bbe2f61...26ac0b0](https://codecov.io/gh/apache/airflow/pull/6469?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (AIRFLOW-6518) Task did not retry when there was temporary metastore db connectivity loss
[
https://issues.apache.org/jira/browse/AIRFLOW-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016625#comment-17016625
]
t oo edited comment on AIRFLOW-6518 at 1/16/20 7:27 AM:
some options:
1. introduce db retry config - https://stackoverflow.com/a/53300049/8874837
2. set conservative defaults, wrap calls to db functions in try/catch with
sleep and another call to the db function
ie
https://github.com/apache/airflow/blob/1.10.6/airflow/models/taskinstance.py#L961-L969
{code}
except AirflowException as e:
self.refresh_from_db()
# for case when task is marked as success/failed externally
# current behavior doesn't hit the success callback
if self.state in {State.SUCCESS, State.FAILED}:
return
else:
self.handle_failure(e, test_mode, context)
raise
{code}
CHANGES TO
{code}
except AirflowException as e:
try:
self.refresh_from_db()
# for case when task is marked as success/failed externally
# current behavior doesn't hit the success callback
if self.state in {State.SUCCESS, State.FAILED}:
return
else:
self.handle_failure(e, test_mode, context)
raise
except (Exception, KeyboardInterrupt) as e:
sleep(5)
self.refresh_from_db()
# for case when task is marked as success/failed externally
# current behavior doesn't hit the success callback
if self.state in {State.SUCCESS, State.FAILED}:
return
else:
self.handle_failure(e, test_mode, context)
raise
{code}
3. make the db function have try/catch with sleep and repeat
ie in refresh_from_db() of models/taskinstance.py
{code}
TI = TaskInstance
qry = session.query(TI).filter(
TI.dag_id == self.dag_id,
TI.task_id == self.task_id,
TI.execution_date == self.execution_date)
if lock_for_update:
ti = qry.with_for_update().first()
else:
ti = qry.first()
{code}
CHANGES TO
{code}
try:
TI = TaskInstance
qry = session.query(TI).filter(
TI.dag_id == self.dag_id,
TI.task_id == self.task_id,
TI.execution_date == self.execution_date)
if lock_for_update:
ti = qry.with_for_update().first()
else:
ti = qry.first()
except:
sleep(5)
TI = TaskInstance
qry = session.query(TI).filter(
TI.dag_id == self.dag_id,
TI.task_id == self.task_id,
TI.execution_date == self.execution_date)
if lock_for_update:
ti = qry.with_for_update().first()
else:
ti = qry.first()
{code}
was (Author: toopt4):
some options:
1. introduce db retry config - https://stackoverflow.com/a/53300049/8874837
2. set conservative defaults, wrap calls to db functions in try/catch with
sleep and another call to the db function
ie
https://github.com/apache/airflow/blob/1.10.6/airflow/models/taskinstance.py#L961-L969
{code}
except AirflowException as e:
self.refresh_from_db()
# for case when task is marked as success/failed externally
# current behavior doesn't hit the success callback
if self.state in {State.SUCCESS, State.FAILED}:
return
else:
self.handle_failure(e, test_mode, context)
raise
{code}
CHANGES TO
{code}
except AirflowException as e:
try:
self.refresh_from_db()
# for case when task is marked as success/failed externally
# current behavior doesn't hit the success callback
if self.state in {State.SUCCESS, State.FAILED}:
return
else:
self.handle_failure(e, test_mode, context)
raise
except (Exception, KeyboardInterrupt) as e:
sleep(5)
self.refresh_from_db()
# for case when task is marked as success/failed externally
# current behavior doesn't hit the success callback
if self.state in {State.SUCCESS, State.FAILED}:
return
else:
self.handle_failure(e, test_mode, context)
raise
{code}
3. make the db function have try/catch with sleep and repeat
> Task did not retry when there was temporary metastore db connectivity loss
> --
>
> Key: AIRFLOW-6518
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6518
> Project: Apache Airflow
>
[jira] [Comment Edited] (AIRFLOW-6518) Task did not retry when there was temporary metastore db connectivity loss
[
https://issues.apache.org/jira/browse/AIRFLOW-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016625#comment-17016625
]
t oo edited comment on AIRFLOW-6518 at 1/16/20 7:24 AM:
some options:
1. introduce db retry config - https://stackoverflow.com/a/53300049/8874837
2. set conservative defaults, wrap calls to db functions in try/catch with
sleep and another call to the db function
ie
https://github.com/apache/airflow/blob/1.10.6/airflow/models/taskinstance.py#L961-L969
{code}
except AirflowException as e:
self.refresh_from_db()
# for case when task is marked as success/failed externally
# current behavior doesn't hit the success callback
if self.state in {State.SUCCESS, State.FAILED}:
return
else:
self.handle_failure(e, test_mode, context)
raise
{code}
CHANGES TO
{code}
except AirflowException as e:
try:
self.refresh_from_db()
# for case when task is marked as success/failed externally
# current behavior doesn't hit the success callback
if self.state in {State.SUCCESS, State.FAILED}:
return
else:
self.handle_failure(e, test_mode, context)
raise
except (Exception, KeyboardInterrupt) as e:
sleep(5)
self.refresh_from_db()
# for case when task is marked as success/failed externally
# current behavior doesn't hit the success callback
if self.state in {State.SUCCESS, State.FAILED}:
return
else:
self.handle_failure(e, test_mode, context)
raise
{code}
3. make the db function have try/catch with sleep and repeat
was (Author: toopt4):
some options:
1. introduce db retry config - https://stackoverflow.com/a/53300049/8874837
2. set conservative defaults, wrap calls to db functions in try/catch with
sleep and another call to the db function
ie
https://github.com/apache/airflow/blob/1.10.6/airflow/models/taskinstance.py#L961-L969
except AirflowException as e:
self.refresh_from_db()
# for case when task is marked as success/failed externally
# current behavior doesn't hit the success callback
if self.state in {State.SUCCESS, State.FAILED}:
return
else:
self.handle_failure(e, test_mode, context)
raise
CHANGES TO
except AirflowException as e:
try:
self.refresh_from_db()
# for case when task is marked as success/failed externally
# current behavior doesn't hit the success callback
if self.state in {State.SUCCESS, State.FAILED}:
return
else:
self.handle_failure(e, test_mode, context)
raise
except (Exception, KeyboardInterrupt) as e:
sleep(5)
self.refresh_from_db()
# for case when task is marked as success/failed externally
# current behavior doesn't hit the success callback
if self.state in {State.SUCCESS, State.FAILED}:
return
else:
self.handle_failure(e, test_mode, context)
raise
3. make the db function have try/catch with sleep and repeat
> Task did not retry when there was temporary metastore db connectivity loss
> --
>
> Key: AIRFLOW-6518
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6518
> Project: Apache Airflow
> Issue Type: Bug
> Components: database, scheduler
>Affects Versions: 1.10.6
>Reporter: t oo
>Priority: Major
>
> My DAG has got retries configured at the task level. I started a dagrun, then
> while a task was running the metastore db crashed, the task failed, but the
> dagrun did not attempt to retry the task (even though task retries are
> configured!), db recovered 3 seconds after the task failed, instead the
> dagrun went to FAILED state.
> *Last part of log of TaskInstance:*
> [2020-01-08 17:34:46,301] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk Traceback (most recent call last):
> [2020-01-08 17:34:46,301] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File "/home/ec2-user/venv/bin/airflow", line 37, in
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk args.func(args)
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/utils/cli.py",
> line 74, in
[jira] [Commented] (AIRFLOW-6518) Task did not retry when there was temporary metastore db connectivity loss
[
https://issues.apache.org/jira/browse/AIRFLOW-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016625#comment-17016625
]
t oo commented on AIRFLOW-6518:
---
some options:
1. introduce db retry config - https://stackoverflow.com/a/53300049/8874837
2. set conservative defaults, wrap calls to db functions in try/catch with
sleep and another call to the db function
ie
https://github.com/apache/airflow/blob/1.10.6/airflow/models/taskinstance.py#L961-L969
except AirflowException as e:
self.refresh_from_db()
# for case when task is marked as success/failed externally
# current behavior doesn't hit the success callback
if self.state in {State.SUCCESS, State.FAILED}:
return
else:
self.handle_failure(e, test_mode, context)
raise
CHANGES TO
except AirflowException as e:
try:
self.refresh_from_db()
# for case when task is marked as success/failed externally
# current behavior doesn't hit the success callback
if self.state in {State.SUCCESS, State.FAILED}:
return
else:
self.handle_failure(e, test_mode, context)
raise
except (Exception, KeyboardInterrupt) as e:
sleep(5)
self.refresh_from_db()
# for case when task is marked as success/failed externally
# current behavior doesn't hit the success callback
if self.state in {State.SUCCESS, State.FAILED}:
return
else:
self.handle_failure(e, test_mode, context)
raise
3. make the db function have try/catch with sleep and repeat
> Task did not retry when there was temporary metastore db connectivity loss
> --
>
> Key: AIRFLOW-6518
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6518
> Project: Apache Airflow
> Issue Type: Bug
> Components: database, scheduler
>Affects Versions: 1.10.6
>Reporter: t oo
>Priority: Major
>
> My DAG has got retries configured at the task level. I started a dagrun, then
> while a task was running the metastore db crashed, the task failed, but the
> dagrun did not attempt to retry the task (even though task retries are
> configured!), db recovered 3 seconds after the task failed, instead the
> dagrun went to FAILED state.
> *Last part of log of TaskInstance:*
> [2020-01-08 17:34:46,301] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk Traceback (most recent call last):
> [2020-01-08 17:34:46,301] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File "/home/ec2-user/venv/bin/airflow", line 37, in
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk args.func(args)
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/utils/cli.py",
> line 74, in wrapper
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk return f(*args, **kwargs)
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/bin/cli.py",
> line 551, in run
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk _run(args, dag, ti)
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/bin/cli.py",
> line 469, in _run
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk pool=args.pool,
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/utils/db.py",
> line 74, in wrapper
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk return func(*args, **kwargs)
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/models/taskinstance.py",
> line 962, in _run_raw_task
> [2020-01-08 17:34:46,302] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk self.refresh_from_db()
> [2020-01-08 17:34:46,303] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk File
> "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/utils/db.py",
> line 74, in wrapper
> [2020-01-08 17:34:46,303] {base_task_runner.py:115} INFO - Job 34662: Subtask
> mytsk return func(*args, **kwargs)
> [2020-01-08
[GitHub] [airflow] TV4Fun commented on issue #7098: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation
TV4Fun commented on issue #7098: [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation URL: https://github.com/apache/airflow/pull/7098#issuecomment-575017704 @mik-laj, unit test and `UPDATING` note added. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6575) Change entropy source in all ci/breeze containers to urandom (unblocking)
[ https://issues.apache.org/jira/browse/AIRFLOW-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016613#comment-17016613 ] ASF subversion and git services commented on AIRFLOW-6575: -- Commit f39f26e91433fe33308bf072035ca31a0b57cf52 in airflow's branch refs/heads/master from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f39f26e ] [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking (#7185) On Travis CI blocking entropy source slows startup time of a number of containers. This change changes the entropy source to unblocking one. > Change entropy source in all ci/breeze containers to urandom (unblocking) > - > > Key: AIRFLOW-6575 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6575 > Project: Apache Airflow > Issue Type: Improvement > Components: breeze, ci >Affects Versions: 2.0.0, 1.10.7 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6575) Change entropy source in all ci/breeze containers to urandom (unblocking)
[ https://issues.apache.org/jira/browse/AIRFLOW-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-6575. --- Fix Version/s: 2.0.0 Resolution: Fixed > Change entropy source in all ci/breeze containers to urandom (unblocking) > - > > Key: AIRFLOW-6575 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6575 > Project: Apache Airflow > Issue Type: Improvement > Components: breeze, ci >Affects Versions: 2.0.0, 1.10.7 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6575) Change entropy source in all ci/breeze containers to urandom (unblocking)
[ https://issues.apache.org/jira/browse/AIRFLOW-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016612#comment-17016612 ] ASF GitHub Bot commented on AIRFLOW-6575: - potiuk commented on pull request #7185: [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking URL: https://github.com/apache/airflow/pull/7185 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Change entropy source in all ci/breeze containers to urandom (unblocking) > - > > Key: AIRFLOW-6575 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6575 > Project: Apache Airflow > Issue Type: Improvement > Components: breeze, ci >Affects Versions: 2.0.0, 1.10.7 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #7185: [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking
potiuk merged pull request #7185: [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking URL: https://github.com/apache/airflow/pull/7185 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] tooptoop4 commented on a change in pull request #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates
tooptoop4 commented on a change in pull request #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#discussion_r367254484 ## File path: tests/ti_deps/deps/test_runnable_exec_date_dep.py ## @@ -39,11 +41,56 @@ def test_exec_date_after_end_date(self): """ If the dag's execution date is in the future this dep should fail """ -ti = self._get_task_instance( -dag_end_date=datetime(2016, 1, 3), Review comment: I am 'end_date=datetime(2016, 11, 5),' line 47 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7148: [AIRFLOW-6472] Correct short option in cli
codecov-io commented on issue #7148: [AIRFLOW-6472] Correct short option in cli URL: https://github.com/apache/airflow/pull/7148#issuecomment-574976676 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7148?src=pr=h1) Report > Merging [#7148](https://codecov.io/gh/apache/airflow/pull/7148?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/59c8a826b8d5d365db68e800cea3de59256530c9?src=pr=desc) will **increase** coverage by `0.17%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7148?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7148 +/- ## == + Coverage 84.91% 85.09% +0.17% == Files 723 723 Lines 3954639546 == + Hits3358133652 +71 + Misses 5965 5894 -71 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7148?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7148/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <100%> (ø)` | :arrow_up: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/7148/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `94.92% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7148/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7148/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7148/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7148/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7148/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.49%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7148/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <0%> (+0.58%)` | :arrow_up: | | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/7148/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `68.78% <0%> (+0.97%)` | :arrow_up: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7148/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (+1.15%)` | :arrow_up: | | ... and [10 more](https://codecov.io/gh/apache/airflow/pull/7148/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7148?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7148?src=pr=footer). Last update [59c8a82...5acdbf8](https://codecov.io/gh/apache/airflow/pull/7148?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] zhongjiajie commented on issue #7148: [AIRFLOW-6472] Correct short option in cli
zhongjiajie commented on issue #7148: [AIRFLOW-6472] Correct short option in cli URL: https://github.com/apache/airflow/pull/7148#issuecomment-574965806 I create a discuss in dev-mail-list https://lists.apache.org/thread.html/r95c53953499a236466e0b762dfdadd5ca2ba9d6e2a3516c699a14380%40%3Cdev.airflow.apache.org%3E This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jithin97 commented on issue #7159: [AIRFLOW-6553] [UI] Add upstream_failed in instance state filter
jithin97 commented on issue #7159: [AIRFLOW-6553] [UI] Add upstream_failed in instance state filter URL: https://github.com/apache/airflow/pull/7159#issuecomment-574946733 > When making UI changes like this it's helpful to include a before-and-after. Could you please? > > It's probably fine, but by you include a screen shot it makes it easier for the reviewer to check the colours used are sensible. @ashb Thanks!!! **Before:** https://user-images.githubusercontent.com/13610191/72486901-a0d35e00-384f-11ea-8d36-8522c8719d23.png;> **After**   BTW this is where I took the color from. https://github.com/apache/airflow/blob/1.10.7/airflow/utils/state.py#L77 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6577) DAG Backfill with timedelta runs twice
Nick Benthem created AIRFLOW-6577:
-
Summary: DAG Backfill with timedelta runs twice
Key: AIRFLOW-6577
URL: https://issues.apache.org/jira/browse/AIRFLOW-6577
Project: Apache Airflow
Issue Type: Bug
Components: DAG, DagRun
Affects Versions: 1.10.7
Environment: ProductName: Mac OS X
ProductVersion: 10.14.6
BuildVersion: 18G2022
Client: Docker Engine - Community
Version: 19.03.5
API version: 1.40
Go version:go1.12.12
Git commit:633a0ea
Built: Wed Nov 13 07:22:34 2019
OS/Arch: darwin/amd64
Experimental: false
Reporter: Nick Benthem
if you use {{timedelta=__anything__}}, and have {{catchup=False}}, it will
cause a DOUBLE run of your DAG! The only workaround i found was to use a cron
timer, i.e.,
schedule_interval='@daily',
Rather than
schedule_interval=timedelta(days=1),
It almost definitely exists in
def create_dag_run(self, dag, session=None):
in
{{/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py
}}
around line {{643}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #7187: [AIRFLOW-6576] fix scheduler crash caused by deleted task with sla misses
codecov-io commented on issue #7187: [AIRFLOW-6576] fix scheduler crash caused by deleted task with sla misses URL: https://github.com/apache/airflow/pull/7187#issuecomment-574935439 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7187?src=pr=h1) Report > Merging [#7187](https://codecov.io/gh/apache/airflow/pull/7187?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/59c8a826b8d5d365db68e800cea3de59256530c9?src=pr=desc) will **decrease** coverage by `52.16%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/airflow/pull/7187?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7187 +/- ## === - Coverage 84.91% 32.75% -52.17% === Files 723 722-1 Lines 3954639539-7 === - Hits3358112951-20630 - Misses 596526588+20623 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7187?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7187/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `12.44% <0%> (-76.32%)` | :arrow_down: | | [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/7187/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [...flow/contrib/example\_dags/example\_qubole\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7187/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9zZW5zb3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [...ample\_dags/example\_emr\_job\_flow\_automatic\_steps.py](https://codecov.io/gh/apache/airflow/pull/7187/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2Vtcl9qb2JfZmxvd19hdXRvbWF0aWNfc3RlcHMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [...contrib/example\_dags/example\_papermill\_operator.py](https://codecov.io/gh/apache/airflow/pull/7187/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3BhcGVybWlsbF9vcGVyYXRvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/hooks/webhdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7187/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy93ZWJoZGZzX2hvb2sucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/operators/gcs\_list\_operator.py](https://codecov.io/gh/apache/airflow/pull/7187/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfbGlzdF9vcGVyYXRvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/pig\_operator.py](https://codecov.io/gh/apache/airflow/pull/7187/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcGlnX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/operators/file\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7187/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9maWxlX3RvX2djcy5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/hooks/gcp\_spanner\_hook.py](https://codecov.io/gh/apache/airflow/pull/7187/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9zcGFubmVyX2hvb2sucHk=) | `0% <0%> (-100%)` | :arrow_down: | | ... and [583 more](https://codecov.io/gh/apache/airflow/pull/7187/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7187?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7187?src=pr=footer). Last update [59c8a82...a1edb71](https://codecov.io/gh/apache/airflow/pull/7187?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7185: [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking
codecov-io edited a comment on issue #7185: [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking URL: https://github.com/apache/airflow/pull/7185#issuecomment-574910352 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=h1) Report > Merging [#7185](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/417feda5eba6cc63c01a9a04a7b48bc24f63a3d1?src=pr=desc) will **increase** coverage by `0.12%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7185 +/- ## == + Coverage 85.25% 85.37% +0.12% == Files 723 723 Lines 3954339543 == + Hits3371133761 +50 + Misses 5832 5782 -50 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <0%> (+0.43%)` | :arrow_up: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `91.73% <0%> (+1.65%)` | :arrow_up: | | [airflow/hooks/postgres\_hook.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9wb3N0Z3Jlc19ob29rLnB5) | `94.36% <0%> (+16.9%)` | :arrow_up: | | [airflow/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfdG9fZ2NzLnB5) | `85.29% <0%> (+32.35%)` | :arrow_up: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `100% <0%> (+100%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=footer). Last update [417feda...b7568be](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-6576) Deleting a task with SLA crashes the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-6576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] QP Hou updated AIRFLOW-6576: Description: When a task with SLA is deleted from a DAG after the SLA miss is logged but before the notification was sent, scheduler will crash with an AirflowException (was: When a task with SLA is deleted from a DAG after the SLA miss is logged by before the notification was sent, scheduler will crash with an AirflowException) > Deleting a task with SLA crashes the scheduler > -- > > Key: AIRFLOW-6576 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6576 > Project: Apache Airflow > Issue Type: New Feature > Components: scheduler >Affects Versions: 2.0.0 >Reporter: QP Hou >Assignee: QP Hou >Priority: Major > > When a task with SLA is deleted from a DAG after the SLA miss is logged but > before the notification was sent, scheduler will crash with an > AirflowException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6576) Deleting a task with SLA crashes the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-6576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016422#comment-17016422 ] ASF GitHub Bot commented on AIRFLOW-6576: - houqp commented on pull request #7187: [AIRFLOW-6576] fix scheduler crash caused by deleted task with sla misses URL: https://github.com/apache/airflow/pull/7187 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Deleting a task with SLA crashes the scheduler > -- > > Key: AIRFLOW-6576 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6576 > Project: Apache Airflow > Issue Type: New Feature > Components: scheduler >Affects Versions: 2.0.0 >Reporter: QP Hou >Assignee: QP Hou >Priority: Major > > When a task with SLA is deleted from a DAG after the SLA miss is logged by > before the notification was sent, scheduler will crash with an > AirflowException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] houqp opened a new pull request #7187: [AIRFLOW-6576] fix scheduler crash caused by deleted task with sla misses
houqp opened a new pull request #7187: [AIRFLOW-6576] fix scheduler crash caused by deleted task with sla misses URL: https://github.com/apache/airflow/pull/7187 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6576) Deleting a task with SLA crashes the scheduler
QP Hou created AIRFLOW-6576: --- Summary: Deleting a task with SLA crashes the scheduler Key: AIRFLOW-6576 URL: https://issues.apache.org/jira/browse/AIRFLOW-6576 Project: Apache Airflow Issue Type: New Feature Components: scheduler Affects Versions: 2.0.0 Reporter: QP Hou Assignee: QP Hou When a task with SLA is deleted from a DAG after the SLA miss is logged by before the notification was sent, scheduler will crash with an AirflowException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #7088: [WIP][AIRFLOW-6494] SSH host_proxy has to be fresh
codecov-io edited a comment on issue #7088: [WIP][AIRFLOW-6494] SSH host_proxy has to be fresh URL: https://github.com/apache/airflow/pull/7088#issuecomment-574915370 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7088?src=pr=h1) Report > Merging [#7088](https://codecov.io/gh/apache/airflow/pull/7088?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/417feda5eba6cc63c01a9a04a7b48bc24f63a3d1?src=pr=desc) will **decrease** coverage by `0.16%`. > The diff coverage is `25%`. [](https://codecov.io/gh/apache/airflow/pull/7088?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7088 +/- ## == - Coverage 85.25% 85.09% -0.17% == Files 723 723 Lines 3954339547 +4 == - Hits3371133651 -60 - Misses 5832 5896 +64 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7088?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/hooks/ssh\_hook.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NzaF9ob29rLnB5) | `86.32% <25%> (-2.18%)` | :arrow_down: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.49%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `96.66% <0%> (ø)` | :arrow_up: | | ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7088?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7088?src=pr=footer). Last update [417feda...43154c6](https://codecov.io/gh/apache/airflow/pull/7088?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
mik-laj commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries URL: https://github.com/apache/airflow/pull/6792#discussion_r367171590 ## File path: airflow/jobs/scheduler_job.py ## @@ -1306,33 +1271,32 @@ def _enqueue_task_instances_with_queued_state(self, simple_dag_bag, :param simple_dag_bag: Should contains all of the task_instances' dags :type simple_dag_bag: airflow.utils.dag_processing.SimpleDagBag """ -TI = models.TaskInstance # actually enqueue them -for simple_task_instance in simple_task_instances: -simple_dag = simple_dag_bag.get_dag(simple_task_instance.dag_id) -command = TI.generate_command( -simple_task_instance.dag_id, -simple_task_instance.task_id, -simple_task_instance.execution_date, +for ti in task_instances: +simple_dag = simple_dag_bag.get_dag(ti.dag_id) +command = ti.generate_command( +ti.dag_id, +ti.task_id, +ti.execution_date, local=True, mark_success=False, ignore_all_deps=False, ignore_depends_on_past=False, ignore_task_deps=False, ignore_ti_state=False, -pool=simple_task_instance.pool, +pool=ti.pool, file_path=simple_dag.full_filepath, pickle_id=simple_dag.pickle_id) -priority = simple_task_instance.priority_weight -queue = simple_task_instance.queue +priority = ti.priority_weight +queue = ti.queue self.log.info( "Sending %s to executor with priority %s and queue %s", -simple_task_instance.key, priority, queue +ti.key, priority, queue ) self.executor.queue_command( Review comment: > Oh queue_task_instance already exists -- I forgot you(?) added that already I wanted to add the queue_simple_task_instance method, but my PR is WIP. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6563) User marked tasks does not have a valid end_date
[ https://issues.apache.org/jira/browse/AIRFLOW-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016412#comment-17016412 ] ASF subversion and git services commented on AIRFLOW-6563: -- Commit 59c8a826b8d5d365db68e800cea3de59256530c9 in airflow's branch refs/heads/master from Yingbo Wang [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=59c8a82 ] [AIRFLOW-6563] Add end_date for marked tasks (#7171) > User marked tasks does not have a valid end_date > > > Key: AIRFLOW-6563 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6563 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.7 >Reporter: Yingbo Wang >Assignee: Yingbo Wang >Priority: Minor > Fix For: master > > > When user mark task instance to success/failed from UI, the marked task did > not update a valid end_date. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] KevinYang21 merged pull request #7171: [AIRFLOW-6563] Add end_date for marked tasks
KevinYang21 merged pull request #7171: [AIRFLOW-6563] Add end_date for marked tasks URL: https://github.com/apache/airflow/pull/7171 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7088: [WIP][AIRFLOW-6494] SSH host_proxy has to be fresh
codecov-io commented on issue #7088: [WIP][AIRFLOW-6494] SSH host_proxy has to be fresh URL: https://github.com/apache/airflow/pull/7088#issuecomment-574915370 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7088?src=pr=h1) Report > Merging [#7088](https://codecov.io/gh/apache/airflow/pull/7088?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/417feda5eba6cc63c01a9a04a7b48bc24f63a3d1?src=pr=desc) will **decrease** coverage by `0.62%`. > The diff coverage is `25%`. [](https://codecov.io/gh/apache/airflow/pull/7088?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7088 +/- ## == - Coverage 85.25% 84.62% -0.63% == Files 723 723 Lines 3954339547 +4 == - Hits3371133468 -243 - Misses 5832 6079 +247 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7088?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/hooks/ssh\_hook.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NzaF9ob29rLnB5) | `86.32% <25%> (-2.18%)` | :arrow_down: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.49%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `93.33% <0%> (-3.34%)` | :arrow_down: | | ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7088/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7088?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7088?src=pr=footer). Last update [417feda...43154c6](https://codecov.io/gh/apache/airflow/pull/7088?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6563) User marked tasks does not have a valid end_date
[ https://issues.apache.org/jira/browse/AIRFLOW-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016410#comment-17016410 ] ASF GitHub Bot commented on AIRFLOW-6563: - KevinYang21 commented on pull request #7171: [AIRFLOW-6563] Add end_date for marked tasks URL: https://github.com/apache/airflow/pull/7171 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > User marked tasks does not have a valid end_date > > > Key: AIRFLOW-6563 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6563 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.7 >Reporter: Yingbo Wang >Assignee: Yingbo Wang >Priority: Minor > Fix For: master > > > When user mark task instance to success/failed from UI, the marked task did > not update a valid end_date. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #7167: [AIRFLOW-5974] AIP-21 Change import paths for celery modules
codecov-io edited a comment on issue #7167: [AIRFLOW-5974] AIP-21 Change import paths for celery modules URL: https://github.com/apache/airflow/pull/7167#issuecomment-574911784 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7167?src=pr=h1) Report > Merging [#7167](https://codecov.io/gh/apache/airflow/pull/7167?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/417feda5eba6cc63c01a9a04a7b48bc24f63a3d1?src=pr=desc) will **increase** coverage by `0.11%`. > The diff coverage is `71.87%`. [](https://codecov.io/gh/apache/airflow/pull/7167?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7167 +/- ## == + Coverage 85.25% 85.36% +0.11% == Files 723 724 +1 Lines 3954339547 +4 == + Hits3371133761 +50 + Misses 5832 5786 -46 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7167?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/sensors/celery\_queue\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvY2VsZXJ5X3F1ZXVlX3NlbnNvci5weQ==) | `0% <0%> (-82.15%)` | :arrow_down: | | [airflow/providers/celery/sensors/celery.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY2VsZXJ5L3NlbnNvcnMvY2VsZXJ5LnB5) | `82.14% <82.14%> (ø)` | | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <0%> (+0.43%)` | :arrow_up: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `91.73% <0%> (+1.65%)` | :arrow_up: | | [airflow/hooks/postgres\_hook.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9wb3N0Z3Jlc19ob29rLnB5) | `94.36% <0%> (+16.9%)` | :arrow_up: | | [airflow/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfdG9fZ2NzLnB5) | `85.29% <0%> (+32.35%)` | :arrow_up: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `100% <0%> (+100%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7167?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7167?src=pr=footer). Last update [417feda...0a878d7](https://codecov.io/gh/apache/airflow/pull/7167?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7185: [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking
codecov-io edited a comment on issue #7185: [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking URL: https://github.com/apache/airflow/pull/7185#issuecomment-574910352 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=h1) Report > Merging [#7185](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/417feda5eba6cc63c01a9a04a7b48bc24f63a3d1?src=pr=desc) will **increase** coverage by `0.12%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7185 +/- ## == + Coverage 85.25% 85.37% +0.12% == Files 723 723 Lines 3954339543 == + Hits3371133761 +50 + Misses 5832 5782 -50 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <0%> (+0.43%)` | :arrow_up: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `91.73% <0%> (+1.65%)` | :arrow_up: | | [airflow/hooks/postgres\_hook.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9wb3N0Z3Jlc19ob29rLnB5) | `94.36% <0%> (+16.9%)` | :arrow_up: | | [airflow/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfdG9fZ2NzLnB5) | `85.29% <0%> (+32.35%)` | :arrow_up: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `100% <0%> (+100%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=footer). Last update [417feda...b7568be](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7167: [AIRFLOW-5974] AIP-21 Change import paths for celery modules
codecov-io commented on issue #7167: [AIRFLOW-5974] AIP-21 Change import paths for celery modules URL: https://github.com/apache/airflow/pull/7167#issuecomment-574911784 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7167?src=pr=h1) Report > Merging [#7167](https://codecov.io/gh/apache/airflow/pull/7167?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/417feda5eba6cc63c01a9a04a7b48bc24f63a3d1?src=pr=desc) will **decrease** coverage by `0.62%`. > The diff coverage is `71.87%`. [](https://codecov.io/gh/apache/airflow/pull/7167?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7167 +/- ## == - Coverage 85.25% 84.62% -0.63% == Files 723 724 +1 Lines 3954339547 +4 == - Hits3371133467 -244 - Misses 5832 6080 +248 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7167?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/sensors/celery\_queue\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvY2VsZXJ5X3F1ZXVlX3NlbnNvci5weQ==) | `0% <0%> (-82.15%)` | :arrow_down: | | [airflow/providers/celery/sensors/celery.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY2VsZXJ5L3NlbnNvcnMvY2VsZXJ5LnB5) | `82.14% <82.14%> (ø)` | | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.49%)` | :arrow_down: | | ... and [8 more](https://codecov.io/gh/apache/airflow/pull/7167/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7167?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7167?src=pr=footer). Last update [417feda...0a878d7](https://codecov.io/gh/apache/airflow/pull/7167?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7185: [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking
codecov-io commented on issue #7185: [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking URL: https://github.com/apache/airflow/pull/7185#issuecomment-574910352 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=h1) Report > Merging [#7185](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/417feda5eba6cc63c01a9a04a7b48bc24f63a3d1?src=pr=desc) will **decrease** coverage by `0.62%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7185 +/- ## == - Coverage 85.25% 84.62% -0.63% == Files 723 723 Lines 3954339543 == - Hits3371133463 -248 - Misses 5832 6080 +248 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.49%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `86.66% <0%> (-10%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `85.52% <0%> (-2.67%)` | :arrow_down: | | ... and [5 more](https://codecov.io/gh/apache/airflow/pull/7185/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=footer). Last update [417feda...b7568be](https://codecov.io/gh/apache/airflow/pull/7185?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work started] (AIRFLOW-4355) Externally triggered DAG is marked as 'success' even if a task has been 'removed'!
[
https://issues.apache.org/jira/browse/AIRFLOW-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on AIRFLOW-4355 started by t oo.
-
> Externally triggered DAG is marked as 'success' even if a task has been
> 'removed'!
> --
>
> Key: AIRFLOW-4355
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4355
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, DagRun, scheduler
>Affects Versions: 1.10.3
>Reporter: t oo
>Assignee: t oo
>Priority: Blocker
> Labels: dynamic
> Fix For: 2.0.0
>
> Attachments: dag_success_even_if_task_removed.png, treeview.png
>
>
> note: all my dags are purely externally triggered
> *Issue:* Dag has 5 parallel tasks that ran successfully and 1 final task that
> somehow got 'removed' state (prior dag runs had 'failed' state) and never ran
> successfully but still the DAG is showing success!
>
> *Command ran* (note that previous commands like airflow trigger_dag -e
> 20190412 qsr_coremytbl were run before and failed for valid reason (ie python
> task failing) ):
> airflow trigger_dag -e 20190412T08:00 qsr_coremytbl --conf '\{"hourstr":"08"}'
>
> *some logs on prior instance of airflow (ec2 was autohealed):*
> [2019-04-18 08:29:40,678] \{logging_mixin.py:95} INFO - [2019-04-18
> 08:29:40,678] {__init__.py:4897} WARNING - Failed to get task ' qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl 2019-04-12 08:00:00+00:00 [None]>'
> for dag ''. Marking it as removed.
> [2019-04-18 08:29:43,582] \{logging_mixin.py:95} INFO - [2019-04-18
> 08:29:43,582] {__init__.py:4906} INFO - Restoring task ' qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl 2019-04-12 08:00:00+00:00 [removed]>'
> which was previously removed from DAG ''
> [2019-04-18 08:29:43,618] \{jobs.py:1787} INFO - Creating / updating
> 08:00:00+00:00 [scheduled]> in ORM
> [2019-04-18 08:29:43,676] \{logging_mixin.py:95} INFO - [2019-04-18
> 08:29:43,676] {__init__.py:4897} WARNING - Failed to get task ' qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl 2019-04-12 08:00:00+00:00
> [scheduled]>' for dag ''. Marking it as removed.
>
> *some logs on newer ec2:*
> [myuser@host logs]$ grep -i hive -R * | sed 's#[0-9]#x#g' | sort | uniq -c |
> grep -v 'airflow-webserver-access.log'
> 2 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx qsr_coremytbl
> REPAIR_HIVE_schemeh.mytbl log -xx-xx xx:xx:xx.xx rsawyerx
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id',
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl')]
> 1 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx qsr_coremytbl
> REPAIR_HIVE_schemeh.mytbl log -xx-xx xx:xx:xx.xx rsawyerx
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id',
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl'), ('format',
> u'json')]
> 1 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx qsr_coremytbl
> REPAIR_HIVE_schemeh.mytbl rendered -xx-xx xx:xx:xx.xx rsawyerx
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id',
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl')]
> 1 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx qsr_coremytbl
> REPAIR_HIVE_schemeh.mytbl task -xx-xx xx:xx:xx.xx rsawyerx
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id',
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl')]
> 1 scheduler/latest/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx]
> \{jobs.py:} INFO - Creating / updating qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl -xx-xx xx:xx:xx+xx:xx
> [scheduled]> in ORM
> 71 scheduler/latest/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx]
> \{logging_mixin.py:xx} INFO - [-xx-xx xx:xx:xx,xxx] {__init__.py:}
> INFO - Restoring task ' -xx-xx xx:xx:xx+xx:xx [removed]>' which was previously removed from DAG
> ''
> 1 scheduler/-xx-xx/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx]
> \{jobs.py:} INFO - Creating / updating qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl -xx-xx xx:xx:xx+xx:xx
> [scheduled]> in ORM
> 71 scheduler/-xx-xx/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx]
> \{logging_mixin.py:xx} INFO - [-xx-xx xx:xx:xx,xxx] {__init__.py:}
> INFO - Restoring task ' -xx-xx xx:xx:xx+xx:xx [removed]>' which was previously removed from DAG
> ''
>
> mysql> *select * from task_instance where task_id like '%REP%';#*
>
> +--+-+++++--++-++--++--++--++++--+-
[jira] [Work started] (AIRFLOW-6473) Show conf in response of dag_state cli command
[
https://issues.apache.org/jira/browse/AIRFLOW-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on AIRFLOW-6473 started by t oo.
-
> Show conf in response of dag_state cli command
> --
>
> Key: AIRFLOW-6473
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6473
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli
>Affects Versions: 1.10.7
>Reporter: t oo
>Assignee: t oo
>Priority: Major
>
> key-value pairs of conf can be passed into trigger_dag command
> ie
> --conf '
> {"ric":"amzn"}
> '
> when running dag_state cli command to get the state (ie success/failed) , the
> conf json should also be returned
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6473) Show conf in response of dag_state cli command
[
https://issues.apache.org/jira/browse/AIRFLOW-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016397#comment-17016397
]
ASF GitHub Bot commented on AIRFLOW-6473:
-
tooptoop4 commented on pull request #7186: [AIRFLOW-6473] Show conf in response
of dag_state cli command
URL: https://github.com/apache/airflow/pull/7186
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Show conf in response of dag_state cli command
> --
>
> Key: AIRFLOW-6473
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6473
> Project: Apache Airflow
> Issue Type: Improvement
> Components: cli
>Affects Versions: 1.10.7
>Reporter: t oo
>Priority: Major
>
> key-value pairs of conf can be passed into trigger_dag command
> ie
> --conf '
> {"ric":"amzn"}
> '
> when running dag_state cli command to get the state (ie success/failed) , the
> conf json should also be returned
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries URL: https://github.com/apache/airflow/pull/6792#discussion_r367161194 ## File path: airflow/jobs/scheduler_job.py ## @@ -1306,33 +1271,32 @@ def _enqueue_task_instances_with_queued_state(self, simple_dag_bag, :param simple_dag_bag: Should contains all of the task_instances' dags :type simple_dag_bag: airflow.utils.dag_processing.SimpleDagBag """ -TI = models.TaskInstance # actually enqueue them -for simple_task_instance in simple_task_instances: -simple_dag = simple_dag_bag.get_dag(simple_task_instance.dag_id) -command = TI.generate_command( -simple_task_instance.dag_id, -simple_task_instance.task_id, -simple_task_instance.execution_date, +for ti in task_instances: +simple_dag = simple_dag_bag.get_dag(ti.dag_id) +command = ti.generate_command( +ti.dag_id, +ti.task_id, +ti.execution_date, local=True, mark_success=False, ignore_all_deps=False, ignore_depends_on_past=False, ignore_task_deps=False, ignore_ti_state=False, -pool=simple_task_instance.pool, +pool=ti.pool, file_path=simple_dag.full_filepath, pickle_id=simple_dag.pickle_id) -priority = simple_task_instance.priority_weight -queue = simple_task_instance.queue +priority = ti.priority_weight +queue = ti.queue self.log.info( "Sending %s to executor with priority %s and queue %s", -simple_task_instance.key, priority, queue +ti.key, priority, queue ) self.executor.queue_command( Review comment: I've put a TODO comment in here. Not happy about doing that but I think this PR is big enough already. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] tooptoop4 opened a new pull request #7186: [AIRFLOW-6473] Show conf in response of dag_state cli command
tooptoop4 opened a new pull request #7186: [AIRFLOW-6473] Show conf in response of dag_state cli command URL: https://github.com/apache/airflow/pull/7186 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7184: [AIRFLOW-6574] Adding private_environment to docker operator.
codecov-io edited a comment on issue #7184: [AIRFLOW-6574] Adding private_environment to docker operator. URL: https://github.com/apache/airflow/pull/7184#issuecomment-574899987 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7184?src=pr=h1) Report > Merging [#7184](https://codecov.io/gh/apache/airflow/pull/7184?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8729ca525ccd2c976248b818440932242720a25b?src=pr=desc) will **decrease** coverage by `0.14%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7184?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7184 +/- ## == - Coverage 85.24% 85.09% -0.15% == Files 710 723 +13 Lines 3948539544 +59 == - Hits3366033651 -9 - Misses 5825 5893 +68 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7184?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZG9ja2VyX29wZXJhdG9yLnB5) | `96.9% <100%> (+0.03%)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/contrib/hooks/azure\_data\_lake\_hook.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F6dXJlX2RhdGFfbGFrZV9ob29rLnB5) | `0% <0%> (-93.11%)` | :arrow_down: | | [airflow/contrib/sensors/azure\_cosmos\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvYXp1cmVfY29zbW9zX3NlbnNvci5weQ==) | `0% <0%> (-81.25%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.49%)` | :arrow_down: | | ... and [36 more](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7184?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7184?src=pr=footer). Last update [8729ca5...33bf3b7](https://codecov.io/gh/apache/airflow/pull/7184?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries URL: https://github.com/apache/airflow/pull/6792#discussion_r367156221 ## File path: airflow/jobs/scheduler_job.py ## @@ -1306,33 +1271,32 @@ def _enqueue_task_instances_with_queued_state(self, simple_dag_bag, :param simple_dag_bag: Should contains all of the task_instances' dags :type simple_dag_bag: airflow.utils.dag_processing.SimpleDagBag """ -TI = models.TaskInstance # actually enqueue them -for simple_task_instance in simple_task_instances: -simple_dag = simple_dag_bag.get_dag(simple_task_instance.dag_id) -command = TI.generate_command( -simple_task_instance.dag_id, -simple_task_instance.task_id, -simple_task_instance.execution_date, +for ti in task_instances: +simple_dag = simple_dag_bag.get_dag(ti.dag_id) +command = ti.generate_command( +ti.dag_id, +ti.task_id, +ti.execution_date, local=True, mark_success=False, ignore_all_deps=False, ignore_depends_on_past=False, ignore_task_deps=False, ignore_ti_state=False, -pool=simple_task_instance.pool, +pool=ti.pool, file_path=simple_dag.full_filepath, pickle_id=simple_dag.pickle_id) -priority = simple_task_instance.priority_weight -queue = simple_task_instance.queue +priority = ti.priority_weight +queue = ti.queue self.log.info( "Sending %s to executor with priority %s and queue %s", -simple_task_instance.key, priority, queue +ti.key, priority, queue ) self.executor.queue_command( Review comment: Fixing this properly will also fix https://issues.apache.org/jira/browse/AIRFLOW-3877 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7184: [AIRFLOW-6574] Adding private_environment to docker operator.
codecov-io commented on issue #7184: [AIRFLOW-6574] Adding private_environment to docker operator. URL: https://github.com/apache/airflow/pull/7184#issuecomment-574899987 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7184?src=pr=h1) Report > Merging [#7184](https://codecov.io/gh/apache/airflow/pull/7184?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8729ca525ccd2c976248b818440932242720a25b?src=pr=desc) will **decrease** coverage by `0.35%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7184?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7184 +/- ## == - Coverage 85.24% 84.88% -0.36% == Files 710 723 +13 Lines 3948539544 +59 == - Hits3366033568 -92 - Misses 5825 5976 +151 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7184?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZG9ja2VyX29wZXJhdG9yLnB5) | `96.9% <100%> (+0.03%)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/hooks/azure\_data\_lake\_hook.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F6dXJlX2RhdGFfbGFrZV9ob29rLnB5) | `0% <0%> (-93.11%)` | :arrow_down: | | [airflow/contrib/sensors/azure\_cosmos\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvYXp1cmVfY29zbW9zX3NlbnNvci5weQ==) | `0% <0%> (-81.25%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.49%)` | :arrow_down: | | ... and [36 more](https://codecov.io/gh/apache/airflow/pull/7184/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7184?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7184?src=pr=footer). Last update [8729ca5...33bf3b7](https://codecov.io/gh/apache/airflow/pull/7184?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries URL: https://github.com/apache/airflow/pull/6792#discussion_r367154648 ## File path: airflow/jobs/scheduler_job.py ## @@ -1306,33 +1271,32 @@ def _enqueue_task_instances_with_queued_state(self, simple_dag_bag, :param simple_dag_bag: Should contains all of the task_instances' dags :type simple_dag_bag: airflow.utils.dag_processing.SimpleDagBag """ -TI = models.TaskInstance # actually enqueue them -for simple_task_instance in simple_task_instances: -simple_dag = simple_dag_bag.get_dag(simple_task_instance.dag_id) -command = TI.generate_command( -simple_task_instance.dag_id, -simple_task_instance.task_id, -simple_task_instance.execution_date, +for ti in task_instances: +simple_dag = simple_dag_bag.get_dag(ti.dag_id) +command = ti.generate_command( +ti.dag_id, +ti.task_id, +ti.execution_date, local=True, mark_success=False, ignore_all_deps=False, ignore_depends_on_past=False, ignore_task_deps=False, ignore_ti_state=False, -pool=simple_task_instance.pool, +pool=ti.pool, file_path=simple_dag.full_filepath, pickle_id=simple_dag.pickle_id) -priority = simple_task_instance.priority_weight -queue = simple_task_instance.queue +priority = ti.priority_weight +queue = ti.queue self.log.info( "Sending %s to executor with priority %s and queue %s", -simple_task_instance.key, priority, queue +ti.key, priority, queue ) self.executor.queue_command( Review comment: This is actually more complex than it seems -- calling queue_task_instance sends it via `ti.command_as_list`, but to get the same behaviour with filepath we'd need to have ti.task.dag be set, but the ti doesn't have a task set. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7185: [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking
potiuk commented on issue #7185: [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking URL: https://github.com/apache/airflow/pull/7185#issuecomment-574898482 Let's see when Travis completes, but I believe this change will speed up the startup - I changed the entropy source to be unblocking one and it should make the startup faster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6575) Change entropy source in all ci/breeze containers to urandom (unblocking)
[ https://issues.apache.org/jira/browse/AIRFLOW-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016388#comment-17016388 ] ASF GitHub Bot commented on AIRFLOW-6575: - potiuk commented on pull request #7185: [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking URL: https://github.com/apache/airflow/pull/7185 On Travis CI blocking entropy source slows startup time of a number of containers. This change changes the entropy source to unblocking one. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Change entropy source in all ci/breeze containers to urandom (unblocking) > - > > Key: AIRFLOW-6575 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6575 > Project: Apache Airflow > Issue Type: Improvement > Components: breeze, ci >Affects Versions: 2.0.0, 1.10.7 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk opened a new pull request #7185: [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking
potiuk opened a new pull request #7185: [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking URL: https://github.com/apache/airflow/pull/7185 On Travis CI blocking entropy source slows startup time of a number of containers. This change changes the entropy source to unblocking one. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries URL: https://github.com/apache/airflow/pull/6792#discussion_r367151692 ## File path: airflow/jobs/scheduler_job.py ## @@ -1306,33 +1271,32 @@ def _enqueue_task_instances_with_queued_state(self, simple_dag_bag, :param simple_dag_bag: Should contains all of the task_instances' dags :type simple_dag_bag: airflow.utils.dag_processing.SimpleDagBag """ -TI = models.TaskInstance # actually enqueue them -for simple_task_instance in simple_task_instances: -simple_dag = simple_dag_bag.get_dag(simple_task_instance.dag_id) -command = TI.generate_command( -simple_task_instance.dag_id, -simple_task_instance.task_id, -simple_task_instance.execution_date, +for ti in task_instances: +simple_dag = simple_dag_bag.get_dag(ti.dag_id) +command = ti.generate_command( +ti.dag_id, +ti.task_id, +ti.execution_date, local=True, mark_success=False, ignore_all_deps=False, ignore_depends_on_past=False, ignore_task_deps=False, ignore_ti_state=False, -pool=simple_task_instance.pool, +pool=ti.pool, file_path=simple_dag.full_filepath, pickle_id=simple_dag.pickle_id) -priority = simple_task_instance.priority_weight -queue = simple_task_instance.queue +priority = ti.priority_weight +queue = ti.queue self.log.info( "Sending %s to executor with priority %s and queue %s", -simple_task_instance.key, priority, queue +ti.key, priority, queue ) self.executor.queue_command( Review comment: Oh queue_task_instance already exists -- I forgot you(?) added that already This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6575) Change entropy source in all ci/breeze containers to urandom (unblocking)
Jarek Potiuk created AIRFLOW-6575: - Summary: Change entropy source in all ci/breeze containers to urandom (unblocking) Key: AIRFLOW-6575 URL: https://issues.apache.org/jira/browse/AIRFLOW-6575 Project: Apache Airflow Issue Type: Improvement Components: breeze, ci Affects Versions: 1.10.7, 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries URL: https://github.com/apache/airflow/pull/6792#discussion_r367151106 ## File path: airflow/models/dagrun.py ## @@ -268,10 +290,14 @@ def update_state(self, session=None): Determines the overall state of the DagRun based on the state of its TaskInstances. -:return: State +:return: state, schedulable_task_instances +:rtype: (State, list[airflow.models.TaskInstance]) Review comment: Yup, that's what I'm changing it too. Link for reference http://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists (at the end of that section, just before "Cross-referencing Python objects") This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
ashb commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#discussion_r367147960 ## File path: tests/models/test_dagrun.py ## @@ -234,8 +234,9 @@ def test_dagrun_deadlock(self): ti_op2.set_state(state=State.NONE, session=session) dr.update_state() -self.assertEqual(dr.state, State.RUNNING) +self.assertEqual(dr.state, State.FAILED) Review comment: This _does_ look more right now, I just wonder why it was in RUNNING state before? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
ashb commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#discussion_r367147759 ## File path: tests/models/test_dagrun.py ## @@ -234,8 +234,9 @@ def test_dagrun_deadlock(self): ti_op2.set_state(state=State.NONE, session=session) dr.update_state() -self.assertEqual(dr.state, State.RUNNING) +self.assertEqual(dr.state, State.FAILED) Review comment: I'm still not 100% on this change. I'll hope to look at this tomorrow (because otherwise I'll merge https://github.com/apache/airflow/pull/6792/files which is going to cause a number of conflicts for one of us) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure
codecov-io edited a comment on issue #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure URL: https://github.com/apache/airflow/pull/7172#issuecomment-574856548 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=h1) Report > Merging [#7172](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8729ca525ccd2c976248b818440932242720a25b?src=pr=desc) will **increase** coverage by `0.13%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7172 +/- ## == + Coverage 85.24% 85.37% +0.13% == Files 710 723 +13 Lines 3948539543 +58 == + Hits3366033761 +101 + Misses 5825 5782 -43 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: | | [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `95.12% <50%> (-2.32%)` | :arrow_down: | | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `97.93% <50%> (-2.07%)` | :arrow_down: | | [airflow/contrib/hooks/azure\_data\_lake\_hook.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F6dXJlX2RhdGFfbGFrZV9ob29rLnB5) | `0% <0%> (-93.11%)` | :arrow_down: | | [airflow/contrib/sensors/azure\_cosmos\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvYXp1cmVfY29zbW9zX3NlbnNvci5weQ==) | `0% <0%> (-81.25%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `88.01% <0%> (-3.43%)` | :arrow_down: | | [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/contrib/operators/adls\_list\_operator.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9hZGxzX2xpc3Rfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/contrib/hooks/azure\_fileshare\_hook.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F6dXJlX2ZpbGVzaGFyZV9ob29rLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/contrib/sensors/wasb\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvd2FzYl9zZW5zb3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | ... and [25 more](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=footer). Last update [8729ca5...5f23840](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
mik-laj commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries URL: https://github.com/apache/airflow/pull/6792#discussion_r367145960 ## File path: airflow/models/dagrun.py ## @@ -268,10 +290,14 @@ def update_state(self, session=None): Determines the overall state of the DagRun based on the state of its TaskInstances. -:return: State +:return: state, schedulable_task_instances +:rtype: (State, list[airflow.models.TaskInstance]) Review comment: It seemed to me that valid PEP types were supported. In that case, it probably should be `tuple[State, list[airflow.models.TaskInstance]]` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries URL: https://github.com/apache/airflow/pull/6792#discussion_r367144619 ## File path: airflow/models/dagrun.py ## @@ -268,10 +290,14 @@ def update_state(self, session=None): Determines the overall state of the DagRun based on the state of its TaskInstances. -:return: State +:return: state, schedulable_task_instances +:rtype: (State, list[airflow.models.TaskInstance]) Review comment: This is what the current version renders as:  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-6564) Display extra diagnostics if initial environment check fails
[ https://issues.apache.org/jira/browse/AIRFLOW-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-6564. --- Fix Version/s: 2.0.0 Resolution: Fixed > Display extra diagnostics if initial environment check fails > > > Key: AIRFLOW-6564 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6564 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6564) Display extra diagnostics if initial environment check fails
[ https://issues.apache.org/jira/browse/AIRFLOW-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016379#comment-17016379 ] ASF GitHub Bot commented on AIRFLOW-6564: - potiuk commented on pull request #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure URL: https://github.com/apache/airflow/pull/7172 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Display extra diagnostics if initial environment check fails > > > Key: AIRFLOW-6564 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6564 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure
codecov-io edited a comment on issue #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure URL: https://github.com/apache/airflow/pull/7172#issuecomment-574856548 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=h1) Report > Merging [#7172](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8729ca525ccd2c976248b818440932242720a25b?src=pr=desc) will **decrease** coverage by `0.07%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7172 +/- ## == - Coverage 85.24% 85.16% -0.08% == Files 710 723 +13 Lines 3948539543 +58 == + Hits3366033678 +18 - Misses 5825 5865 +40 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: | | [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `95.12% <50%> (-2.32%)` | :arrow_down: | | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `97.93% <50%> (-2.07%)` | :arrow_down: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/hooks/azure\_data\_lake\_hook.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F6dXJlX2RhdGFfbGFrZV9ob29rLnB5) | `0% <0%> (-93.11%)` | :arrow_down: | | [airflow/contrib/sensors/azure\_cosmos\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvYXp1cmVfY29zbW9zX3NlbnNvci5weQ==) | `0% <0%> (-81.25%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `88.01% <0%> (-3.43%)` | :arrow_down: | | [airflow/providers/apache/hive/hooks/hive.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvaG9va3MvaGl2ZS5weQ==) | `76.02% <0%> (-1.54%)` | :arrow_down: | | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `67.8% <0%> (-0.98%)` | :arrow_down: | | ... and [30 more](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=footer). Last update [8729ca5...5f23840](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk merged pull request #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure
potiuk merged pull request #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure URL: https://github.com/apache/airflow/pull/7172 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries URL: https://github.com/apache/airflow/pull/6792#discussion_r367144376 ## File path: airflow/models/dagrun.py ## @@ -268,10 +290,14 @@ def update_state(self, session=None): Determines the overall state of the DagRun based on the state of its TaskInstances. -:return: State +:return: state, schedulable_task_instances +:rtype: (State, list[airflow.models.TaskInstance]) Review comment: For sphinx docs it should be `tuple` and `list`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7157: [AIRFLOW-6251] add config for max tasks per dag
codecov-io commented on issue #7157: [AIRFLOW-6251] add config for max tasks per dag URL: https://github.com/apache/airflow/pull/7157#issuecomment-574886874 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7157?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@92521aa`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `86.66%`. [](https://codecov.io/gh/apache/airflow/pull/7157?src=pr=tree) ```diff @@ Coverage Diff@@ ## master #7157 +/- ## Coverage ? 85.3% Files ? 723 Lines ? 40701 Branches ? 0 Hits ? 34720 Misses?5981 Partials ? 0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7157?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/pool.py](https://codecov.io/gh/apache/airflow/pull/7157/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvcG9vbC5weQ==) | `97.36% <ø> (ø)` | | | [airflow/contrib/operators/databricks\_operator.py](https://codecov.io/gh/apache/airflow/pull/7157/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9kYXRhYnJpY2tzX29wZXJhdG9yLnB5) | `92.24% <ø> (ø)` | | | [.../providers/apache/spark/hooks/spark\_jdbc\_script.py](https://codecov.io/gh/apache/airflow/pull/7157/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL3NwYXJrL2hvb2tzL3NwYXJrX2pkYmNfc2NyaXB0LnB5) | `0% <ø> (ø)` | | | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7157/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | | | [...ders/google/cloud/example\_dags/example\_dataproc.py](https://codecov.io/gh/apache/airflow/pull/7157/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX2RhdGFwcm9jLnB5) | `0% <0%> (ø)` | | | [airflow/macros/hive.py](https://codecov.io/gh/apache/airflow/pull/7157/diff?src=pr=tree#diff-YWlyZmxvdy9tYWNyb3MvaGl2ZS5weQ==) | `38.7% <0%> (ø)` | | | [airflow/utils/log/gcs\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/7157/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZ2NzX3Rhc2tfaGFuZGxlci5weQ==) | `0% <0%> (ø)` | | | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/7157/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `68.78% <0%> (ø)` | | | [airflow/contrib/operators/vertica\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7157/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy92ZXJ0aWNhX3RvX2hpdmUucHk=) | `0% <0%> (ø)` | | | [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7157/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.59% <100%> (ø)` | | | ... and [98 more](https://codecov.io/gh/apache/airflow/pull/7157/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7157?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7157?src=pr=footer). Last update [92521aa...bf5d94d](https://codecov.io/gh/apache/airflow/pull/7157?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
ashb commented on issue #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries URL: https://github.com/apache/airflow/pull/6792#issuecomment-574885270 Tests aren't quite passing -- one failure due to a non-fixed ordering of return values. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries URL: https://github.com/apache/airflow/pull/6792#discussion_r367138242 ## File path: airflow/jobs/scheduler_job.py ## @@ -1306,33 +1271,32 @@ def _enqueue_task_instances_with_queued_state(self, simple_dag_bag, :param simple_dag_bag: Should contains all of the task_instances' dags :type simple_dag_bag: airflow.utils.dag_processing.SimpleDagBag """ -TI = models.TaskInstance # actually enqueue them -for simple_task_instance in simple_task_instances: -simple_dag = simple_dag_bag.get_dag(simple_task_instance.dag_id) -command = TI.generate_command( -simple_task_instance.dag_id, -simple_task_instance.task_id, -simple_task_instance.execution_date, +for ti in task_instances: +simple_dag = simple_dag_bag.get_dag(ti.dag_id) +command = ti.generate_command( +ti.dag_id, +ti.task_id, +ti.execution_date, local=True, mark_success=False, ignore_all_deps=False, ignore_depends_on_past=False, ignore_task_deps=False, ignore_ti_state=False, -pool=simple_task_instance.pool, +pool=ti.pool, file_path=simple_dag.full_filepath, pickle_id=simple_dag.pickle_id) -priority = simple_task_instance.priority_weight -queue = simple_task_instance.queue +priority = ti.priority_weight +queue = ti.queue self.log.info( "Sending %s to executor with priority %s and queue %s", -simple_task_instance.key, priority, queue +ti.key, priority, queue ) self.executor.queue_command( Review comment: I guess we should This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #7184: [AIRFLOW 6574] Adding private_environment to docker operator.
boring-cyborg[bot] commented on issue #7184: [AIRFLOW 6574] Adding private_environment to docker operator. URL: https://github.com/apache/airflow/pull/7184#issuecomment-574878742 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://apache-airflow-slack.herokuapp.com/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] CaptainCuddleCube opened a new pull request #7184: [AIRFLOW 6574] Adding private_environment to docker operator.
CaptainCuddleCube opened a new pull request #7184: [AIRFLOW 6574] Adding private_environment to docker operator. URL: https://github.com/apache/airflow/pull/7184 The docker operator currently does not have a means to pass in an environment dict that is not exposed by the frontend's task instance view, along with all the other Task instance attributes. This PR proposes a private_environment parameter that should prevent the frontend from exposes this attribute. This would be useful for the cases where the docker container will be getting the passwords by the environment variables. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6569) Broken sentry integration
[
https://issues.apache.org/jira/browse/AIRFLOW-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016356#comment-17016356
]
Mike Clarke commented on AIRFLOW-6569:
--
[~robinedwards] There's a public flush interface exposed at
`sentry_sdk.Hub.current.flush()`, it seems like this might be a good solution
for making sure the exceptions are flushed before the `os._exit` call if sentry
is configured/enabled. Does that sound reasonable?
{quote}it doesn't collect errors from the rest of the system only tasks
{quote}
Can you share a bit more about what you mean here?
> Broken sentry integration
> -
>
> Key: AIRFLOW-6569
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6569
> Project: Apache Airflow
> Issue Type: Bug
> Components: configuration, hooks
>Affects Versions: 2.0.0, 1.10.7
>Reporter: Robin Edwards
>Priority: Minor
>
> I believe the new forking mechanism AIRFLOW-5931 has unintentionally broken
> the sentry integration.
> Sentry relies on the atexit
> http://man7.org/linux/man-pages/man3/atexit.3.html signal to flush collected
> errors to their servers. Previously as the task was executed in a new process
> as opposed to forked this got invoked. However now os._exit() is called
> (which is semantically correct with child processes)
> https://docs.python.org/3/library/os.html#os._exit
> Point os._exit is called in airflow:
> https://github.com/apache/airflow/pull/6627/files#diff-736081a3535ff0b9e60ada2f51154ca4R84
> Also related on sentry bug tracker:
> https://github.com/getsentry/sentry-python/issues/291
> Unfortunately sentry doesn't provide (from what i can find) a public
> interface for flushing errors to their system. The return value of their
> init() functions returns an object containg a client but the property is
> `_client` so it would be wrong to rely on it.
> I've side stepped this in two ways, you can disable the forking feature
> through patching CAN_FORK to False. But after seeing the performance
> improvement on my workers I opted to monkey patch the whole _exec_by_fork()
> and naughtily call sys.exit instead as a temporary fix.
> I personally dont find the actual sentry integration in airflow useful as it
> doesn't collect errors from the rest of the system only tasks. I've been
> wiring it in through my log config module since before the integration was
> added however its still effected by the above change.
> My personal vote (unless anyone has a better idea) would be to remove the
> integration completely document the way of setting it up through the logging
> class and providing a 'post_execute' hook of some form on the
> StandardTaskRunner where people can flush errors using what not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #7162: [AIRFLOW-6557] Add test for newly added fields in BaseOperator
codecov-io edited a comment on issue #7162: [AIRFLOW-6557] Add test for newly added fields in BaseOperator URL: https://github.com/apache/airflow/pull/7162#issuecomment-574227743 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=h1) Report > Merging [#7162](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b38766e83495803e0a95912aaa25d727938fc9c1?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7162 +/- ## == - Coverage85.4% 85.12% -0.29% == Files 723 723 Lines 3953739537 == - Hits3376833657 -111 - Misses 5769 5880 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.49%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=footer). Last update [b38766e...2acd697](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7162: [AIRFLOW-6557] Add test for newly added fields in BaseOperator
codecov-io edited a comment on issue #7162: [AIRFLOW-6557] Add test for newly added fields in BaseOperator URL: https://github.com/apache/airflow/pull/7162#issuecomment-574227743 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=h1) Report > Merging [#7162](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b38766e83495803e0a95912aaa25d727938fc9c1?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7162 +/- ## == - Coverage85.4% 85.12% -0.29% == Files 723 723 Lines 3953739537 == - Hits3376833657 -111 - Misses 5769 5880 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfdG9fZ2NzLnB5) | `85.29% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.49%)` | :arrow_down: | | [airflow/hooks/postgres\_hook.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9wb3N0Z3Jlc19ob29rLnB5) | `94.36% <0%> (ø)` | :arrow_up: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `91.73% <0%> (ø)` | :arrow_up: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <0%> (ø)` | :arrow_up: | | ... and [4 more](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=footer). Last update [b38766e...2acd697](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6574) Docker operator needs a private environment dict
Ashton Hudson created AIRFLOW-6574: -- Summary: Docker operator needs a private environment dict Key: AIRFLOW-6574 URL: https://issues.apache.org/jira/browse/AIRFLOW-6574 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: 1.10.7 Reporter: Ashton Hudson Assignee: Ashton Hudson The docker operator currently assigns the environment dict to the operators self, which leads to the effect that when viewing the task in the browser, it displays all the environment variables. This is an issue if the docker container gets it's database credentials via the environment variables. A proposed solution is to create a private_environment dict that is added to the operator's class with a leading underscore. Since the browser renderer excludes all class attributes with a leading underscore - the information won't be leaked. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL query building for hot-path queries
ashb commented on a change in pull request #6792: [AIRFLOW-5930] Use cached-SQL
query building for hot-path queries
URL: https://github.com/apache/airflow/pull/6792#discussion_r367127376
##
File path: airflow/jobs/scheduler_job.py
##
@@ -1282,21 +1253,15 @@ def _change_state_for_executable_task_instances(self,
task_instances,
task_instance.queued_dttm = timezone.utcnow()
session.merge(task_instance)
-# Generate a list of SimpleTaskInstance for the use of queuing
-# them in the executor.
-simple_task_instances = [SimpleTaskInstance(ti) for ti in
- tis_to_set_to_queued]
-
task_instance_str = "\n\t".join(
[repr(x) for x in tis_to_set_to_queued])
session.commit()
self.log.info("Setting the following %s tasks to queued state:\n\t%s",
len(tis_to_set_to_queued), task_instance_str)
-return simple_task_instances
+return tis_to_set_to_queued
-def _enqueue_task_instances_with_queued_state(self, simple_dag_bag,
- simple_task_instances):
+def _enqueue_task_instances_with_queued_state(self, simple_dag_bag,
task_instances):
Review comment:
This is called with the result of
`self._change_state_for_executable_task_instances` which used to return this
```
simple_task_instances = [SimpleTaskInstance(ti) for ti in
tis_to_set_to_queued]
```
But one of the slow downs I noticed was continually re-looking up the TI, so
it has been changed to pass full TaskInstance objects around (as we are all in
the same process).
I'll update the docs to reflect the new code/type
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7162: [AIRFLOW-6557] Add test for newly added fields in BaseOperator
codecov-io edited a comment on issue #7162: [AIRFLOW-6557] Add test for newly added fields in BaseOperator URL: https://github.com/apache/airflow/pull/7162#issuecomment-574227743 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=h1) Report > Merging [#7162](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/b38766e83495803e0a95912aaa25d727938fc9c1?src=pr=desc) will **decrease** coverage by `52.15%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7162 +/- ## === - Coverage 84.94% 32.79% -52.16% === Files 723 722-1 Lines 3953739524 -13 === - Hits3358512961-20624 - Misses 595226563+20611 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=tree) | Coverage Δ | | |---|---|---| | [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [...flow/contrib/example\_dags/example\_qubole\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9zZW5zb3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [...ample\_dags/example\_emr\_job\_flow\_automatic\_steps.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2Vtcl9qb2JfZmxvd19hdXRvbWF0aWNfc3RlcHMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [...contrib/example\_dags/example\_papermill\_operator.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3BhcGVybWlsbF9vcGVyYXRvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/hooks/webhdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy93ZWJoZGZzX2hvb2sucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/operators/gcs\_list\_operator.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfbGlzdF9vcGVyYXRvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/pig\_operator.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcGlnX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/operators/file\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9maWxlX3RvX2djcy5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/hooks/gcp\_spanner\_hook.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9zcGFubmVyX2hvb2sucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/hive\_operator.py](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvaGl2ZV9vcGVyYXRvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | ... and [582 more](https://codecov.io/gh/apache/airflow/pull/7162/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=footer). Last update [b38766e...2acd697](https://codecov.io/gh/apache/airflow/pull/7162?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #7160: [AIRFLOW-1467] Dynamic pooling via allowing tasks to use more than one pool slot (depending upon the need)
kaxil commented on issue #7160: [AIRFLOW-1467] Dynamic pooling via allowing tasks to use more than one pool slot (depending upon the need) URL: https://github.com/apache/airflow/pull/7160#issuecomment-574870008 Thanks for waiting @lokeshlal , I am currently on Leave but would be reviewing it very soon :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure
potiuk commented on a change in pull request #7172: [AIRFLOW-6564] Additional
diagnostics information on CI check failure
URL: https://github.com/apache/airflow/pull/7172#discussion_r367118737
##
File path: airflow/bin/cli.py
##
@@ -837,6 +837,12 @@ class CLIFactory:
'help': "Runs a shell to access the database",
'args': tuple(),
},
+{
+'func': lazy_load_command('airflow.cli.commands.db_command.check'),
+'name': 'check',
+'help': "Chack if the database can be reached.",
Review comment:
Right!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure
kaxil commented on a change in pull request #7172: [AIRFLOW-6564] Additional
diagnostics information on CI check failure
URL: https://github.com/apache/airflow/pull/7172#discussion_r367117870
##
File path: airflow/bin/cli.py
##
@@ -837,6 +837,12 @@ class CLIFactory:
'help': "Runs a shell to access the database",
'args': tuple(),
},
+{
+'func': lazy_load_command('airflow.cli.commands.db_command.check'),
+'name': 'check',
+'help': "Chack if the database can be reached.",
Review comment:
```suggestion
'help': "Check if the database can be reached.",
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure
potiuk commented on a change in pull request #7172: [AIRFLOW-6564] Additional
diagnostics information on CI check failure
URL: https://github.com/apache/airflow/pull/7172#discussion_r367118278
##
File path: airflow/utils/db.py
##
@@ -501,3 +501,13 @@ def resetdb():
Base.metadata.drop_all(connection) # pylint: disable=no-member
initdb()
+
+
+@provide_session
+def check(session=None):
+"""
+Checks if the database works.
+:param session: session of the sqlalchemy
+"""
+session.execute('select 1 as is_alive;')
+log.info("Connection successful.")
Review comment:
Again - we do not have neither scheduler nor webserver nor any of the
workers. It's just checking if the environment we have is "sane" - i.e. if the
databases are there and can be queried, if all the integration for CI tests are
reachable at the destination endpoints.
We explicitly only check database availability before we start tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure
potiuk commented on a change in pull request #7172: [AIRFLOW-6564] Additional
diagnostics information on CI check failure
URL: https://github.com/apache/airflow/pull/7172#discussion_r367117375
##
File path: airflow/bin/cli.py
##
@@ -837,6 +837,12 @@ class CLIFactory:
'help': "Runs a shell to access the database",
'args': tuple(),
},
+{
+'func': lazy_load_command('airflow.cli.commands.db_command.check'),
+'name': 'check',
+'help': "Chack if the database can be reached.",
+'args': tuple(),
+},
Review comment:
But we do not run webserver nor airflow at this point yet. It's just
environment we check.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure
kaxil commented on a change in pull request #7172: [AIRFLOW-6564] Additional
diagnostics information on CI check failure
URL: https://github.com/apache/airflow/pull/7172#discussion_r367116140
##
File path: airflow/utils/db.py
##
@@ -501,3 +501,13 @@ def resetdb():
Base.metadata.drop_all(connection) # pylint: disable=no-member
initdb()
+
+
+@provide_session
+def check(session=None):
+"""
+Checks if the database works.
+:param session: session of the sqlalchemy
+"""
+session.execute('select 1 as is_alive;')
+log.info("Connection successful.")
Review comment:
WDYT?
If it is just for CI, how about just using curl to access the endpoint?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure
codecov-io edited a comment on issue #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure URL: https://github.com/apache/airflow/pull/7172#issuecomment-574856548 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=h1) Report > Merging [#7172](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8729ca525ccd2c976248b818440932242720a25b?src=pr=desc) will **increase** coverage by `0.13%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7172 +/- ## == + Coverage 85.24% 85.37% +0.13% == Files 710 723 +13 Lines 3948539543 +58 == + Hits3366033761 +101 + Misses 5825 5782 -43 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: | | [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `95.12% <50%> (-2.32%)` | :arrow_down: | | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `97.93% <50%> (-2.07%)` | :arrow_down: | | [airflow/contrib/hooks/azure\_data\_lake\_hook.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F6dXJlX2RhdGFfbGFrZV9ob29rLnB5) | `0% <0%> (-93.11%)` | :arrow_down: | | [airflow/contrib/sensors/azure\_cosmos\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvYXp1cmVfY29zbW9zX3NlbnNvci5weQ==) | `0% <0%> (-81.25%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `88.01% <0%> (-3.43%)` | :arrow_down: | | [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/contrib/operators/adls\_list\_operator.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9hZGxzX2xpc3Rfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/contrib/hooks/azure\_fileshare\_hook.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F6dXJlX2ZpbGVzaGFyZV9ob29rLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/contrib/sensors/wasb\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvd2FzYl9zZW5zb3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | ... and [25 more](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=footer). Last update [8729ca5...ead4811](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure
codecov-io edited a comment on issue #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure URL: https://github.com/apache/airflow/pull/7172#issuecomment-574856548 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=h1) Report > Merging [#7172](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8729ca525ccd2c976248b818440932242720a25b?src=pr=desc) will **increase** coverage by `0.13%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7172 +/- ## == + Coverage 85.24% 85.37% +0.13% == Files 710 723 +13 Lines 3948539543 +58 == + Hits3366033761 +101 + Misses 5825 5782 -43 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: | | [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `95.12% <50%> (-2.32%)` | :arrow_down: | | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `97.93% <50%> (-2.07%)` | :arrow_down: | | [airflow/contrib/hooks/azure\_data\_lake\_hook.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F6dXJlX2RhdGFfbGFrZV9ob29rLnB5) | `0% <0%> (-93.11%)` | :arrow_down: | | [airflow/contrib/sensors/azure\_cosmos\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvYXp1cmVfY29zbW9zX3NlbnNvci5weQ==) | `0% <0%> (-81.25%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `88.01% <0%> (-3.43%)` | :arrow_down: | | [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/contrib/operators/adls\_list\_operator.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9hZGxzX2xpc3Rfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/contrib/hooks/azure\_fileshare\_hook.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2F6dXJlX2ZpbGVzaGFyZV9ob29rLnB5) | `100% <0%> (ø)` | :arrow_up: | | [airflow/contrib/sensors/wasb\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvd2FzYl9zZW5zb3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | ... and [25 more](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=footer). Last update [8729ca5...ead4811](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure
kaxil commented on a change in pull request #7172: [AIRFLOW-6564] Additional
diagnostics information on CI check failure
URL: https://github.com/apache/airflow/pull/7172#discussion_r367115505
##
File path: airflow/utils/db.py
##
@@ -501,3 +501,13 @@ def resetdb():
Base.metadata.drop_all(connection) # pylint: disable=no-member
initdb()
+
+
+@provide_session
+def check(session=None):
+"""
+Checks if the database works.
+:param session: session of the sqlalchemy
+"""
+session.execute('select 1 as is_alive;')
+log.info("Connection successful.")
Review comment:
I think the `health` endpoint I mentioned above would be more useful as it
checks if the Scheduler can connect to the metadata DB.
`session.execute('select 1 as is_alive;')` would check if the machine where the
cli is run can execute that query on the Metadb -> i.e. it will check
connection between Machine running CLI with DB
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure
kaxil commented on a change in pull request #7172: [AIRFLOW-6564] Additional
diagnostics information on CI check failure
URL: https://github.com/apache/airflow/pull/7172#discussion_r367114338
##
File path: airflow/bin/cli.py
##
@@ -837,6 +837,12 @@ class CLIFactory:
'help': "Runs a shell to access the database",
'args': tuple(),
},
+{
+'func': lazy_load_command('airflow.cli.commands.db_command.check'),
+'name': 'check',
+'help': "Chack if the database can be reached.",
+'args': tuple(),
+},
Review comment:
We have the `/health` endpoint already, do you think that would cover this
and we won't need a new command?
Endpoint:
https://github.com/apache/airflow/blob/master/airflow/www/views.py#L182-L210
>An endpoint helping check the health status of the Airflow instance,
including metadatabase and scheduler.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates
kaxil commented on a change in pull request #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#discussion_r367111957 ## File path: airflow/models/dag.py ## @@ -781,13 +788,16 @@ def get_task_instances( start_date = (timezone.utcnow() - timedelta(30)).date() start_date = timezone.make_aware( datetime.combine(start_date, datetime.min.time())) -end_date = end_date or timezone.utcnow() + tis = session.query(TaskInstance).filter( TaskInstance.dag_id == self.dag_id, TaskInstance.execution_date >= start_date, -TaskInstance.execution_date <= end_date, TaskInstance.task_id.in_([t.task_id for t in self.tasks]), ) +# This allows ALLOW_TRIGGER_IN_FUTURE config to take affect, rather than mandating exec_date <= UTC +if end_date or not self.allow_future_exec_dates: +end_date = end_date or timezone.utcnow() +tis = tis.filter(TaskInstance.execution_date <= end_date) Review comment: Let's add a test for this piece of code. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates
kaxil commented on a change in pull request #7038: [AIRFLOW-4495] allow
externally triggered dags to run for future exec dates
URL: https://github.com/apache/airflow/pull/7038#discussion_r367110594
##
File path: tests/ti_deps/deps/test_runnable_exec_date_dep.py
##
@@ -39,11 +41,56 @@ def test_exec_date_after_end_date(self):
"""
If the dag's execution date is in the future this dep should fail
"""
-ti = self._get_task_instance(
-dag_end_date=datetime(2016, 1, 3),
-task_end_date=datetime(2016, 1, 3),
-execution_date=datetime(2016, 1, 2),
-)
+dag = DAG(
+'test_localtaskjob_heartbeat',
+start_date=datetime(2015, 1, 1),
+end_date=datetime(2016, 11, 5),
+schedule_interval=None)
+
+with dag:
+op1 = DummyOperator(task_id='op1')
+
+ti = TaskInstance(task=op1, execution_date=datetime(2016, 11, 2))
+self.assertFalse(RunnableExecDateDep().is_met(ti=ti))
+
+@conf_vars({
+('scheduler', 'allow_trigger_in_future'): 'True'
+})
+@freeze_time('2016-01-01')
+def test_exec_date_after_end_date_with_allow_config(self):
Review comment:
Maybe you could use `parameterized` as both the tests share almost the same
code.
You could parameterize `schedule_interval` and `execution_date` if you would
like.
tbh same execution date should work too
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates
kaxil commented on a change in pull request #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#discussion_r367110029 ## File path: tests/ti_deps/deps/test_runnable_exec_date_dep.py ## @@ -39,11 +41,56 @@ def test_exec_date_after_end_date(self): """ If the dag's execution date is in the future this dep should fail """ -ti = self._get_task_instance( -dag_end_date=datetime(2016, 1, 3), Review comment: Let's include dag_end_date too in the test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6556) Improving unclear and incomplete documentation
[ https://issues.apache.org/jira/browse/AIRFLOW-6556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016323#comment-17016323 ] Jarek Potiuk commented on AIRFLOW-6556: --- Thanks [~jward]! Some of these we already have on our list. It's good to have the list here. I will discuss it with others involved in the docs and we'll see what we can do. I hope, once we have some drafts we can count on you and your team to review and give feedback on what we propose. > Improving unclear and incomplete documentation > -- > > Key: AIRFLOW-6556 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6556 > Project: Apache Airflow > Issue Type: Improvement > Components: documentation >Affects Versions: master >Reporter: Jacob Ward >Assignee: Jarek Potiuk >Priority: Trivial > > To help improve documentation it was discussed in the mailing list that users > of Airflow should have somewhere to report missing, incomplete or unclear > documentation. Any users who find this should comment on this ticket. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure
codecov-io commented on issue #7172: [AIRFLOW-6564] Additional diagnostics information on CI check failure URL: https://github.com/apache/airflow/pull/7172#issuecomment-574856548 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=h1) Report > Merging [#7172](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8729ca525ccd2c976248b818440932242720a25b?src=pr=desc) will **decrease** coverage by `52.48%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#7172 +/- ## === - Coverage 85.24% 32.76% -52.49% === Files 710 722 +12 Lines 3948539530 +45 === - Hits3366012951-20709 - Misses 582526579+20754 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `92.63% <ø> (-2.11%)` | :arrow_down: | | [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `29.26% <50%> (-68.17%)` | :arrow_down: | | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `97.93% <50%> (-2.07%)` | :arrow_down: | | [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [...flow/contrib/example\_dags/example\_qubole\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9zZW5zb3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [...ample\_dags/example\_emr\_job\_flow\_automatic\_steps.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2Vtcl9qb2JfZmxvd19hdXRvbWF0aWNfc3RlcHMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [...contrib/example\_dags/example\_papermill\_operator.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3BhcGVybWlsbF9vcGVyYXRvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/hooks/webhdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy93ZWJoZGZzX2hvb2sucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/operators/gcs\_list\_operator.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfbGlzdF9vcGVyYXRvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/pig\_operator.py](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcGlnX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | ... and [591 more](https://codecov.io/gh/apache/airflow/pull/7172/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=footer). Last update [8729ca5...ead4811](https://codecov.io/gh/apache/airflow/pull/7172?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil merged pull request #7183: [AIRFLOW-XXXX] Sync Updating.md with v1.10.* branch
kaxil merged pull request #7183: [AIRFLOW-] Sync Updating.md with v1.10.* branch URL: https://github.com/apache/airflow/pull/7183 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #7162: [AIRFLOW-6557] Add test for newly added fields in BaseOperator
potiuk commented on issue #7162: [AIRFLOW-6557] Add test for newly added fields in BaseOperator URL: https://github.com/apache/airflow/pull/7162#issuecomment-574854810 >  > > :) The bot caught you @potiuk Good bot! Nice bot! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-6573) Bump `text-unidecode` version
[ https://issues.apache.org/jira/browse/AIRFLOW-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray R resolved AIRFLOW-6573. Resolution: Fixed > Bump `text-unidecode` version > - > > Key: AIRFLOW-6573 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6573 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: master >Reporter: Ray R >Priority: Trivial > > The `text-unidecode` package is now at version 1.3. This version includes > official Python 3.7 support within that package so we should bump to that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil commented on issue #7162: [AIRFLOW-6557] Add test for newly added fields in BaseOperator
kaxil commented on issue #7162: [AIRFLOW-6557] Add test for newly added fields in BaseOperator URL: https://github.com/apache/airflow/pull/7162#issuecomment-574852543  :) The bot caught you @potiuk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7162: [AIRFLOW-6557] Add test for newly added fields in BaseOperator
kaxil commented on a change in pull request #7162: [AIRFLOW-6557] Add test for
newly added fields in BaseOperator
URL: https://github.com/apache/airflow/pull/7162#discussion_r367102031
##
File path: tests/serialization/test_dag_serialization.py
##
@@ -543,6 +543,66 @@ def test_dag_serialized_fields_with_schema(self):
dag_params: set = set(dag_schema.keys()) - ignored_keys
self.assertEqual(set(DAG.get_serialized_fields()), dag_params)
+def test_no_new_fields_added_to_base_operator(self):
+"""
+This test verifies that there are no new fields added to BaseOperator.
And reminds that
+tests should be added for it.
+"""
+base_operator = BaseOperator(task_id="10")
+fields = base_operator.__dict__
+self.assertEqual({'_dag': None,
+ '_downstream_task_ids': set(),
+ '_inlets': [],
+ '_log': base_operator.log,
+ '_outlets': [],
+ '_upstream_task_ids': set(),
+ 'depends_on_past': False,
+ 'do_xcom_push': True,
+ 'email': None,
+ 'email_on_failure': True,
+ 'email_on_retry': True,
+ 'end_date': None,
+ 'execution_timeout': None,
+ 'executor_config': {},
+ 'inlets': [],
+ 'max_retry_delay': None,
+ 'on_execute_callback': None,
+ 'on_failure_callback': None,
+ 'on_retry_callback': None,
+ 'on_success_callback': None,
+ 'outlets': [],
+ 'owner': 'airflow',
+ 'params': {},
+ 'pool': 'default_pool',
+ 'priority_weight': 1,
+ 'queue': 'default',
+ 'resources': None,
+ 'retries': 0,
+ 'retry_delay': timedelta(0, 300),
+ 'retry_exponential_backoff': False,
+ 'run_as_user': None,
+ 'sla': None,
+ 'start_date': None,
+ 'subdag': None,
+ 'task_concurrency': None,
+ 'task_id': '10',
+ 'trigger_rule': 'all_success',
+ 'wait_for_downstream': False,
+ 'weight_rule': 'downstream'}, fields,
+ """
+!!!
+
+ ACTION NEEDED! PLEASE READ THIS CAREFULLY AND CORRECT TESTS CAREFULLY
+
+ Some fields were added to the BaseOperator! Please add them to the list above
and make sure that
+ you add support for DAG serialization - you should add the field to
+ `airflow/serialization/schema.json` and add it in
`serialized_simple_dag_ground_truth` above
Review comment:
I think this should good enough for the time being
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #7162: [AIRFLOW-6557] Add test for newly added fields in BaseOperator
potiuk commented on a change in pull request #7162: [AIRFLOW-6557] Add test for
newly added fields in BaseOperator
URL: https://github.com/apache/airflow/pull/7162#discussion_r367100569
##
File path: tests/serialization/test_dag_serialization.py
##
@@ -543,6 +543,66 @@ def test_dag_serialized_fields_with_schema(self):
dag_params: set = set(dag_schema.keys()) - ignored_keys
self.assertEqual(set(DAG.get_serialized_fields()), dag_params)
+def test_no_new_fields_added_to_base_operator(self):
+"""
+This test verifies that there are no new fields added to BaseOperator.
And reminds that
+tests should be added for it.
+"""
+base_operator = BaseOperator(task_id="10")
+fields = base_operator.__dict__
+self.assertEqual({'_dag': None,
+ '_downstream_task_ids': set(),
+ '_inlets': [],
+ '_log': base_operator.log,
+ '_outlets': [],
+ '_upstream_task_ids': set(),
+ 'depends_on_past': False,
+ 'do_xcom_push': True,
+ 'email': None,
+ 'email_on_failure': True,
+ 'email_on_retry': True,
+ 'end_date': None,
+ 'execution_timeout': None,
+ 'executor_config': {},
+ 'inlets': [],
+ 'max_retry_delay': None,
+ 'on_execute_callback': None,
+ 'on_failure_callback': None,
+ 'on_retry_callback': None,
+ 'on_success_callback': None,
+ 'outlets': [],
+ 'owner': 'airflow',
+ 'params': {},
+ 'pool': 'default_pool',
+ 'priority_weight': 1,
+ 'queue': 'default',
+ 'resources': None,
+ 'retries': 0,
+ 'retry_delay': timedelta(0, 300),
+ 'retry_exponential_backoff': False,
+ 'run_as_user': None,
+ 'sla': None,
+ 'start_date': None,
+ 'subdag': None,
+ 'task_concurrency': None,
+ 'task_id': '10',
+ 'trigger_rule': 'all_success',
+ 'wait_for_downstream': False,
+ 'weight_rule': 'downstream'}, fields,
+ """
+!!!
+
+ ACTION NEEDED! PLEASE READ THIS CAREFULLY AND CORRECT TESTS CAREFULLY
+
+ Some fields were added to the BaseOperator! Please add them to the list above
and make sure that
+ you add support for DAG serialization - you should add the field to
+ `airflow/serialization/schema.json` and add it in
`serialized_simple_dag_ground_truth` above
Review comment:
Similar rules as for protobuf. I will modify the message. We cannot test a
lot of that fully automatically yet but we can at least provide the right
message.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #7183: [AIRFLOW-XXXX] Sync Updating.md with v1.10.* branch
kaxil commented on a change in pull request #7183: [AIRFLOW-] Sync Updating.md with v1.10.* branch URL: https://github.com/apache/airflow/pull/7183#discussion_r367099554 ## File path: UPDATING.md ## @@ -1090,12 +1101,12 @@ Operators involved: * GCP Function Operators * GcfFunctionDeployOperator * GCP Cloud SQL Operators -* CloudSQLCreateInstanceOperator -* CloudSQLInstancePatchOperator -* CloudSQLDeleteInstanceOperator -* CloudSQLCreateInstanceDatabaseOperator -* CloudSQLPatchInstanceDatabaseOperator -* CloudSQLDeleteInstanceDatabaseOperator +* CloudSqlInstanceCreateOperator +* CloudSqlInstancePatchOperator Review comment: The Updating.md for the previous version should contain the original name that existed in that version. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil opened a new pull request #7183: [AIRFLOW-XXXX] Sync Updating.md with v1.10.* branch
kaxil opened a new pull request #7183: [AIRFLOW-] Sync Updating.md with v1.10.* branch URL: https://github.com/apache/airflow/pull/7183 Sync'd it with v1-10-* test & https://github.com/apache/airflow/commit/b79b642cb82e0b7d7eaceb8b32c86974d717ab0e commit --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[airflow-on-k8s-operator] branch master updated: Fixed Links to new Repo
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow-on-k8s-operator.git The following commit(s) were added to refs/heads/master by this push: new 4154808 Fixed Links to new Repo 4154808 is described below commit 4154808a8be8c895e3a58fe446fb6fd233b91e84 Author: Kaxil Naik AuthorDate: Wed Jan 15 20:42:17 2020 + Fixed Links to new Repo Also removed mention of Cloud Composer to stay vendor-neutral --- README.md | 16 +--- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 3b92eed..11c4d9f 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,4 @@ -[](https://goreportcard.com/report/github.com/GoogleCloudPlatform/airflow-operator) - -**This is not an officially supported Google product.** +[](https://goreportcard.com/report/github.com/apache/airflow-on-k8s-operator) ## Community @@ -22,9 +20,9 @@ The Airflow Operator is still under active development and has not been extensiv [One Click Deployment](https://console.cloud.google.com/marketplace/details/google/airflow-operator) from Google Cloud Marketplace to your [GKE cluster](https://cloud.google.com/kubernetes-engine/) -Get started quickly with the Airflow Operator using the [Quick Start Guide](https://github.com/GoogleCloudPlatform/airflow-operator/blob/master/docs/quickstart.md) +Get started quickly with the Airflow Operator using the [Quick Start Guide](docs/quickstart.md) -For more information check the [Design](https://github.com/GoogleCloudPlatform/airflow-operator/blob/master/docs/design.md) and detailed [User Guide](https://github.com/GoogleCloudPlatform/airflow-operator/blob/master/docs/userguide.md) +For more information check the [Design](docs/design.md) and detailed [User Guide](docs/userguide.md) ## Airflow Operator Overview Airflow Operator is a custom [Kubernetes operator](https://coreos.com/blog/introducing-operators.html) that makes it easy to deploy and manage [Apache Airflow](https://airflow.apache.org/) on Kubernetes. Apache Airflow is a platform to programmatically author, schedule and monitor workflows. Using the Airflow Operator, an Airflow cluster is split into 2 parts represented by the `AirflowBase` and `AirflowCluster` custom resources. @@ -35,15 +33,11 @@ The Airflow Operator performs these jobs: * Supports creation of Airflow schedulers with different Executors * Supports sharing of the `AirflowBase` across mulitple `AirflowClusters` -Checkout out the [Design](https://github.com/GoogleCloudPlatform/airflow-operator/blob/master/docs/design.md) +Checkout out the [Design](docs/design.md)  ## Development -Refer to the [Design](https://github.com/GoogleCloudPlatform/airflow-operator/blob/master/docs/design.md) and [Development Guide](https://github.com/GoogleCloudPlatform/airflow-operator/blob/master/docs/development.md). - -## Managed Airflow solution - -[Google Cloud Composer](https://cloud.google.com/composer/) is a fully managed workflow orchestration service targeting customers that need a workflow manager in the cloud. +Refer to the [Design](docs/design.md) and [Development Guide](docs/development.md).
[GitHub] [airflow] potiuk merged pull request #7182: [AIRFLOW-XXXX] Fix typo in UPDATING.md
potiuk merged pull request #7182: [AIRFLOW-] Fix typo in UPDATING.md URL: https://github.com/apache/airflow/pull/7182 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk merged pull request #7181: [AIRFLOW-XXXX] Fix typo in tests/conftest.py
potiuk merged pull request #7181: [AIRFLOW-] Fix typo in tests/conftest.py URL: https://github.com/apache/airflow/pull/7181 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6573) Bump `text-unidecode` version
[ https://issues.apache.org/jira/browse/AIRFLOW-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016307#comment-17016307 ] ASF subversion and git services commented on AIRFLOW-6573: -- Commit d6cecbfc3b393294c2a1deabf8a4ed5b77f96010 in airflow's branch refs/heads/master from Ray [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d6cecbf ] [AIRFLOW-6573] bump `text-unidecode` version for Python 3.7 (#7179) > Bump `text-unidecode` version > - > > Key: AIRFLOW-6573 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6573 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: master >Reporter: Ray R >Priority: Trivial > > The `text-unidecode` package is now at version 1.3. This version includes > official Python 3.7 support within that package so we should bump to that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6573) Bump `text-unidecode` version
[ https://issues.apache.org/jira/browse/AIRFLOW-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016305#comment-17016305 ] ASF GitHub Bot commented on AIRFLOW-6573: - potiuk commented on pull request #7179: [AIRFLOW-6573] bump `text-unidecode` version for Python 3.7 URL: https://github.com/apache/airflow/pull/7179 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Bump `text-unidecode` version > - > > Key: AIRFLOW-6573 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6573 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: master >Reporter: Ray R >Priority: Trivial > > The `text-unidecode` package is now at version 1.3. This version includes > official Python 3.7 support within that package so we should bump to that. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #7179: [AIRFLOW-6573] bump `text-unidecode` version for Python 3.7
potiuk merged pull request #7179: [AIRFLOW-6573] bump `text-unidecode` version for Python 3.7 URL: https://github.com/apache/airflow/pull/7179 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7181: [AIRFLOW-XXXX] Fix typo in tests/conftest.py
codecov-io edited a comment on issue #7181: [AIRFLOW-] Fix typo in tests/conftest.py URL: https://github.com/apache/airflow/pull/7181#issuecomment-574843211 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7181?src=pr=h1) Report > Merging [#7181](https://codecov.io/gh/apache/airflow/pull/7181?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8729ca525ccd2c976248b818440932242720a25b?src=pr=desc) will **decrease** coverage by `0.12%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7181?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7181 +/- ## == - Coverage 85.24% 85.12% -0.13% == Files 710 710 Lines 3948539485 == - Hits3366033612 -48 - Misses 5825 5873 +48 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7181?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfdG9fZ2NzLnB5) | `85.29% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.49%)` | :arrow_down: | | ... and [10 more](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7181?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7181?src=pr=footer). Last update [8729ca5...d21287b](https://codecov.io/gh/apache/airflow/pull/7181?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7181: [AIRFLOW-XXXX] Fix typo in tests/conftest.py
codecov-io commented on issue #7181: [AIRFLOW-] Fix typo in tests/conftest.py URL: https://github.com/apache/airflow/pull/7181#issuecomment-574843211 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7181?src=pr=h1) Report > Merging [#7181](https://codecov.io/gh/apache/airflow/pull/7181?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/8729ca525ccd2c976248b818440932242720a25b?src=pr=desc) will **decrease** coverage by `0.58%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7181?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7181 +/- ## == - Coverage 85.24% 84.66% -0.59% == Files 710 710 Lines 3948539485 == - Hits3366033430 -230 - Misses 5825 6055 +230 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7181?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/operators/postgres\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfdG9fZ2NzLnB5) | `52.94% <0%> (-32.36%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.31% <0%> (-20.49%)` | :arrow_down: | | ... and [10 more](https://codecov.io/gh/apache/airflow/pull/7181/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7181?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7181?src=pr=footer). Last update [8729ca5...d21287b](https://codecov.io/gh/apache/airflow/pull/7181?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
