[GitHub] [airflow] tooptoop4 commented on a change in pull request #6954: [AIRFLOW-4355] removed task should not lead to dagrun success
tooptoop4 commented on a change in pull request #6954: [AIRFLOW-4355] removed
task should not lead to dagrun success
URL: https://github.com/apache/airflow/pull/6954#discussion_r363625074
##
File path: airflow/models/dagrun.py
##
@@ -311,32 +313,51 @@ def update_state(self, session=None):
leaf_tis = [ti for ti in tis if ti.task_id in {t.task_id for t in
dag.leaves}]
-# if all roots finished and at least one failed, the run failed
-if not unfinished_tasks and any(
-leaf_ti.state in {State.FAILED, State.UPSTREAM_FAILED} for leaf_ti
in leaf_tis
-):
-self.log.info('Marking run %s failed', self)
-self.set_state(State.FAILED)
-dag.handle_callback(self, success=False, reason='task_failure',
-session=session)
-
-# if all leafs succeeded and no unfinished tasks, the run succeeded
-elif not unfinished_tasks and all(
-leaf_ti.state in {State.SUCCESS, State.SKIPPED} for leaf_ti in
leaf_tis
-):
-self.log.info('Marking run %s successful', self)
-self.set_state(State.SUCCESS)
-dag.handle_callback(self, success=True, reason='success',
session=session)
-
-# if *all tasks* are deadlocked, the run failed
-elif (unfinished_tasks and none_depends_on_past and
- none_task_concurrency and no_dependencies_met):
-self.log.info('Deadlock; marking run %s failed', self)
-self.set_state(State.FAILED)
-dag.handle_callback(self, success=False,
reason='all_tasks_deadlocked',
-session=session)
-
-# finally, if the roots aren't done, the dag is still running
+if len(tis) == len(dag.tasks):
Review comment:
see 'def verify_integrity(', the number of TaskInstance records in the db
can be changed frequently
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] tooptoop4 commented on a change in pull request #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates
tooptoop4 commented on a change in pull request #7038: [AIRFLOW-4495] allow
externally triggered dags to run for future exec dates
URL: https://github.com/apache/airflow/pull/7038#discussion_r363623656
##
File path: airflow/ti_deps/deps/runnable_exec_date_dep.py
##
@@ -30,11 +31,17 @@ class RunnableExecDateDep(BaseTIDep):
def _get_dep_statuses(self, ti, session, dep_context):
cur_date = timezone.utcnow()
+# don't consider runs that are executed in the future unless
+# specified by config and schedule_interval is None
if ti.execution_date > cur_date:
-yield self._failing_status(
-reason="Execution date {0} is in the future (the current "
- "date is {1}).".format(ti.execution_date.isoformat(),
- cur_date.isoformat()))
+if ti.task.dag.schedule_interval or not conf.getboolean(
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-6419) dag_processor_manager/webserver/scheduler logs should be created under date folder
[ https://issues.apache.org/jira/browse/AIRFLOW-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6419: -- Description: dag level logs are written under separate date folders. This is great because the old dates are not 'modified/accessed' so they can be easily purged by utilities like tmpwatch This JIRA is about making other logs (such as dag_processor_manager/webserver/scheduler.etc) go under separate date folders to allow easy purging. the log from redirecting 'airflow scheduler' to stdout grows over 100mb a day in my env was: scheduler logs are written under separate date folders. This is great because the old dates are not 'modified/accessed' so they can be easily purged by utilities like tmpwatch This JIRA is about making other logs (such as dag_processor_manager/webserver.etc) go under separate date folders to allow easy purging Summary: dag_processor_manager/webserver/scheduler logs should be created under date folder (was: dag_processor_manager/webserver logs should be created under date folder) > dag_processor_manager/webserver/scheduler logs should be created under date > folder > -- > > Key: AIRFLOW-6419 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6419 > Project: Apache Airflow > Issue Type: Improvement > Components: logging >Affects Versions: 1.10.7 >Reporter: t oo >Priority: Major > > dag level logs are written under separate date folders. This is great because > the old dates are not 'modified/accessed' so they can be easily purged by > utilities like tmpwatch > This JIRA is about making other logs (such as > dag_processor_manager/webserver/scheduler.etc) go under separate date folders > to allow easy purging. the log from redirecting 'airflow scheduler' to stdout > grows over 100mb a day in my env -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection
albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve
connection
URL: https://github.com/apache/airflow/pull/7075#issuecomment-571470394
@tooptoop4 I think you might want to try this sample DAG to reproduce the
issue.
a) create an environment var for spark connection.
```
export AIRFLOW_CONN_SPARK_DEFAULT='{"conn_type": spark, "host":,
"port":}'
```
b) create a DAG file to run
```python
from airflow import DAG
from airflow.contrib.operators.spark_submit_operator import
SparkSubmitOperator
from datetime import datetime, timedelta
import os
job_file = 'path/to/job/file'
default_args = {
'depends_on_past': False,
'start_date': ,
'retries': ,
'retry_delay':
}
dag = DAG('spark-submit-hook', default_args=default_args,
schedule_interval=)
avg = SparkSubmitOperator(task_id=, dag=dag,
application=job_file,
spark_binary='path/to/spark-submit')
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] tooptoop4 commented on a change in pull request #6342: [AIRFLOW-5662] fix incorrect naming for scheduler used slot metric
tooptoop4 commented on a change in pull request #6342: [AIRFLOW-5662] fix
incorrect naming for scheduler used slot metric
URL: https://github.com/apache/airflow/pull/6342#discussion_r363619884
##
File path: airflow/jobs/scheduler_job.py
##
@@ -1568,15 +1567,18 @@ def _execute_helper(self):
self.heartbeat()
last_self_heartbeat_time = timezone.utcnow()
+self._collect_metrics()
+
loop_end_time = time.time()
loop_duration = loop_end_time - loop_start_time
self.log.debug(
"Ran scheduling loop in %.2f seconds",
loop_duration)
+is_unit_test = conf.getboolean('core', 'unit_test_mode')
Review comment:
don't add this line, its already defined earlier
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/e863f8a08cd876d683a2a8552958f6663e1540a2?src=pr=desc) will **decrease** coverage by `0.21%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.01% 84.79% -0.22% == Files 680 680 Lines 3875338812 +59 == - Hits3294432909 -35 - Misses 5809 5903 +94 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.45% <0%> (-4.55%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <0%> (-0.08%)` | :arrow_down: | | [airflow/contrib/sensors/gcp\_transfer\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZ2NwX3RyYW5zZmVyX3NlbnNvci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [.../contrib/operators/gcs\_to\_gcs\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fZ2NzX3RyYW5zZmVyX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [e863f8a...f46cc63](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection
albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571468228 > > @tooptoop4 if we don't remove the spark check on line 177, how to use this hook to track driver status deployed on yarn, mesos, or k8s? Since I think `spark://` is only for standalone mode. > > Or this hook is created only for standalone mode? > > yes.there is no concept of async driver status poll for other modes , read https://spark.apache.org/docs/latest/running-on-yarn.html ! in other modes the submit to launch is synchronous . i think u can cancel this @albertusk95 I couldn't find any info stating that there's no async driver polling for YARN anyway from the provided link. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] tooptoop4 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection
tooptoop4 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571462616 > @tooptoop4 if we don't remove the spark check on line 177, how to use this hook to track driver status deployed on yarn, mesos, or k8s? Since I think `spark://` is only for standalone mode. > > Or this hook is created only for standalone mode? yes.there is no concept of async driver status poll for other modes , read https://spark.apache.org/docs/latest/running-on-yarn.html ! in other modes the submit to launch is synchronous . i think u can cancel this @albertusk95 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] tooptoop4 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection
tooptoop4 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571462616 > @tooptoop4 if we don't remove the spark check on line 177, how to use this hook to track driver status deployed on yarn, mesos, or k8s? Since I think `spark://` is only for standalone mode. > > Or this hook is created only for standalone mode? yes.there is no concept of async driver status poll for other modes , read https://spark.apache.org/docs/latest/running-on-yarn.html ! in other modes the submit to launch is synchronous . i think u can cancel this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection
albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571462963 > @tooptoop4 the tests pass only for the case when the connection information (host, port, conn_type, etc.) are stored in **database**. I tried this hook by storing the connection info as an **environment variable**. > > This failed because the URI parser returned irrelevant results for all types of cluster mode deployment. For instance, `URI=spark://host:port` will be parsed into `host:port` without the `spark://`. Obviously it returns this exception: > > ``` > airflow.exceptions.AirflowException: Cannot execute: [path/to/spark-submit, '--master', host:port, job_file.py] > ``` @tooptoop4 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] tooptoop4 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection
tooptoop4 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571462616 spark hook works for me and tests pass without this pr. can u send the add connection cli command u create to hit the issue? > @tooptoop4 if we don't remove the spark check on line 177, how to use this hook to track driver status deployed on yarn, mesos, or k8s? Since I think `spark://` is only for standalone mode. > > Or this hook is created only for standalone mode? yes.there is no concept of async driver status poll for other modes , read https://spark.apache.org/docs/latest/running-on-yarn.html ! in other modes the submit to launch is synchronous . i think u can cancel this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6497) Scheduler creates DagBag in the same process with outdated info
[
https://issues.apache.org/jira/browse/AIRFLOW-6497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009401#comment-17009401
]
Qian Yu commented on AIRFLOW-6497:
--
[~bolke] [~potiuk] [~kamil.bregula] any idea / suggestions?
> Scheduler creates DagBag in the same process with outdated info
> ---
>
> Key: AIRFLOW-6497
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6497
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
>Affects Versions: 1.10.7
>Reporter: Qian Yu
>Priority: Major
>
> The following code in scheduler_job.py seems to be called in the same process
> as the scheduler. It creates a DagBag. But since scheduler is a long running
> process, it does not pick up the latest changes made to DAGs. For example,
> changes to retries count, on_failure_callback, newly added tasks, etc are not
> reflected.
>
> {code:python}
> if ti.try_number == try_number and ti.state == State.QUEUED:
> msg = ("Executor reports task instance {} finished ({}) "
>"although the task says its {}. Was the task "
>"killed externally?".format(ti, state, ti.state))
> Stats.incr('scheduler.tasks.killed_externally')
> self.log.error(msg)
> try:
> simple_dag = simple_dag_bag.get_dag(dag_id)
> dagbag = models.DagBag(simple_dag.full_filepath)
> dag = dagbag.get_dag(dag_id)
> ti.task = dag.get_task(task_id)
> ti.handle_failure(msg)
> except Exception:
> self.log.error("Cannot load the dag bag to handle
> failure for %s"
>". Setting task to FAILED without
> callbacks or "
>"retries. Do you have enough
> resources?", ti)
> ti.state = State.FAILED
> session.merge(ti)
> session.commit()
> {code}
> This causes errors such as AttributeError due to stale code being hit. E.g.
> when someone added a .join attribute to CustomOperator without bouncing the
> scheduler, this is what he would get after a CeleryWorker timeout error
> causes this line to be hit:
> {code}
> [2020-01-05 22:25:45,951] {dagbag.py:207} ERROR - Failed to import:
> /dags/dag1.py
> Traceback (most recent call last):
> File "/lib/python3.6/site-packages/airflow/models/dagbag.py", line 204, in
> process_file
> m = imp.load_source(mod_name, filepath)
> File "/usr/lib/python3.6/imp.py", line 172, in load_source
> module = _load(spec)
> File "", line 684, in _load
> File "", line 665, in _load_unlocked
> File "", line 678, in exec_module
> File "", line 219, in _call_with_frames_removed
> File "/dags/dag1.py", line 280, in
> task1 >> task2.join
> AttributeError: 'CustomOperator' object has no attribute 'join'
> [2020-01-05 22:25:45,951] {scheduler_job.py:1314} ERROR - Cannot load the dag
> bag to handle failure for [queued]>. Setting task to FAILED without callbacks or retries. Do you have
> enough resou
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (AIRFLOW-6497) Scheduler creates DagBag in the same process with outdated info
Qian Yu created AIRFLOW-6497:
Summary: Scheduler creates DagBag in the same process with
outdated info
Key: AIRFLOW-6497
URL: https://issues.apache.org/jira/browse/AIRFLOW-6497
Project: Apache Airflow
Issue Type: Bug
Components: scheduler
Affects Versions: 1.10.7
Reporter: Qian Yu
The following code in scheduler_job.py seems to be called in the same process
as the scheduler. It creates a DagBag. But since scheduler is a long running
process, it does not pick up the latest changes made to DAGs. For example,
changes to retries count, on_failure_callback, newly added tasks, etc are not
reflected.
{code:python}
if ti.try_number == try_number and ti.state == State.QUEUED:
msg = ("Executor reports task instance {} finished ({}) "
"although the task says its {}. Was the task "
"killed externally?".format(ti, state, ti.state))
Stats.incr('scheduler.tasks.killed_externally')
self.log.error(msg)
try:
simple_dag = simple_dag_bag.get_dag(dag_id)
dagbag = models.DagBag(simple_dag.full_filepath)
dag = dagbag.get_dag(dag_id)
ti.task = dag.get_task(task_id)
ti.handle_failure(msg)
except Exception:
self.log.error("Cannot load the dag bag to handle
failure for %s"
". Setting task to FAILED without
callbacks or "
"retries. Do you have enough
resources?", ti)
ti.state = State.FAILED
session.merge(ti)
session.commit()
{code}
This causes errors such as AttributeError due to stale code being hit. E.g.
when someone added a .join attribute to CustomOperator without bouncing the
scheduler, this is what he would get after a CeleryWorker timeout error causes
this line to be hit:
{code}
[2020-01-05 22:25:45,951] {dagbag.py:207} ERROR - Failed to import:
/dags/dag1.py
Traceback (most recent call last):
File "/lib/python3.6/site-packages/airflow/models/dagbag.py", line 204, in
process_file
m = imp.load_source(mod_name, filepath)
File "/usr/lib/python3.6/imp.py", line 172, in load_source
module = _load(spec)
File "", line 684, in _load
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/dags/dag1.py", line 280, in
task1 >> task2.join
AttributeError: 'CustomOperator' object has no attribute 'join'
[2020-01-05 22:25:45,951] {scheduler_job.py:1314} ERROR - Cannot load the dag
bag to handle failure for . Setting task to FAILED without callbacks or retries. Do you have
enough resou
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6487) Romove unnecessary for loop
[ https://issues.apache.org/jira/browse/AIRFLOW-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenqiang Li updated AIRFLOW-6487: - Description: Romove unnecessary for loop: In DagFileProcessorManager, dag files are processed periodically. Processor manager will put dag into queue except file_paths_in_progress, file_paths_recently_processed and files_paths_at_run_limit. While min_file_process_interval = 0 in config by default, there is unnecessary for-loop when get file_paths_recently_processed. was: Romove unnecessary for loop: In DagFileProcessorManager, dag files are processed periodically. Processor manager will put dag into queue except file_paths_in_progress, file_paths_recently_processed and files_paths_at_run_limit. While min_file_process_interval = 0 and num_runs = -1 in config by default, there is unnecessary for-loop when get file_paths_recently_processed and files_paths_at_run_limit. > Romove unnecessary for loop > --- > > Key: AIRFLOW-6487 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6487 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: master >Reporter: Wenqiang Li >Assignee: Wenqiang Li >Priority: Major > > Romove unnecessary for loop: > In DagFileProcessorManager, dag files are processed periodically. Processor > manager will put dag into queue except file_paths_in_progress, > file_paths_recently_processed and files_paths_at_run_limit. While > min_file_process_interval = 0 in config by default, there is unnecessary > for-loop when get file_paths_recently_processed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-6405) Bigquery Update Table Properties Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jithin Sukumar reassigned AIRFLOW-6405: --- Assignee: Jithin Sukumar > Bigquery Update Table Properties Operator > - > > Key: AIRFLOW-6405 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6405 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp, operators >Affects Versions: 1.10.7 >Reporter: Jithin Sukumar >Assignee: Jithin Sukumar >Priority: Minor > > Currently, Airflow doesn't seem to support BigQuery update table operations > [1] (specifically to update the properties of a table). Is this already under > development? > (The use case is conditionally updating the `expiration time` of BQ tables.) > > References: > [1]: [https://cloud.google.com/bigquery/docs/managing-tables] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] Gabriel39 removed a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
Gabriel39 removed a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571433348 Why it failed? Can someone help me? thx This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038812 +2 == - Hits3301932910 -109 - Misses 5791 5902 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `88% <0%> (+2.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [748e3c3...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038812 +2 == - Hits3301932910 -109 - Misses 5791 5902 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `88% <0%> (+2.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [748e3c3...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/e863f8a08cd876d683a2a8552958f6663e1540a2?src=pr=desc) will **decrease** coverage by `0.21%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.01% 84.79% -0.22% == Files 680 680 Lines 3875338812 +59 == - Hits3294432910 -34 - Misses 5809 5902 +93 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.45% <0%> (-4.55%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <0%> (-0.08%)` | :arrow_down: | | [airflow/contrib/sensors/gcp\_transfer\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZ2NwX3RyYW5zZmVyX3NlbnNvci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [.../contrib/operators/gcs\_to\_gcs\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fZ2NzX3RyYW5zZmVyX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [e863f8a...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038812 +2 == - Hits3301932910 -109 - Misses 5791 5902 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `88% <0%> (+2.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [748e3c3...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038812 +2 == - Hits3301932910 -109 - Misses 5791 5902 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `88% <0%> (+2.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [748e3c3...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/e863f8a08cd876d683a2a8552958f6663e1540a2?src=pr=desc) will **decrease** coverage by `0.21%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.01% 84.79% -0.22% == Files 680 680 Lines 3875338812 +59 == - Hits3294432910 -34 - Misses 5809 5902 +93 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.45% <0%> (-4.55%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <0%> (-0.08%)` | :arrow_down: | | [airflow/contrib/sensors/gcp\_transfer\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZ2NwX3RyYW5zZmVyX3NlbnNvci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [.../contrib/operators/gcs\_to\_gcs\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fZ2NzX3RyYW5zZmVyX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [e863f8a...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038812 +2 == - Hits3301932910 -109 - Misses 5791 5902 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `88% <0%> (+2.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [748e3c3...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038812 +2 == - Hits3301932910 -109 - Misses 5791 5902 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `88% <0%> (+2.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [748e3c3...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/e863f8a08cd876d683a2a8552958f6663e1540a2?src=pr=desc) will **decrease** coverage by `0.21%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.01% 84.79% -0.22% == Files 680 680 Lines 3875338812 +59 == - Hits3294432910 -34 - Misses 5809 5902 +93 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.45% <0%> (-4.55%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <0%> (-0.08%)` | :arrow_down: | | [airflow/contrib/sensors/gcp\_transfer\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZ2NwX3RyYW5zZmVyX3NlbnNvci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [.../contrib/operators/gcs\_to\_gcs\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fZ2NzX3RyYW5zZmVyX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [e863f8a...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038812 +2 == - Hits3301932910 -109 - Misses 5791 5902 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `88% <0%> (+2.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [748e3c3...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038812 +2 == - Hits3301932910 -109 - Misses 5791 5902 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `88% <0%> (+2.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [748e3c3...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038812 +2 == - Hits3301932910 -109 - Misses 5791 5902 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `88% <0%> (+2.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [748e3c3...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/e863f8a08cd876d683a2a8552958f6663e1540a2?src=pr=desc) will **decrease** coverage by `0.21%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.01% 84.79% -0.22% == Files 680 680 Lines 3875338812 +59 == - Hits3294432910 -34 - Misses 5809 5902 +93 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.45% <0%> (-4.55%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <0%> (-0.08%)` | :arrow_down: | | [airflow/contrib/sensors/gcp\_transfer\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZ2NwX3RyYW5zZmVyX3NlbnNvci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [.../contrib/operators/gcs\_to\_gcs\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fZ2NzX3RyYW5zZmVyX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [e863f8a...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038812 +2 == - Hits3301932910 -109 - Misses 5791 5902 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `88% <0%> (+2.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [748e3c3...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/e863f8a08cd876d683a2a8552958f6663e1540a2?src=pr=desc) will **decrease** coverage by `0.21%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.01% 84.79% -0.22% == Files 680 680 Lines 3875338812 +59 == - Hits3294432910 -34 - Misses 5809 5902 +93 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.45% <0%> (-4.55%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <0%> (-0.08%)` | :arrow_down: | | [airflow/contrib/sensors/gcp\_transfer\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZ2NwX3RyYW5zZmVyX3NlbnNvci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [.../contrib/operators/gcs\_to\_gcs\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fZ2NzX3RyYW5zZmVyX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [e863f8a...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/e863f8a08cd876d683a2a8552958f6663e1540a2?src=pr=desc) will **decrease** coverage by `0.21%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.01% 84.79% -0.22% == Files 680 680 Lines 3875338812 +59 == - Hits3294432910 -34 - Misses 5809 5902 +93 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.45% <0%> (-4.55%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <0%> (-0.08%)` | :arrow_down: | | [airflow/contrib/sensors/gcp\_transfer\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZ2NwX3RyYW5zZmVyX3NlbnNvci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [.../contrib/operators/gcs\_to\_gcs\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fZ2NzX3RyYW5zZmVyX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [e863f8a...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io commented on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038812 +2 == - Hits3301932910 -109 - Misses 5791 5902 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `88% <0%> (+2.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [748e3c3...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038812 +2 == - Hits3301932910 -109 - Misses 5791 5902 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `88% <0%> (+2.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [748e3c3...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038812 +2 == - Hits3301932910 -109 - Misses 5791 5902 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `88% <0%> (+2.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [748e3c3...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038812 +2 == - Hits3301932910 -109 - Misses 5791 5902 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `88% <0%> (+2.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [748e3c3...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
codecov-io edited a comment on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571443074 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=h1) Report > Merging [#7079](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7079 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038812 +2 == - Hits3301932910 -109 - Misses 5791 5902 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.66% <100%> (-0.34%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `91.59% <0%> (-0.29%)` | :arrow_down: | | [airflow/utils/log/file\_processor\_handler.py](https://codecov.io/gh/apache/airflow/pull/7079/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV9wcm9jZXNzb3JfaGFuZGxlci5weQ==) | `88% <0%> (+2.66%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=footer). Last update [748e3c3...199bcc8](https://codecov.io/gh/apache/airflow/pull/7079?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil opened a new pull request #7092: [AIRFLOW-6347] BugFix: Can't get task logs when serialization is enabled
kaxil opened a new pull request #7092: [AIRFLOW-6347] BugFix: Can't get task
logs when serialization is enabled
URL: https://github.com/apache/airflow/pull/7092
Quoting from Jira:
**Problem**:
When I set next config options in `airflow.cfg`:
```ini
[core]
store_serialized_dags = True
min_serialized_dag_update_interval = 3
```
View with task logs shows infinity Js spinner, while in webserver log I see
next error:
```
[2019-12-25 14:48:57,640] {{app.py:1891}} ERROR - Exception on
/get_logs_with_metadata [GET]
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in
wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in
full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in
handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39,
in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in
full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in
dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/www_rbac/decorators.py",
line 121, in wrapper
return f(self, *args, **kwargs)
File
"/usr/local/lib/python3.7/site-packages/flask_appbuilder/security/decorators.py",
line 101, in wraps
return f(self, *args, **kwargs)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/www_rbac/decorators.py",
line 56, in wrapper
return f(*args, **kwargs)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py",
line 74, in wrapper
return func(*args, **kwargs)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/www_rbac/views.py",
line 637, in get_logs_with_metadata
logs, metadata = _get_logs_with_metadata(try_number, metadata)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/www_rbac/views.py",
line 628, in _get_logs_with_metadata
logs, metadatas = handler.read(ti, try_number, metadata=metadata)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/utils/log/file_task_handler.py",
line 169, in read
log, metadata = self._read(task_instance, try_number_element, metadata)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/utils/log/file_task_handler.py",
line 98, in _read
log_relative_path = self._render_filename(ti, try_number)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/utils/log/file_task_handler.py",
line 75, in _render_filename
jinja_context = ti.get_template_context()
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py",
line 74, in wrapper
return func(*args, **kwargs)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py",
line 1149, in get_template_context
if 'tables' in task.params:
TypeError: argument of type 'NoneType' is not iterable
```
**Solution:**
This is because we use `self.params = params or {}` and where default value
of `params` is `None` in DAG and Operators and Dag S10n ignore all the None
field.
While de-serializing `params` was being set to None because of the following
line:
```python
keys_to_set_none = dag.get_serialized_fields() - encoded_dag.keys() -
cls._CONSTRUCTOR_PARAMS.keys()
for k in keys_to_set_none:
setattr(dag, k, None)
```
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License
[jira] [Commented] (AIRFLOW-6347) Can't get task logs in webUI when DAG serialization is enabled
[
https://issues.apache.org/jira/browse/AIRFLOW-6347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009374#comment-17009374
]
ASF GitHub Bot commented on AIRFLOW-6347:
-
kaxil commented on pull request #7092: [AIRFLOW-6347] BugFix: Can't get task
logs when serialization is enabled
URL: https://github.com/apache/airflow/pull/7092
Quoting from Jira:
**Problem**:
When I set next config options in `airflow.cfg`:
```ini
[core]
store_serialized_dags = True
min_serialized_dag_update_interval = 3
```
View with task logs shows infinity Js spinner, while in webserver log I see
next error:
```
[2019-12-25 14:48:57,640] {{app.py:1891}} ERROR - Exception on
/get_logs_with_metadata [GET]
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in
wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in
full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in
handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39,
in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in
full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in
dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/www_rbac/decorators.py",
line 121, in wrapper
return f(self, *args, **kwargs)
File
"/usr/local/lib/python3.7/site-packages/flask_appbuilder/security/decorators.py",
line 101, in wraps
return f(self, *args, **kwargs)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/www_rbac/decorators.py",
line 56, in wrapper
return f(*args, **kwargs)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py",
line 74, in wrapper
return func(*args, **kwargs)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/www_rbac/views.py",
line 637, in get_logs_with_metadata
logs, metadata = _get_logs_with_metadata(try_number, metadata)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/www_rbac/views.py",
line 628, in _get_logs_with_metadata
logs, metadatas = handler.read(ti, try_number, metadata=metadata)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/utils/log/file_task_handler.py",
line 169, in read
log, metadata = self._read(task_instance, try_number_element, metadata)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/utils/log/file_task_handler.py",
line 98, in _read
log_relative_path = self._render_filename(ti, try_number)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/utils/log/file_task_handler.py",
line 75, in _render_filename
jinja_context = ti.get_template_context()
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py",
line 74, in wrapper
return func(*args, **kwargs)
File
"/usr/local/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py",
line 1149, in get_template_context
if 'tables' in task.params:
TypeError: argument of type 'NoneType' is not iterable
```
**Solution:**
This is because we use `self.params = params or {}` and where default value
of `params` is `None` in DAG and Operators and Dag S10n ignore all the None
field.
While de-serializing `params` was being set to None because of the following
line:
```python
keys_to_set_none = dag.get_serialized_fields() - encoded_dag.keys() -
cls._CONSTRUCTOR_PARAMS.keys()
for k in keys_to_set_none:
setattr(dag, k, None)
```
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement
[jira] [Commented] (AIRFLOW-6489) Add BATS unit testing for bash scripts
[ https://issues.apache.org/jira/browse/AIRFLOW-6489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009372#comment-17009372 ] ASF GitHub Bot commented on AIRFLOW-6489: - potiuk commented on pull request #7091: [AIRFLOW-6489] Separate integrations [WIP] URL: https://github.com/apache/airflow/pull/7091 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add BATS unit testing for bash scripts > -- > > Key: AIRFLOW-6489 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6489 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0, 1.10.7 >Reporter: Jarek Potiuk >Priority: Major > > We have far too much bash code around that is not automatically tested. > This is the first step to change it (simplifications and more tests are > coming soon). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk opened a new pull request #7091: [AIRFLOW-6489] Separate integrations [WIP]
potiuk opened a new pull request #7091: [AIRFLOW-6489] Separate integrations [WIP] URL: https://github.com/apache/airflow/pull/7091 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Gabriel39 commented on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop
Gabriel39 commented on issue #7079: [AIRFLOW-6487] Romove unnecessary for-loop URL: https://github.com/apache/airflow/pull/7079#issuecomment-571433348 Why it failed? Can someone help me? thx This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection
albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571424197 @tooptoop4 if we don't remove the spark check on line 177, how to use this hook to track driver status deployed on yarn, mesos, or k8s? Since I think `spark://` is only for standalone mode. Or this hook is created only for standalone mode? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection
albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571429121 @tooptoop4 the tests pass only for the case when the connection information (host, port, conn_type, etc.) are stored in **database**. I tried this hook by storing the connection info as an **environment variable**. This failed because the URI parser returned irrelevant results for all types of cluster mode deployment. For instance, `URI=spark://host:port` will be parsed into `host:port` without the `spark://`. Obviously it returns this exception: ``` airflow.exceptions.AirflowException: Cannot execute: [path/to/spark-submit, '--master', host:port, job_file.py] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6496) Choosing integration at Breeze start
Jarek Potiuk created AIRFLOW-6496: - Summary: Choosing integration at Breeze start Key: AIRFLOW-6496 URL: https://issues.apache.org/jira/browse/AIRFLOW-6496 Project: Apache Airflow Issue Type: Improvement Components: ci Affects Versions: 1.10.7, 2.0.0 Reporter: Jarek Potiuk You should be able to choose which integration you want to start when you run Breeze. At later stage we will ad markers to tests to run them only when given environment is available.. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection
albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571429121 @tooptoop4 the tests pass only for the case when the connection information (host, port, conn_type, etc.) are stored in **database**. I tried this hook by storing the connection info as an **environment variable**. This failed because the URI parser returned irrelevant results for all types of cluster mode deployment. For instance, `URI=spark://master-address:port` will be parsed into `master-address:port` without the `spark://`. Obviously it returns this exception: ``` airflow.exceptions.AirflowException: Cannot execute: [path/to/spark-submit, '--master', host:port, job_file.py] ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection
albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571429121 @tooptoop4 the tests pass only for the case when the connection information (host, port, conn_type, etc.) are stored in **database**. I tried this hook by storing the connection info as an **environment variable**. This failed because the URI parser returned irrelevant results for all types of cluster mode deployment (yarn, standalone, mesos, k8s) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection
albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571429121 @tooptoop4 the tests pass only for the case when the connection information (host, port, conn_type, etc.) are stored in database. I tried this hook by storing the connection info as an environment variable. This failed because the URI parser returned irrelevant results for all types of cluster mode deployment (yarn, standalone, mesos, k8s) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging [#7090](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/64204d3d605b6bd37b09a7347da77acc31df2269?src=pr=desc) will **decrease** coverage by `0.19%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) ```diff @@Coverage Diff@@ ## master#7090 +/- ## = - Coverage 84.98% 84.79% -0.2% = Files 680 680 Lines 3875338811 +58 = - Hits3293532910 -25 - Misses 5818 5901 +83 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/task/task\_runner/standard\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL3N0YW5kYXJkX3Rhc2tfcnVubmVyLnB5) | `67.21% <50%> (+0.54%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.45% <0%> (-4.55%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <0%> (-0.08%)` | :arrow_down: | | [airflow/contrib/sensors/gcp\_transfer\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZ2NwX3RyYW5zZmVyX3NlbnNvci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [.../contrib/operators/gcs\_to\_gcs\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fZ2NzX3RyYW5zZmVyX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | ... and [9 more](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=footer). Last update [64204d3...6e56595](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging [#7090](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/64204d3d605b6bd37b09a7347da77acc31df2269?src=pr=desc) will **decrease** coverage by `0.19%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) ```diff @@Coverage Diff@@ ## master#7090 +/- ## = - Coverage 84.98% 84.79% -0.2% = Files 680 680 Lines 3875338811 +58 = - Hits3293532910 -25 - Misses 5818 5901 +83 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/task/task\_runner/standard\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL3N0YW5kYXJkX3Rhc2tfcnVubmVyLnB5) | `67.21% <50%> (+0.54%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.45% <0%> (-4.55%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <0%> (-0.08%)` | :arrow_down: | | [airflow/contrib/sensors/gcp\_transfer\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZ2NwX3RyYW5zZmVyX3NlbnNvci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [.../contrib/operators/gcs\_to\_gcs\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fZ2NzX3RyYW5zZmVyX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | ... and [9 more](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=footer). Last update [64204d3...6e56595](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging [#7090](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7090 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038811 +1 == - Hits3301932910 -109 - Misses 5791 5901 +110 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/task/task\_runner/standard\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL3N0YW5kYXJkX3Rhc2tfcnVubmVyLnB5) | `67.21% <50%> (+0.54%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=footer). Last update [748e3c3...6e56595](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging [#7090](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/64204d3d605b6bd37b09a7347da77acc31df2269?src=pr=desc) will **decrease** coverage by `0.19%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) ```diff @@Coverage Diff@@ ## master#7090 +/- ## = - Coverage 84.98% 84.79% -0.2% = Files 680 680 Lines 3875338811 +58 = - Hits3293532910 -25 - Misses 5818 5901 +83 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/task/task\_runner/standard\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL3N0YW5kYXJkX3Rhc2tfcnVubmVyLnB5) | `67.21% <50%> (+0.54%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.45% <0%> (-4.55%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <0%> (-0.08%)` | :arrow_down: | | [airflow/contrib/sensors/gcp\_transfer\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZ2NwX3RyYW5zZmVyX3NlbnNvci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [.../contrib/operators/gcs\_to\_gcs\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fZ2NzX3RyYW5zZmVyX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | ... and [9 more](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=footer). Last update [64204d3...6e56595](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging [#7090](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7090 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038811 +1 == - Hits3301932910 -109 - Misses 5791 5901 +110 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/task/task\_runner/standard\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL3N0YW5kYXJkX3Rhc2tfcnVubmVyLnB5) | `67.21% <50%> (+0.54%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=footer). Last update [748e3c3...6e56595](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging [#7090](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/64204d3d605b6bd37b09a7347da77acc31df2269?src=pr=desc) will **decrease** coverage by `0.19%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) ```diff @@Coverage Diff@@ ## master#7090 +/- ## = - Coverage 84.98% 84.79% -0.2% = Files 680 680 Lines 3875338811 +58 = - Hits3293532910 -25 - Misses 5818 5901 +83 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/task/task\_runner/standard\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL3N0YW5kYXJkX3Rhc2tfcnVubmVyLnB5) | `67.21% <50%> (+0.54%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.45% <0%> (-4.55%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <0%> (-0.08%)` | :arrow_down: | | [airflow/contrib/sensors/gcp\_transfer\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZ2NwX3RyYW5zZmVyX3NlbnNvci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [.../contrib/operators/gcs\_to\_gcs\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fZ2NzX3RyYW5zZmVyX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | ... and [9 more](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=footer). Last update [64204d3...6e56595](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging [#7090](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7090 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038811 +1 == - Hits3301932910 -109 - Misses 5791 5901 +110 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/task/task\_runner/standard\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL3N0YW5kYXJkX3Rhc2tfcnVubmVyLnB5) | `67.21% <50%> (+0.54%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=footer). Last update [748e3c3...6e56595](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging [#7090](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7090 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038811 +1 == - Hits3301932910 -109 - Misses 5791 5901 +110 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/task/task\_runner/standard\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL3N0YW5kYXJkX3Rhc2tfcnVubmVyLnB5) | `67.21% <50%> (+0.54%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=footer). Last update [748e3c3...6e56595](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner
codecov-io edited a comment on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging [#7090](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/64204d3d605b6bd37b09a7347da77acc31df2269?src=pr=desc) will **decrease** coverage by `0.19%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) ```diff @@Coverage Diff@@ ## master#7090 +/- ## = - Coverage 84.98% 84.79% -0.2% = Files 680 680 Lines 3875338811 +58 = - Hits3293532910 -25 - Misses 5818 5901 +83 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/task/task\_runner/standard\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL3N0YW5kYXJkX3Rhc2tfcnVubmVyLnB5) | `67.21% <50%> (+0.54%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.45% <0%> (-4.55%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <0%> (-0.08%)` | :arrow_down: | | [airflow/contrib/sensors/gcp\_transfer\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZ2NwX3RyYW5zZmVyX3NlbnNvci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [.../contrib/operators/gcs\_to\_gcs\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3NfdG9fZ2NzX3RyYW5zZmVyX29wZXJhdG9yLnB5) | `0% <0%> (ø)` | :arrow_up: | | ... and [9 more](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=footer). Last update [64204d3...6e56595](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner
codecov-io commented on issue #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090#issuecomment-571428318 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=h1) Report > Merging [#7090](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/748e3c3519b6cfeb82929efa6cc2d11ca0154b49?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7090 +/- ## == - Coverage 85.07% 84.79% -0.29% == Files 680 680 Lines 3881038811 +1 == - Hits3301932910 -109 - Misses 5791 5901 +110 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/task/task\_runner/standard\_task\_runner.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL3N0YW5kYXJkX3Rhc2tfcnVubmVyLnB5) | `67.21% <50%> (+0.54%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7090/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=footer). Last update [748e3c3...6e56595](https://codecov.io/gh/apache/airflow/pull/7090?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection
albertusk95 edited a comment on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571424197 @tooptoop4 if we don't remove the spark check on line 177, how to use this hook to track driver status deployed on yarn, mesos, or k8s? Since I think `spark://` is only for standalone mode. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection
albertusk95 commented on issue #7075: [AIRFLOW-6212] SparkSubmitHook resolve connection URL: https://github.com/apache/airflow/pull/7075#issuecomment-571424197 @tooptoop4 if we don't remove the spark check on line 177, how to use this hook to track driver status deployed on yarn, mesos, or k8s? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6495) Load DAG only once when running a task using StandardTaskRunner
[ https://issues.apache.org/jira/browse/AIRFLOW-6495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009323#comment-17009323 ] ASF GitHub Bot commented on AIRFLOW-6495: - mik-laj commented on pull request #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090 If the process is created using a fork, it will share all loaded modules that were available before forking. This allows us to avoid loading DAG again. Sometimes the DAG loading process is costly because it do database queries or other costly operation, so we should limit it if possible. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) - [X] Description above provides context of the change - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [X] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Load DAG only once when running a task using StandardTaskRunner > --- > > Key: AIRFLOW-6495 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6495 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 1.10.7 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner
mik-laj opened a new pull request #7090: [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner URL: https://github.com/apache/airflow/pull/7090 If the process is created using a fork, it will share all loaded modules that were available before forking. This allows us to avoid loading DAG again. Sometimes the DAG loading process is costly because it do database queries or other costly operation, so we should limit it if possible. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) - [X] Description above provides context of the change - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [X] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6495) Load DAG only once when running a task using StandardTaskRunner
Kamil Bregula created AIRFLOW-6495: -- Summary: Load DAG only once when running a task using StandardTaskRunner Key: AIRFLOW-6495 URL: https://issues.apache.org/jira/browse/AIRFLOW-6495 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: 1.10.7 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6381) DAG name changes style of some columns but not all
[ https://issues.apache.org/jira/browse/AIRFLOW-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009315#comment-17009315 ] ASF GitHub Bot commented on AIRFLOW-6381: - kaxil commented on pull request #6985: [AIRFLOW-6381] Remove styling based on DAG id from DAGs page URL: https://github.com/apache/airflow/pull/6985 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > DAG name changes style of some columns but not all > -- > > Key: AIRFLOW-6381 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6381 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.2 >Reporter: Jay K >Assignee: Jay K >Priority: Trivial > Fix For: 1.10.8 > > Attachments: image-2019-12-27-22-38-55-209.png > > > !image-2019-12-27-22-38-55-209.png|width=782,height=114! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6381) DAG name changes style of some columns but not all
[ https://issues.apache.org/jira/browse/AIRFLOW-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-6381: Fix Version/s: (was: 1.10.8) 2.0.0 > DAG name changes style of some columns but not all > -- > > Key: AIRFLOW-6381 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6381 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.2 >Reporter: Jay K >Assignee: Jay K >Priority: Trivial > Fix For: 2.0.0 > > Attachments: image-2019-12-27-22-38-55-209.png > > > !image-2019-12-27-22-38-55-209.png|width=782,height=114! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6381) DAG name changes style of some columns but not all
[ https://issues.apache.org/jira/browse/AIRFLOW-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-6381: Fix Version/s: (was: 2.0.0) 1.10.8 > DAG name changes style of some columns but not all > -- > > Key: AIRFLOW-6381 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6381 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.2 >Reporter: Jay K >Assignee: Jay K >Priority: Trivial > Fix For: 1.10.8 > > Attachments: image-2019-12-27-22-38-55-209.png > > > !image-2019-12-27-22-38-55-209.png|width=782,height=114! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6381) DAG name changes style of some columns but not all
[ https://issues.apache.org/jira/browse/AIRFLOW-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009316#comment-17009316 ] ASF subversion and git services commented on AIRFLOW-6381: -- Commit 748e3c3519b6cfeb82929efa6cc2d11ca0154b49 in airflow's branch refs/heads/master from Jay [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=748e3c3 ] [AIRFLOW-6381] Remove styling based on DAG id from DAGs page (#6985) > DAG name changes style of some columns but not all > -- > > Key: AIRFLOW-6381 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6381 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.2 >Reporter: Jay K >Assignee: Jay K >Priority: Trivial > Fix For: 2.0.0 > > Attachments: image-2019-12-27-22-38-55-209.png > > > !image-2019-12-27-22-38-55-209.png|width=782,height=114! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil merged pull request #6985: [AIRFLOW-6381] Remove styling based on DAG id from DAGs page
kaxil merged pull request #6985: [AIRFLOW-6381] Remove styling based on DAG id from DAGs page URL: https://github.com/apache/airflow/pull/6985 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] boring-cyborg[bot] commented on issue #6985: [AIRFLOW-6381] Remove styling based on DAG id from DAGs page
boring-cyborg[bot] commented on issue #6985: [AIRFLOW-6381] Remove styling based on DAG id from DAGs page URL: https://github.com/apache/airflow/pull/6985#issuecomment-571409212 Awesome work, congrats on your first merged pull request! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6993: [AIRFLOW-4502] new cli command - task_states_for_dag_run
mik-laj commented on a change in pull request #6993: [AIRFLOW-4502] new cli command - task_states_for_dag_run URL: https://github.com/apache/airflow/pull/6993#discussion_r363566924 ## File path: tests/cli/commands/test_task_command.py ## @@ -184,6 +184,20 @@ def test_task_state(self): 'tasks', 'state', 'example_bash_operator', 'runme_0', DEFAULT_DATE.isoformat()])) +def test_task_states_for_dag_run(self): +with redirect_stdout(io.StringIO()) as stdout: +task_command.task_states_for_dag_run(self.parser.parse_args([ +'tasks', 'states_for_dag_run', 'example_bash_operator', DEFAULT_DATE.isoformat()])) + +# can't get perfect match as start/end date are variable Review comment: I think you should save this data to the database in the code of test, and then the data will be constant. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] yuqian90 commented on a change in pull request #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates
yuqian90 commented on a change in pull request #7038: [AIRFLOW-4495] allow
externally triggered dags to run for future exec dates
URL: https://github.com/apache/airflow/pull/7038#discussion_r363565941
##
File path: airflow/jobs/scheduler_job.py
##
@@ -629,13 +629,18 @@ def _process_task_instances(self, dag,
task_instances_list, session=None):
active_dag_runs = []
for run in dag_runs:
self.log.info("Examining DAG run %s", run)
-# don't consider runs that are executed in the future
+# don't consider runs that are executed in the future unless
+# specified by config and schedule_interval is None
if run.execution_date > timezone.utcnow():
-self.log.error(
-"Execution date is in future: %s",
-run.execution_date
-)
-continue
+if run.dag.schedule_interval or not conf.getboolean(
Review comment:
This can now be:
```python
if run.execution_date > timezone.utcnow() and not
run.dag.allow_future_exec_dates:
self.log.error(
"Execution date is in future: %s",
run.execution_date
)
continue
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] yuqian90 commented on a change in pull request #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates
yuqian90 commented on a change in pull request #7038: [AIRFLOW-4495] allow
externally triggered dags to run for future exec dates
URL: https://github.com/apache/airflow/pull/7038#discussion_r363566046
##
File path: airflow/ti_deps/deps/runnable_exec_date_dep.py
##
@@ -30,11 +31,17 @@ class RunnableExecDateDep(BaseTIDep):
def _get_dep_statuses(self, ti, session, dep_context):
cur_date = timezone.utcnow()
+# don't consider runs that are executed in the future unless
+# specified by config and schedule_interval is None
if ti.execution_date > cur_date:
-yield self._failing_status(
-reason="Execution date {0} is in the future (the current "
- "date is {1}).".format(ti.execution_date.isoformat(),
- cur_date.isoformat()))
+if ti.task.dag.schedule_interval or not conf.getboolean(
Review comment:
This can be:
```python
if not ti.task.dag.allow_future_exec_dates:
...
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009301#comment-17009301
]
Jarek Potiuk commented on AIRFLOW-2516:
---
Cool!. Keep my fingers crossed. What's so confusing in here ? I think it's
quite OK?
> Deadlock found when trying to update task_instance table
>
>
> Key: AIRFLOW-2516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2516
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
>Affects Versions: 1.8.0
>Reporter: Jeff Liu
>Assignee: Jarek Potiuk
>Priority: Major
> Attachments: Screenshot 2019-12-30 at 10.42.52.png,
> image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png,
> jobs.py, jobs_fixed_deadlock_possibly_1.9.py,
> jobs_fixed_deadlock_possibly_1.9.py,
> scheduler_job_fixed_deadlock_possibly_1.10.6.py
>
>
>
>
> {code:java}
> [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask:
> [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue:
> airflow run production_wipeout_wipe_manager.Carat Carat_20180227
> 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py
> [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback
> (most recent call last):
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/bin/airflow", line 27, in
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> args.func(args)
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> pool=args.pool,
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in
> wrapper
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result =
> func(*args, **kwargs)
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in
> _run_raw_task
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> self.handle_failure(e, test_mode, context)
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in
> handle_failure
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> session.merge(self)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1920, in merge
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask:
> _resolve_conflict_map=_resolve_conflict_map)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1974, in _merge
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged =
> self.query(mapper.class_).get(key[1])
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882,
> in get
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident,
> loading.load_on_pk_identity)
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952,
> in _get_impl
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> db_load_fn(self, primary_key_identity)
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247,
> in load_on_pk_i
> dentity
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> q.one()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884,
> in one
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> self.one_or_none()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854,
> in one_or_none
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> list(self)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925,
> in __iter__
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return
> self._execute_and_instances(context)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO
[GitHub] [airflow] mik-laj commented on a change in pull request #6985: [AIRFLOW-6381] Remove styling based on DAG id from DAGs page
mik-laj commented on a change in pull request #6985: [AIRFLOW-6381] Remove styling based on DAG id from DAGs page URL: https://github.com/apache/airflow/pull/6985#discussion_r363566077 ## File path: airflow/www/templates/airflow/dags.html ## @@ -88,7 +88,7 @@ DAGs - Review comment: I've seen it before, but I believed it had a purpose. :-D This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #7086: [AIRFLOW-XXXX] Fix typos and broken links in development docs
mik-laj commented on a change in pull request #7086: [AIRFLOW-] Fix typos and broken links in development docs URL: https://github.com/apache/airflow/pull/7086#discussion_r363565780 ## File path: TESTING.rst ## @@ -278,7 +278,7 @@ Local and Remote Debugging == One of the great benefits of using the local virtualenv and Breeze is an option to run -local debugging in your IDE graphical interface. You can also use ``ipdb`` +local debugging in `your IDE graphical interface `__. You can also use ``ipdb`` if you prefer `console debugging <#breeze-debugging-with-ipdb>`__. Review comment: @potiuk Do you know where to find these documentation? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Reopened] (AIRFLOW-4355) Externally triggered DAG is marked as 'success' even if a task has been 'removed'!
[
https://issues.apache.org/jira/browse/AIRFLOW-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarek Potiuk reopened AIRFLOW-4355:
---
Resolved by mistake.
> Externally triggered DAG is marked as 'success' even if a task has been
> 'removed'!
> --
>
> Key: AIRFLOW-4355
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4355
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, DagRun, scheduler
>Affects Versions: 1.10.3
>Reporter: t oo
>Assignee: t oo
>Priority: Blocker
> Labels: dynamic
> Fix For: 2.0.0
>
> Attachments: dag_success_even_if_task_removed.png, treeview.png
>
>
> note: all my dags are purely externally triggered
> *Issue:* Dag has 5 parallel tasks that ran successfully and 1 final task that
> somehow got 'removed' state (prior dag runs had 'failed' state) and never ran
> successfully but still the DAG is showing success!
>
> *Command ran* (note that previous commands like airflow trigger_dag -e
> 20190412 qsr_coremytbl were run before and failed for valid reason (ie python
> task failing) ):
> airflow trigger_dag -e 20190412T08:00 qsr_coremytbl --conf '\{"hourstr":"08"}'
>
> *some logs on prior instance of airflow (ec2 was autohealed):*
> [2019-04-18 08:29:40,678] \{logging_mixin.py:95} INFO - [2019-04-18
> 08:29:40,678] {__init__.py:4897} WARNING - Failed to get task ' qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl 2019-04-12 08:00:00+00:00 [None]>'
> for dag ''. Marking it as removed.
> [2019-04-18 08:29:43,582] \{logging_mixin.py:95} INFO - [2019-04-18
> 08:29:43,582] {__init__.py:4906} INFO - Restoring task ' qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl 2019-04-12 08:00:00+00:00 [removed]>'
> which was previously removed from DAG ''
> [2019-04-18 08:29:43,618] \{jobs.py:1787} INFO - Creating / updating
> 08:00:00+00:00 [scheduled]> in ORM
> [2019-04-18 08:29:43,676] \{logging_mixin.py:95} INFO - [2019-04-18
> 08:29:43,676] {__init__.py:4897} WARNING - Failed to get task ' qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl 2019-04-12 08:00:00+00:00
> [scheduled]>' for dag ''. Marking it as removed.
>
> *some logs on newer ec2:*
> [myuser@host logs]$ grep -i hive -R * | sed 's#[0-9]#x#g' | sort | uniq -c |
> grep -v 'airflow-webserver-access.log'
> 2 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx qsr_coremytbl
> REPAIR_HIVE_schemeh.mytbl log -xx-xx xx:xx:xx.xx rsawyerx
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id',
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl')]
> 1 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx qsr_coremytbl
> REPAIR_HIVE_schemeh.mytbl log -xx-xx xx:xx:xx.xx rsawyerx
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id',
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl'), ('format',
> u'json')]
> 1 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx qsr_coremytbl
> REPAIR_HIVE_schemeh.mytbl rendered -xx-xx xx:xx:xx.xx rsawyerx
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id',
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl')]
> 1 audit/airflow-audit.log:-xx-xx xx:xx:xx.xx qsr_coremytbl
> REPAIR_HIVE_schemeh.mytbl task -xx-xx xx:xx:xx.xx rsawyerx
> [('execution_date', u'-xx-xxTxx:xx:xx+xx:xx'), ('task_id',
> u'REPAIR_HIVE_schemeh.mytbl'), ('dag_id', u'qsr_coremytbl')]
> 1 scheduler/latest/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx]
> \{jobs.py:} INFO - Creating / updating qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl -xx-xx xx:xx:xx+xx:xx
> [scheduled]> in ORM
> 71 scheduler/latest/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx]
> \{logging_mixin.py:xx} INFO - [-xx-xx xx:xx:xx,xxx] {__init__.py:}
> INFO - Restoring task ' -xx-xx xx:xx:xx+xx:xx [removed]>' which was previously removed from DAG
> ''
> 1 scheduler/-xx-xx/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx]
> \{jobs.py:} INFO - Creating / updating qsr_coremytbl.REPAIR_HIVE_schemeh.mytbl -xx-xx xx:xx:xx+xx:xx
> [scheduled]> in ORM
> 71 scheduler/-xx-xx/qsr_dag_generation.py.log:[-xx-xx xx:xx:xx,xxx]
> \{logging_mixin.py:xx} INFO - [-xx-xx xx:xx:xx,xxx] {__init__.py:}
> INFO - Restoring task ' -xx-xx xx:xx:xx+xx:xx [removed]>' which was previously removed from DAG
> ''
>
> mysql> *select * from task_instance where task_id like '%REP%';#*
>
>
[GitHub] [airflow] mik-laj commented on a change in pull request #6954: [AIRFLOW-4355] removed task should not lead to dagrun success
mik-laj commented on a change in pull request #6954: [AIRFLOW-4355] removed
task should not lead to dagrun success
URL: https://github.com/apache/airflow/pull/6954#discussion_r363565513
##
File path: airflow/models/dagrun.py
##
@@ -311,32 +313,51 @@ def update_state(self, session=None):
leaf_tis = [ti for ti in tis if ti.task_id in {t.task_id for t in
dag.leaves}]
-# if all roots finished and at least one failed, the run failed
-if not unfinished_tasks and any(
-leaf_ti.state in {State.FAILED, State.UPSTREAM_FAILED} for leaf_ti
in leaf_tis
-):
-self.log.info('Marking run %s failed', self)
-self.set_state(State.FAILED)
-dag.handle_callback(self, success=False, reason='task_failure',
-session=session)
-
-# if all leafs succeeded and no unfinished tasks, the run succeeded
-elif not unfinished_tasks and all(
-leaf_ti.state in {State.SUCCESS, State.SKIPPED} for leaf_ti in
leaf_tis
-):
-self.log.info('Marking run %s successful', self)
-self.set_state(State.SUCCESS)
-dag.handle_callback(self, success=True, reason='success',
session=session)
-
-# if *all tasks* are deadlocked, the run failed
-elif (unfinished_tasks and none_depends_on_past and
- none_task_concurrency and no_dependencies_met):
-self.log.info('Deadlock; marking run %s failed', self)
-self.set_state(State.FAILED)
-dag.handle_callback(self, success=False,
reason='all_tasks_deadlocked',
-session=session)
-
-# finally, if the roots aren't done, the dag is still running
+if len(tis) == len(dag.tasks):
Review comment:
Can you tell me a little more about this condition? In my opinion, it will
always be fulfilled because we have a transactional database. When will this
condition be not met?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success
mik-laj commented on issue #6954: [AIRFLOW-4355] removed task should not lead
to dagrun success
URL: https://github.com/apache/airflow/pull/6954#issuecomment-571398463
It works for me.
```
@conf_vars({
('scheduler', 'removed_tasks_lead_to_dagrun_failure'): 'True'
})
def test_dagrun_removed_tasks_lead_to_dagrun_failure_true(self):
session = settings.Session()
on_failure_callback = mock.MagicMock()
dag = DAG(
'test_dagrun_removed_tasks_lead_to_dagrun_failure_true',
start_date=DEFAULT_DATE,
default_args={'owner': 'owner1'},
on_failure_callback=on_failure_callback
)
dag.clear()
with dag:
op1 = DummyOperator(task_id='A')
op2 = DummyOperator(task_id='B')
op2.set_upstream(op1)
dag.clear()
now = timezone.utcnow()
dr = dag.create_dagrun(run_id='test_dagrun_deadlock',
state=State.RUNNING,
execution_date=now,
start_date=now)
ti_op1 = dr.get_task_instance(task_id=op1.task_id)
ti_op1.set_state(state=State.REMOVED, session=session)
ti_op2 = dr.get_task_instance(task_id=op2.task_id)
ti_op2.set_state(state=State.SUCCESS, session=session)
del dag.task_dict[ti_op1.task_id]
dr.update_state()
self.assertEqual(dr.state, State.FAILED)
kall = on_failure_callback
callback_context = kall.call_args[0][0]
self.assertEqual('removed_tasks', callback_context['reason'])
@conf_vars({
('scheduler', 'removed_tasks_lead_to_dagrun_failure'): 'False'
})
def test_dagrun_removed_tasks_lead_to_dagrun_failure_false(self):
session = settings.Session()
on_failure_callback = mock.MagicMock()
dag = DAG(
'test_dagrun_removed_tasks_lead_to_dagrun_failure_false',
start_date=DEFAULT_DATE,
default_args={'owner': 'owner1'},
on_success_callback=on_failure_callback
)
dag.clear()
with dag:
op1 = DummyOperator(task_id='A')
op2 = DummyOperator(task_id='B')
op2.set_upstream(op1)
dag.clear()
now = timezone.utcnow()
dr = dag.create_dagrun(run_id='test_dagrun_deadlock',
state=State.RUNNING,
execution_date=now,
start_date=now)
ti_op1 = dr.get_task_instance(task_id=op1.task_id)
ti_op1.set_state(state=State.REMOVED, session=session)
ti_op2 = dr.get_task_instance(task_id=op2.task_id)
ti_op2.set_state(state=State.SUCCESS, session=session)
del dag.task_dict[ti_op1.task_id]
dr.update_state()
self.assertEqual(dr.state, State.SUCCESS)
kall = on_failure_callback
callback_context = kall.call_args[0][0]
self.assertEqual('success', callback_context['reason'])
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success
mik-laj edited a comment on issue #6954: [AIRFLOW-4355] removed task should not
lead to dagrun success
URL: https://github.com/apache/airflow/pull/6954#issuecomment-571398463
It works for me.
```python
@conf_vars({
('scheduler', 'removed_tasks_lead_to_dagrun_failure'): 'True'
})
def test_dagrun_removed_tasks_lead_to_dagrun_failure_true(self):
session = settings.Session()
on_failure_callback = mock.MagicMock()
dag = DAG(
'test_dagrun_removed_tasks_lead_to_dagrun_failure_true',
start_date=DEFAULT_DATE,
default_args={'owner': 'owner1'},
on_failure_callback=on_failure_callback
)
dag.clear()
with dag:
op1 = DummyOperator(task_id='A')
op2 = DummyOperator(task_id='B')
op2.set_upstream(op1)
dag.clear()
now = timezone.utcnow()
dr = dag.create_dagrun(run_id='test_dagrun_deadlock',
state=State.RUNNING,
execution_date=now,
start_date=now)
ti_op1 = dr.get_task_instance(task_id=op1.task_id)
ti_op1.set_state(state=State.REMOVED, session=session)
ti_op2 = dr.get_task_instance(task_id=op2.task_id)
ti_op2.set_state(state=State.SUCCESS, session=session)
del dag.task_dict[ti_op1.task_id]
dr.update_state()
self.assertEqual(dr.state, State.FAILED)
kall = on_failure_callback
callback_context = kall.call_args[0][0]
self.assertEqual('removed_tasks', callback_context['reason'])
@conf_vars({
('scheduler', 'removed_tasks_lead_to_dagrun_failure'): 'False'
})
def test_dagrun_removed_tasks_lead_to_dagrun_failure_false(self):
session = settings.Session()
on_failure_callback = mock.MagicMock()
dag = DAG(
'test_dagrun_removed_tasks_lead_to_dagrun_failure_false',
start_date=DEFAULT_DATE,
default_args={'owner': 'owner1'},
on_success_callback=on_failure_callback
)
dag.clear()
with dag:
op1 = DummyOperator(task_id='A')
op2 = DummyOperator(task_id='B')
op2.set_upstream(op1)
dag.clear()
now = timezone.utcnow()
dr = dag.create_dagrun(run_id='test_dagrun_deadlock',
state=State.RUNNING,
execution_date=now,
start_date=now)
ti_op1 = dr.get_task_instance(task_id=op1.task_id)
ti_op1.set_state(state=State.REMOVED, session=session)
ti_op2 = dr.get_task_instance(task_id=op2.task_id)
ti_op2.set_state(state=State.SUCCESS, session=session)
del dag.task_dict[ti_op1.task_id]
dr.update_state()
self.assertEqual(dr.state, State.SUCCESS)
kall = on_failure_callback
callback_context = kall.call_args[0][0]
self.assertEqual('success', callback_context['reason'])
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success
codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#issuecomment-569545898 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=h1) Report > Merging [#6954](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3d82e8cf6a9b6b0af3fafef0769f77ced1974cbf?src=pr=desc) will **increase** coverage by `0.09%`. > The diff coverage is `69.23%`. [](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6954 +/- ## == + Coverage 84.68% 84.77% +0.09% == Files 680 680 Lines 3868638822 +136 == + Hits3276132913 +152 + Misses 5925 5909 -16 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/dagrun.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFncnVuLnB5) | `92.72% <69.23%> (-3.86%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.45% <0%> (-4.55%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `88.84% <0%> (-0.46%)` | :arrow_down: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `90.62% <0%> (-0.34%)` | :arrow_down: | | ... and [19 more](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=footer). Last update [3d82e8c...1ff911c](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success
codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#issuecomment-569545898 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=h1) Report > Merging [#6954](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0614edbb59657e4998d80e7092214c8e657861a0?src=pr=desc) will **increase** coverage by `0.21%`. > The diff coverage is `78.27%`. [](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6954 +/- ## == + Coverage 84.56% 84.77% +0.21% == Files 679 680 +1 Lines 3854738822 +275 == + Hits3259632913 +317 + Misses 5951 5909 -42 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: | | [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: | | [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `97.43% <ø> (ø)` | :arrow_up: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <ø> (-0.08%)` | :arrow_down: | | [...flow/example\_dags/example\_docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <ø> (ø)` | :arrow_up: | | [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.29% <ø> (+3.19%)` | :arrow_up: | | ... and [170 more](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=footer). Last update [0614edb...1ff911c](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success
codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#issuecomment-569545898 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=h1) Report > Merging [#6954](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3d82e8cf6a9b6b0af3fafef0769f77ced1974cbf?src=pr=desc) will **decrease** coverage by `0.12%`. > The diff coverage is `69.23%`. [](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6954 +/- ## == - Coverage 84.68% 84.55% -0.13% == Files 680 680 Lines 3868638822 +136 == + Hits3276132827 +66 - Misses 5925 5995 +70 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/dagrun.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFncnVuLnB5) | `92.72% <69.23%> (-3.86%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.45% <0%> (-4.55%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `88.84% <0%> (-0.46%)` | :arrow_down: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `90.62% <0%> (-0.34%)` | :arrow_down: | | ... and [13 more](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=footer). Last update [3d82e8c...1ff911c](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates
codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#issuecomment-570935774 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=h1) Report > Merging [#7038](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/41a9bfd13166be6cf61eb61f14c6fb363cd0b472?src=pr=desc) will **decrease** coverage by `0.26%`. > The diff coverage is `72.72%`. [](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7038 +/- ## == - Coverage 85.06% 84.79% -0.27% == Files 680 680 Lines 3875738835 +78 == - Hits3296832931 -37 - Misses 5789 5904 +115 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `88.71% <0%> (-0.13%)` | :arrow_down: | | [airflow/ti\_deps/deps/runnable\_exec\_date\_dep.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcnVubmFibGVfZXhlY19kYXRlX2RlcC5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `90.67% <100%> (+0.05%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.65% <0%> (-4.35%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <0%> (-0.08%)` | :arrow_down: | | ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=footer). Last update [41a9bfd...108176e](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates
codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#issuecomment-570935774 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=h1) Report > Merging [#7038](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/41a9bfd13166be6cf61eb61f14c6fb363cd0b472?src=pr=desc) will **decrease** coverage by `0.26%`. > The diff coverage is `72.72%`. [](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7038 +/- ## == - Coverage 85.06% 84.79% -0.27% == Files 680 680 Lines 3875738835 +78 == - Hits3296832931 -37 - Misses 5789 5904 +115 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `88.71% <0%> (-0.13%)` | :arrow_down: | | [airflow/ti\_deps/deps/runnable\_exec\_date\_dep.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcnVubmFibGVfZXhlY19kYXRlX2RlcC5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `90.67% <100%> (+0.05%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.65% <0%> (-4.35%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <0%> (-0.08%)` | :arrow_down: | | ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=footer). Last update [41a9bfd...108176e](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates
codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#issuecomment-570935774 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=h1) Report > Merging [#7038](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/41a9bfd13166be6cf61eb61f14c6fb363cd0b472?src=pr=desc) will **decrease** coverage by `0.26%`. > The diff coverage is `72.72%`. [](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7038 +/- ## == - Coverage 85.06% 84.79% -0.27% == Files 680 680 Lines 3875738835 +78 == - Hits3296832931 -37 - Misses 5789 5904 +115 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `88.71% <0%> (-0.13%)` | :arrow_down: | | [airflow/ti\_deps/deps/runnable\_exec\_date\_dep.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcnVubmFibGVfZXhlY19kYXRlX2RlcC5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `90.67% <100%> (+0.05%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.65% <0%> (-4.35%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <0%> (-0.08%)` | :arrow_down: | | ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=footer). Last update [41a9bfd...108176e](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates
codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#issuecomment-570935774 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=h1) Report > Merging [#7038](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/41a9bfd13166be6cf61eb61f14c6fb363cd0b472?src=pr=desc) will **decrease** coverage by `0.26%`. > The diff coverage is `72.72%`. [](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7038 +/- ## == - Coverage 85.06% 84.79% -0.27% == Files 680 680 Lines 3875738835 +78 == - Hits3296832931 -37 - Misses 5789 5904 +115 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `88.71% <0%> (-0.13%)` | :arrow_down: | | [airflow/ti\_deps/deps/runnable\_exec\_date\_dep.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcnVubmFibGVfZXhlY19kYXRlX2RlcC5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/models/dag.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnLnB5) | `90.67% <100%> (+0.05%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/contrib/operators/gcp\_transfer\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfdHJhbnNmZXJfb3BlcmF0b3IucHk=) | `95.65% <0%> (-4.35%)` | :arrow_down: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <0%> (-0.08%)` | :arrow_down: | | ... and [6 more](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=footer). Last update [41a9bfd...108176e](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009281#comment-17009281
]
Zhidong commented on AIRFLOW-2516:
--
Hi [~potiuk], no new deadlock observed since last update (4days ago).
Please take a look at following statements from [Locking
Reads|https://dev.mysql.com/doc/refman/8.0/en/innodb-locking-reads.html], it
confuses me a lot.
{code:java}
SELECT ... FOR UPDATE
For index records the search encounters, locks the rows and any associated
index entries, the same as if you issued an UPDATE statement for those rows.
A SELECT ... FOR UPDATE reads the latest available data, setting exclusive
locks on each row it reads. Thus, it sets the same locks a searched SQL UPDATE
would set on the rows.
{code}
> Deadlock found when trying to update task_instance table
>
>
> Key: AIRFLOW-2516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2516
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
>Affects Versions: 1.8.0
>Reporter: Jeff Liu
>Assignee: Jarek Potiuk
>Priority: Major
> Attachments: Screenshot 2019-12-30 at 10.42.52.png,
> image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png,
> jobs.py, jobs_fixed_deadlock_possibly_1.9.py,
> jobs_fixed_deadlock_possibly_1.9.py,
> scheduler_job_fixed_deadlock_possibly_1.10.6.py
>
>
>
>
> {code:java}
> [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask:
> [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue:
> airflow run production_wipeout_wipe_manager.Carat Carat_20180227
> 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py
> [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback
> (most recent call last):
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/bin/airflow", line 27, in
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> args.func(args)
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> pool=args.pool,
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in
> wrapper
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result =
> func(*args, **kwargs)
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in
> _run_raw_task
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> self.handle_failure(e, test_mode, context)
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in
> handle_failure
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> session.merge(self)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1920, in merge
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask:
> _resolve_conflict_map=_resolve_conflict_map)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1974, in _merge
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged =
> self.query(mapper.class_).get(key[1])
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882,
> in get
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident,
> loading.load_on_pk_identity)
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952,
> in _get_impl
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> db_load_fn(self, primary_key_identity)
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247,
> in load_on_pk_i
> dentity
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> q.one()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884,
> in one
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> self.one_or_none()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
>
[GitHub] [airflow] codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success
codecov-io edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success URL: https://github.com/apache/airflow/pull/6954#issuecomment-569545898 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=h1) Report > Merging [#6954](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/0614edbb59657e4998d80e7092214c8e657861a0?src=pr=desc) will **increase** coverage by `0.21%`. > The diff coverage is `78.23%`. [](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6954 +/- ## == + Coverage 84.56% 84.77% +0.21% == Files 679 680 +1 Lines 3854738822 +275 == + Hits3259632913 +317 + Misses 5951 5909 -42 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/gcp/operators/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2RhdGFmbG93LnB5) | `99.07% <ø> (ø)` | :arrow_up: | | [airflow/gcp/hooks/dataflow.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGF0YWZsb3cucHk=) | `89.28% <ø> (ø)` | :arrow_up: | | [airflow/utils/weight\_rule.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy93ZWlnaHRfcnVsZS5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [airflow/example\_dags/docker\_copy\_data.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZG9ja2VyX2NvcHlfZGF0YS5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `97.43% <ø> (ø)` | :arrow_up: | | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.01% <ø> (-0.08%)` | :arrow_down: | | [...flow/example\_dags/example\_docker\_swarm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfc3dhcm1fb3BlcmF0b3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | [airflow/gcp/utils/mlengine\_operator\_utils.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvdXRpbHMvbWxlbmdpbmVfb3BlcmF0b3JfdXRpbHMucHk=) | `95.34% <ø> (ø)` | :arrow_up: | | [airflow/utils/state.py](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `96.29% <ø> (+3.19%)` | :arrow_up: | | ... and [170 more](https://codecov.io/gh/apache/airflow/pull/6954/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=footer). Last update [0614edb...1ff911c](https://codecov.io/gh/apache/airflow/pull/6954?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates
codecov-io edited a comment on issue #7038: [AIRFLOW-4495] allow externally triggered dags to run for future exec dates URL: https://github.com/apache/airflow/pull/7038#issuecomment-570935774 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@b5bd9ab`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `98.91%`. [](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=tree) ```diff @@Coverage Diff@@ ## master#7038 +/- ## = Coverage ? 84.79% = Files ? 680 Lines ?38835 Branches ?0 = Hits ?32931 Misses? 5904 Partials ?0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/ti\_deps/deps/prev\_dagrun\_dep.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcHJldl9kYWdydW5fZGVwLnB5) | `80.55% <ø> (ø)` | | | [...irflow/contrib/operators/emr\_add\_steps\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9lbXJfYWRkX3N0ZXBzX29wZXJhdG9yLnB5) | `91.42% <ø> (ø)` | | | [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `98.61% <ø> (ø)` | | | [airflow/sensors/external\_task\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL2V4dGVybmFsX3Rhc2tfc2Vuc29yLnB5) | `85.96% <ø> (ø)` | | | [...ib/operators/azure\_container\_instances\_operator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9henVyZV9jb250YWluZXJfaW5zdGFuY2VzX29wZXJhdG9yLnB5) | `78.14% <ø> (ø)` | | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.25% <ø> (ø)` | | | [airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=) | `52.5% <0%> (ø)` | | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `89.74% <100%> (ø)` | | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.4% <100%> (ø)` | | | [airflow/api/common/experimental/trigger\_dag.py](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC90cmlnZ2VyX2RhZy5weQ==) | `97.95% <100%> (ø)` | | | ... and [1 more](https://codecov.io/gh/apache/airflow/pull/7038/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=footer). Last update [b5bd9ab...108176e](https://codecov.io/gh/apache/airflow/pull/7038?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] zhongjiajie commented on issue #6881: [AIRFLOW-6326] Sort cli commands and arg
zhongjiajie commented on issue #6881: [AIRFLOW-6326] Sort cli commands and arg URL: https://github.com/apache/airflow/pull/6881#issuecomment-571391602 Code now sort by long option and CI green, the behavior like below ```sh usage: airflow tasks run [-h] [--cfg_path CFG_PATH] [-f] [-A] [-i] [-I] [-int] [-l] [-m] [-p PICKLE] [--pool POOL] [--ship_dag] [-sd SUBDIR] dag_id task_id execution_date positional arguments: dag_idThe id of the dag task_id The id of the task execution_dateThe execution date of the DAG optional arguments: -h, --helpshow this help message and exit --cfg_path CFG_PATH Path to config file to use instead of airflow.cfg -f, --force Ignore previous task instance state, rerun regardless if task already succeeded/failed -A, --ignore_all_dependencies Ignores all non-critical dependencies, including ignore_ti_state and ignore_task_deps -i, --ignore_dependencies Ignore task-specific dependencies, e.g. upstream, depends_on_past, and retry delay dependencies -I, --ignore_depends_on_past Ignore depends_on_past dependencies (but respect upstream dependencies) -int, --interactive Do not capture standard output and error streams (useful for interactive debugging) -l, --local Run the task using the LocalExecutor -m, --mark_successMark jobs as succeeded without running them -p PICKLE, --pickle PICKLE Serialized pickle object of the entire dag (used internally) --pool POOL Resource pool to use --ship_dagPickles (serializes) the DAG and ships it to the worker -sd SUBDIR, --subdir SUBDIR File location or directory from which to look for the dag. Defaults to '[AIRFLOW_HOME]/dags' where [AIRFLOW_HOME] is the value you set for 'AIRFLOW_HOME' config you set in 'airflow.cfg' ``` ```sh usage: airflow dags trigger [-h] [-c CONF] [-e EXEC_DATE] [-r RUN_ID] [-sd SUBDIR] dag_id positional arguments: dag_idThe id of the dag optional arguments: -h, --helpshow this help message and exit -c CONF, --conf CONF JSON string that gets pickled into the DagRun's conf attribute -e EXEC_DATE, --exec_date EXEC_DATE The execution date of the DAG -r RUN_ID, --run_id RUN_ID Helps to identify this run -sd SUBDIR, --subdir SUBDIR File location or directory from which to look for the dag. Defaults to '[AIRFLOW_HOME]/dags' where [AIRFLOW_HOME] is the value you set for 'AIRFLOW_HOME' config you set in 'airflow.cfg' ``` PTAL @potiuk @mik-laj This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] tooptoop4 commented on issue #7089: [AIRFLOW-6452] scheduler_job.py - remove excess sleep/log/duration calls
tooptoop4 commented on issue #7089: [AIRFLOW-6452] scheduler_job.py - remove excess sleep/log/duration calls URL: https://github.com/apache/airflow/pull/7089#issuecomment-571391385 before this PR it had 2 sleeps (no need!), it seems the one I removed was there before the processor_poll_interval sleep was added. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-6451) self._print_stat() in dag_processing.py should be skippable by config option
[ https://issues.apache.org/jira/browse/AIRFLOW-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] t oo updated AIRFLOW-6451: -- Description: perf benefit clean up extra poll, logs, typos was:perf benefit > self._print_stat() in dag_processing.py should be skippable by config option > > > Key: AIRFLOW-6451 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6451 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 1.10.7 >Reporter: t oo >Priority: Minor > > perf benefit > clean up extra poll, logs, typos -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (AIRFLOW-6452) scheduler_job.py - remove excess sleep/log/duration calls
[
https://issues.apache.org/jira/browse/AIRFLOW-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on AIRFLOW-6452 started by t oo.
-
> scheduler_job.py - remove excess sleep/log/duration calls
> -
>
> Key: AIRFLOW-6452
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6452
> Project: Apache Airflow
> Issue Type: Improvement
> Components: scheduler
>Affects Versions: 1.10.7
>Reporter: t oo
>Assignee: t oo
>Priority: Minor
>
> remove a lot of these debug calls, wrap some in boolean of loglevel, remove
> the 2nd sleep and stuff about getting duration/start/end.etc:
> self.log.debug("Starting Loop...")
> loop_start_time = time.time()
> if self.using_sqlite:
> self.processor_agent.heartbeat()
> # For the sqlite case w/ 1 thread, wait until the processor
> # is finished to avoid concurrent access to the DB.
> self.log.debug(
> "Waiting for processors to finish since we're using
> sqlite")
> self.processor_agent.wait_until_finished()
> self.log.debug("Harvesting DAG parsing results")
> simple_dags = self._get_simple_dags()
> self.log.debug("Harvested {} SimpleDAGs".format(len(simple_dags)))
> # Send tasks for execution if available
> simple_dag_bag = SimpleDagBag(simple_dags)
> if not
> self._validate_and_run_task_instances(simple_dag_bag=simple_dag_bag):
> continue
> # Heartbeat the scheduler periodically
> time_since_last_heartbeat = (timezone.utcnow() -
>
> last_self_heartbeat_time).total_seconds()
> if time_since_last_heartbeat > self.heartrate:
> self.log.debug("Heartbeating the scheduler")
> self.heartbeat()
> last_self_heartbeat_time = timezone.utcnow()
> loop_end_time = time.time()
> loop_duration = loop_end_time - loop_start_time
> self.log.debug(
> "Ran scheduling loop in %.2f seconds",
> loop_duration)
> if not is_unit_test:
> self.log.debug("Sleeping for %.2f seconds",
> self._processor_poll_interval)
> time.sleep(self._processor_poll_interval)
> if self.processor_agent.done:
> self.log.info("Exiting scheduler loop as all files"
> " have been processed {}
> times".format(self.num_runs))
> break
> if loop_duration < 1 and not is_unit_test:
> sleep_length = 1 - loop_duration
> self.log.debug(
> "Sleeping for {0:.2f} seconds to prevent excessive
> logging"
> .format(sleep_length))
> sleep(sleep_length)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhidong reassigned AIRFLOW-2516:
Assignee: Jarek Potiuk (was: Zhidong)
> Deadlock found when trying to update task_instance table
>
>
> Key: AIRFLOW-2516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2516
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
>Affects Versions: 1.8.0
>Reporter: Jeff Liu
>Assignee: Jarek Potiuk
>Priority: Major
> Attachments: Screenshot 2019-12-30 at 10.42.52.png,
> image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png,
> jobs.py, jobs_fixed_deadlock_possibly_1.9.py,
> jobs_fixed_deadlock_possibly_1.9.py,
> scheduler_job_fixed_deadlock_possibly_1.10.6.py
>
>
>
>
> {code:java}
> [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask:
> [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue:
> airflow run production_wipeout_wipe_manager.Carat Carat_20180227
> 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py
> [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback
> (most recent call last):
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/bin/airflow", line 27, in
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> args.func(args)
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> pool=args.pool,
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in
> wrapper
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result =
> func(*args, **kwargs)
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in
> _run_raw_task
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> self.handle_failure(e, test_mode, context)
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in
> handle_failure
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> session.merge(self)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1920, in merge
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask:
> _resolve_conflict_map=_resolve_conflict_map)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1974, in _merge
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged =
> self.query(mapper.class_).get(key[1])
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882,
> in get
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident,
> loading.load_on_pk_identity)
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952,
> in _get_impl
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> db_load_fn(self, primary_key_identity)
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247,
> in load_on_pk_i
> dentity
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> q.one()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884,
> in one
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> self.one_or_none()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854,
> in one_or_none
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> list(self)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925,
> in __iter__
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return
> self._execute_and_instances(context)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line
[jira] [Commented] (AIRFLOW-6452) scheduler_job.py - remove excess sleep/log/duration calls
[
https://issues.apache.org/jira/browse/AIRFLOW-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009275#comment-17009275
]
ASF GitHub Bot commented on AIRFLOW-6452:
-
tooptoop4 commented on pull request #7089: [AIRFLOW-6452] scheduler_job.py -
remove excess sleep/log/duration calls
URL: https://github.com/apache/airflow/pull/7089
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
- [ ] Description above provides context of the change
- [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> scheduler_job.py - remove excess sleep/log/duration calls
> -
>
> Key: AIRFLOW-6452
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6452
> Project: Apache Airflow
> Issue Type: Improvement
> Components: scheduler
>Affects Versions: 1.10.7
>Reporter: t oo
>Priority: Minor
>
> remove a lot of these debug calls, wrap some in boolean of loglevel, remove
> the 2nd sleep and stuff about getting duration/start/end.etc:
> self.log.debug("Starting Loop...")
> loop_start_time = time.time()
> if self.using_sqlite:
> self.processor_agent.heartbeat()
> # For the sqlite case w/ 1 thread, wait until the processor
> # is finished to avoid concurrent access to the DB.
> self.log.debug(
> "Waiting for processors to finish since we're using
> sqlite")
> self.processor_agent.wait_until_finished()
> self.log.debug("Harvesting DAG parsing results")
> simple_dags = self._get_simple_dags()
> self.log.debug("Harvested {} SimpleDAGs".format(len(simple_dags)))
> # Send tasks for execution if available
> simple_dag_bag = SimpleDagBag(simple_dags)
> if not
> self._validate_and_run_task_instances(simple_dag_bag=simple_dag_bag):
> continue
> # Heartbeat the scheduler periodically
> time_since_last_heartbeat = (timezone.utcnow() -
>
> last_self_heartbeat_time).total_seconds()
> if time_since_last_heartbeat > self.heartrate:
> self.log.debug("Heartbeating the scheduler")
> self.heartbeat()
> last_self_heartbeat_time = timezone.utcnow()
> loop_end_time = time.time()
> loop_duration = loop_end_time - loop_start_time
> self.log.debug(
> "Ran scheduling loop in %.2f seconds",
> loop_duration)
> if not is_unit_test:
> self.log.debug("Sleeping for %.2f seconds",
> self._processor_poll_interval)
> time.sleep(self._processor_poll_interval)
> if self.processor_agent.done:
> self.log.info("Exiting scheduler loop as all files"
> " have been processed {}
> times".format(self.num_runs))
> break
> if loop_duration < 1 and not is_unit_test:
> sleep_length = 1 - loop_duration
> self.log.debug(
> "Sleeping for {0:.2f} seconds to prevent excessive
> logging"
> .format(sleep_length))
>
[jira] [Assigned] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhidong reassigned AIRFLOW-2516:
Assignee: Zhidong
> Deadlock found when trying to update task_instance table
>
>
> Key: AIRFLOW-2516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2516
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
>Affects Versions: 1.8.0
>Reporter: Jeff Liu
>Assignee: Zhidong
>Priority: Major
> Attachments: Screenshot 2019-12-30 at 10.42.52.png,
> image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png,
> jobs.py, jobs_fixed_deadlock_possibly_1.9.py,
> jobs_fixed_deadlock_possibly_1.9.py,
> scheduler_job_fixed_deadlock_possibly_1.10.6.py
>
>
>
>
> {code:java}
> [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask:
> [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue:
> airflow run production_wipeout_wipe_manager.Carat Carat_20180227
> 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py
> [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback
> (most recent call last):
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/bin/airflow", line 27, in
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> args.func(args)
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> pool=args.pool,
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in
> wrapper
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result =
> func(*args, **kwargs)
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in
> _run_raw_task
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> self.handle_failure(e, test_mode, context)
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in
> handle_failure
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> session.merge(self)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1920, in merge
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask:
> _resolve_conflict_map=_resolve_conflict_map)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1974, in _merge
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged =
> self.query(mapper.class_).get(key[1])
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882,
> in get
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident,
> loading.load_on_pk_identity)
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952,
> in _get_impl
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> db_load_fn(self, primary_key_identity)
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247,
> in load_on_pk_i
> dentity
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> q.one()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884,
> in one
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> self.one_or_none()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854,
> in one_or_none
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> list(self)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925,
> in __iter__
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return
> self._execute_and_instances(context)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946,
> in
[GitHub] [airflow] tooptoop4 opened a new pull request #7089: [AIRFLOW-6452] scheduler_job.py - remove excess sleep/log/duration calls
tooptoop4 opened a new pull request #7089: [AIRFLOW-6452] scheduler_job.py - remove excess sleep/log/duration calls URL: https://github.com/apache/airflow/pull/7089 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) - [ ] Description above provides context of the change - [ ] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] tooptoop4 edited a comment on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success
tooptoop4 edited a comment on issue #6954: [AIRFLOW-4355] removed task should
not lead to dagrun success
URL: https://github.com/apache/airflow/pull/6954#issuecomment-571383906
> Can you add some tests?
it is tough to reproduce the scenario.
I tried below but it always says DAGRun is still 'running' rather than
'failed':
`
def test_dagrun_remove_callback(self):
def on_removed_callable(context):
self.assertEqual(
context['dag_run'].dag_id,
'test_dagrun_remove_callback'
)
conf.set("scheduler", "REMOVED_TASKS_LEAD_TO_DAGRUN_FAILURE", "True")
dag = DAG(
dag_id='test_dagrun_remove_callback',
start_date=datetime.datetime(2017, 1, 1),
on_failure_callback=on_removed_callable,
)
dag_task1 = DummyOperator(
task_id='test_state_removed1',
dag=dag)
initial_task_states = {
'test_state_removed1': State.REMOVED
}
dag_run = self.create_dag_run(dag=dag,
state=State.RUNNING,
task_states=initial_task_states)
# dag_run.verify_integrity()
updated_dag_state = dag_run.update_state()
self.assertEqual(State.FAILED, updated_dag_state)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] tooptoop4 commented on issue #6954: [AIRFLOW-4355] removed task should not lead to dagrun success
tooptoop4 commented on issue #6954: [AIRFLOW-4355] removed task should not lead
to dagrun success
URL: https://github.com/apache/airflow/pull/6954#issuecomment-571383906
> Can you add some tests?
it is tough to reproduce the scenario.
I tried below but it always says DAGRun is still 'running' rather than
'failed':
` def test_dagrun_remove_callback(self):
def on_removed_callable(context):
self.assertEqual(
context['dag_run'].dag_id,
'test_dagrun_remove_callback'
)
conf.set("scheduler", "REMOVED_TASKS_LEAD_TO_DAGRUN_FAILURE", "True")
dag = DAG(
dag_id='test_dagrun_remove_callback',
start_date=datetime.datetime(2017, 1, 1),
on_failure_callback=on_removed_callable,
)
dag_task1 = DummyOperator(
task_id='test_state_removed1',
dag=dag)
initial_task_states = {
'test_state_removed1': State.REMOVED
}
dag_run = self.create_dag_run(dag=dag,
state=State.RUNNING,
task_states=initial_task_states)
# dag_run.verify_integrity()
updated_dag_state = dag_run.update_state()
self.assertEqual(State.FAILED, updated_dag_state)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] tooptoop4 commented on a change in pull request #6954: [AIRFLOW-4355] removed task should not lead to dagrun success
tooptoop4 commented on a change in pull request #6954: [AIRFLOW-4355] removed
task should not lead to dagrun success
URL: https://github.com/apache/airflow/pull/6954#discussion_r363549714
##
File path: airflow/models/dagrun.py
##
@@ -311,32 +313,42 @@ def update_state(self, session=None):
leaf_tis = [ti for ti in tis if ti.task_id in {t.task_id for t in
dag.leaves}]
-# if all roots finished and at least one failed, the run failed
-if not unfinished_tasks and any(
-leaf_ti.state in {State.FAILED, State.UPSTREAM_FAILED} for leaf_ti
in leaf_tis
-):
-self.log.info('Marking run %s failed', self)
-self.set_state(State.FAILED)
-dag.handle_callback(self, success=False, reason='task_failure',
-session=session)
-
-# if all leafs succeeded and no unfinished tasks, the run succeeded
-elif not unfinished_tasks and all(
-leaf_ti.state in {State.SUCCESS, State.SKIPPED} for leaf_ti in
leaf_tis
-):
-self.log.info('Marking run %s successful', self)
-self.set_state(State.SUCCESS)
-dag.handle_callback(self, success=True, reason='success',
session=session)
-
-# if *all tasks* are deadlocked, the run failed
-elif (unfinished_tasks and none_depends_on_past and
- none_task_concurrency and no_dependencies_met):
-self.log.info('Deadlock; marking run %s failed', self)
-self.set_state(State.FAILED)
-dag.handle_callback(self, success=False,
reason='all_tasks_deadlocked',
-session=session)
-
-# finally, if the roots aren't done, the dag is still running
+if conf.getboolean('scheduler',
'REMOVED_TASKS_LEAD_TO_DAGRUN_FAILURE', fallback=False):
+# REMOVED state counted as unfinished
+unfinished_tasks = self.get_task_instances(
+state=State.unfinished_or_removed(),
+session=session
+)
Review comment:
updated
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
