[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6898: [AIRFLOW-6432] fix EmrAddStepsOperator broken ref & faulty test & add test case for missing cluster
nuclearpinguin commented on a change in pull request #6898: [AIRFLOW-6432] fix
EmrAddStepsOperator broken ref & faulty test & add test case for missing cluster
URL: https://github.com/apache/airflow/pull/6898#discussion_r362162056
##
File path: tests/contrib/operators/test_emr_add_steps_operator.py
##
@@ -107,23 +108,45 @@ def test_execute_returns_step_id(self):
def test_init_with_cluster_name(self):
expected_job_flow_id = 'j-1231231234'
-self.emr_client_mock.get_cluster_id_by_name.return_value =
expected_job_flow_id
self.emr_client_mock.add_job_flow_steps.return_value =
ADD_STEPS_SUCCESS_RETURN
with patch('boto3.session.Session', self.boto3_session_mock):
+with
patch('airflow.contrib.hooks.emr_hook.EmrHook.get_cluster_id_by_name') \
+as mock_get_cluster_id_by_name:
+mock_get_cluster_id_by_name.return_value = expected_job_flow_id
+
+operator = EmrAddStepsOperator(
+task_id='test_task',
+job_flow_name='test_cluster',
+cluster_states=['RUNNING', 'WAITING'],
+aws_conn_id='aws_default',
+dag=DAG('test_dag_id', default_args=self.args)
+)
+
+operator.execute(self.mock_context)
+
+ti = self.mock_context['ti']
+
+ti.xcom_push.assert_called_once_with(key='job_flow_id',
value=expected_job_flow_id)
+
+def test_init_with_nonexistent_cluster_name(self):
+cluster_name = 'test_cluster'
+
+with
patch('airflow.contrib.hooks.emr_hook.EmrHook.get_cluster_id_by_name') \
+as mock_get_cluster_id_by_name:
+mock_get_cluster_id_by_name.return_value = None
+
operator = EmrAddStepsOperator(
task_id='test_task',
-job_flow_name='test_cluster',
+job_flow_name=cluster_name,
cluster_states=['RUNNING', 'WAITING'],
aws_conn_id='aws_default',
dag=DAG('test_dag_id', default_args=self.args)
)
-operator.execute(self.mock_context)
-
-ti = self.mock_context['ti']
-
-ti.xcom_push.assert_any_call(key='job_flow_id',
value=expected_job_flow_id)
+with self.assertRaises(AirflowException) as error:
+operator.execute(self.mock_context)
+self.assertEqual(str(error.exception), 'No cluster found for name
= test_cluster')
Review comment:
```suggestion
self.assertEqual(str(error.exception), f'No cluster found for
name: {cluster_name}')
```
The test need an adjustment after the change in core :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin merged pull request #6942: [AIRFLOW-XXXX] Fix development packages installtion instructions
nuclearpinguin merged pull request #6942: [AIRFLOW-] Fix development packages installtion instructions URL: https://github.com/apache/airflow/pull/6942 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-6359) spark_submit_hook.py polling too frequent
[
https://issues.apache.org/jira/browse/AIRFLOW-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomasz Urbaszek updated AIRFLOW-6359:
-
Fix Version/s: 1.10.8
> spark_submit_hook.py polling too frequent
> -
>
> Key: AIRFLOW-6359
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6359
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.6
>Reporter: t oo
>Assignee: t oo
>Priority: Major
> Fix For: 1.10.8
>
>
> spark polling every second is too frequent, use config to change to desired
> interval, ie 15 seconds
> contrib/hooks/spark_submit_hook.py Poll spark server at a custom interval
> instead of every second
>
> BEFORE
> Sleep for 1 second as we do not want to spam the cluster
> time.sleep(1)
>
> AFTER
> import airflow
> from airflow import configuration as conf
> Sleep for n second as we do not want to spam the cluster
> _poll_interval = conf.getint('sparksubmit', 'poll_interval')
> time.sleep(_poll_interval)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6359) spark_submit_hook.py polling too frequent
[
https://issues.apache.org/jira/browse/AIRFLOW-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomasz Urbaszek resolved AIRFLOW-6359.
--
Resolution: Fixed
> spark_submit_hook.py polling too frequent
> -
>
> Key: AIRFLOW-6359
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6359
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.6
>Reporter: t oo
>Assignee: t oo
>Priority: Major
>
> spark polling every second is too frequent, use config to change to desired
> interval, ie 15 seconds
> contrib/hooks/spark_submit_hook.py Poll spark server at a custom interval
> instead of every second
>
> BEFORE
> Sleep for 1 second as we do not want to spam the cluster
> time.sleep(1)
>
> AFTER
> import airflow
> from airflow import configuration as conf
> Sleep for n second as we do not want to spam the cluster
> _poll_interval = conf.getint('sparksubmit', 'poll_interval')
> time.sleep(_poll_interval)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6359) spark_submit_hook.py polling too frequent
[
https://issues.apache.org/jira/browse/AIRFLOW-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005961#comment-17005961
]
ASF subversion and git services commented on AIRFLOW-6359:
--
Commit 83ba8e638fb539afd6dac495a8c59012d0aaa9f4 in airflow's branch
refs/heads/master from tooptoop4
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=83ba8e6 ]
[AIRFLOW-6359] spark_submit_hook.py status polling interval config (#6909)
* AIRFLOW-6359
* pep checks
* make travis CI refire
* new section = bad
* move poll_interval into the operator instead of .cfg
* review comments
> spark_submit_hook.py polling too frequent
> -
>
> Key: AIRFLOW-6359
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6359
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.6
>Reporter: t oo
>Assignee: t oo
>Priority: Major
>
> spark polling every second is too frequent, use config to change to desired
> interval, ie 15 seconds
> contrib/hooks/spark_submit_hook.py Poll spark server at a custom interval
> instead of every second
>
> BEFORE
> Sleep for 1 second as we do not want to spam the cluster
> time.sleep(1)
>
> AFTER
> import airflow
> from airflow import configuration as conf
> Sleep for n second as we do not want to spam the cluster
> _poll_interval = conf.getint('sparksubmit', 'poll_interval')
> time.sleep(_poll_interval)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6359) spark_submit_hook.py polling too frequent
[
https://issues.apache.org/jira/browse/AIRFLOW-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005959#comment-17005959
]
ASF GitHub Bot commented on AIRFLOW-6359:
-
nuclearpinguin commented on pull request #6909: [AIRFLOW-6359]
spark_submit_hook.py status polling interval config
URL: https://github.com/apache/airflow/pull/6909
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> spark_submit_hook.py polling too frequent
> -
>
> Key: AIRFLOW-6359
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6359
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.6
>Reporter: t oo
>Assignee: t oo
>Priority: Major
>
> spark polling every second is too frequent, use config to change to desired
> interval, ie 15 seconds
> contrib/hooks/spark_submit_hook.py Poll spark server at a custom interval
> instead of every second
>
> BEFORE
> Sleep for 1 second as we do not want to spam the cluster
> time.sleep(1)
>
> AFTER
> import airflow
> from airflow import configuration as conf
> Sleep for n second as we do not want to spam the cluster
> _poll_interval = conf.getint('sparksubmit', 'poll_interval')
> time.sleep(_poll_interval)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6359) spark_submit_hook.py polling too frequent
[
https://issues.apache.org/jira/browse/AIRFLOW-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005960#comment-17005960
]
ASF subversion and git services commented on AIRFLOW-6359:
--
Commit 83ba8e638fb539afd6dac495a8c59012d0aaa9f4 in airflow's branch
refs/heads/master from tooptoop4
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=83ba8e6 ]
[AIRFLOW-6359] spark_submit_hook.py status polling interval config (#6909)
* AIRFLOW-6359
* pep checks
* make travis CI refire
* new section = bad
* move poll_interval into the operator instead of .cfg
* review comments
> spark_submit_hook.py polling too frequent
> -
>
> Key: AIRFLOW-6359
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6359
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.6
>Reporter: t oo
>Assignee: t oo
>Priority: Major
>
> spark polling every second is too frequent, use config to change to desired
> interval, ie 15 seconds
> contrib/hooks/spark_submit_hook.py Poll spark server at a custom interval
> instead of every second
>
> BEFORE
> Sleep for 1 second as we do not want to spam the cluster
> time.sleep(1)
>
> AFTER
> import airflow
> from airflow import configuration as conf
> Sleep for n second as we do not want to spam the cluster
> _poll_interval = conf.getint('sparksubmit', 'poll_interval')
> time.sleep(_poll_interval)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] nuclearpinguin merged pull request #6909: [AIRFLOW-6359] spark_submit_hook.py status polling interval config
nuclearpinguin merged pull request #6909: [AIRFLOW-6359] spark_submit_hook.py status polling interval config URL: https://github.com/apache/airflow/pull/6909 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6966: [AIRFLOW-6108] [AIP-21] Rename CloudDLP operators
nuclearpinguin commented on a change in pull request #6966: [AIRFLOW-6108] [AIP-21] Rename CloudDLP operators URL: https://github.com/apache/airflow/pull/6966#discussion_r362160399 ## File path: airflow/gcp/operators/dlp.py ## @@ -1032,7 +1032,7 @@ def execute(self, context): ) -class CloudDLPGetJobTripperOperator(BaseOperator): +class CloudDLPGetJobTriggerOperator(BaseOperator): Review comment: ```suggestion class CloudDLPGetDLPJobTriggerOperator(BaseOperator): ``` Let's be consistent. @mik-laj do you think we need to repeat DLP two times in the name? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] TobKed closed pull request #6973: [AIRFLOW-6407] BigQuery hook - add tests for create_external_table me…
TobKed closed pull request #6973: [AIRFLOW-6407] BigQuery hook - add tests for create_external_table me… URL: https://github.com/apache/airflow/pull/6973 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6407) BigQuery hook - add tests for create_external_table method
[ https://issues.apache.org/jira/browse/AIRFLOW-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005952#comment-17005952 ] ASF GitHub Bot commented on AIRFLOW-6407: - TobKed commented on pull request #6973: [AIRFLOW-6407] BigQuery hook - add tests for create_external_table me… URL: https://github.com/apache/airflow/pull/6973 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > BigQuery hook - add tests for create_external_table method > -- > > Key: AIRFLOW-6407 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6407 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, hooks, tests >Affects Versions: 1.10.7 >Reporter: Tobiasz Kedzierski >Assignee: Tobiasz Kedzierski >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6408) BigQuery hook - add tests for patch_table method
[ https://issues.apache.org/jira/browse/AIRFLOW-6408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005951#comment-17005951 ] ASF GitHub Bot commented on AIRFLOW-6408: - TobKed commented on pull request #6972: [AIRFLOW-6408] BigQuery hook - add tests for patch_table method URL: https://github.com/apache/airflow/pull/6972 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > BigQuery hook - add tests for patch_table method > > > Key: AIRFLOW-6408 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6408 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, hooks, tests >Affects Versions: 1.10.7 >Reporter: Tobiasz Kedzierski >Assignee: Tobiasz Kedzierski >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] TobKed commented on issue #6973: [AIRFLOW-6407] BigQuery hook - add tests for create_external_table me…
TobKed commented on issue #6973: [AIRFLOW-6407] BigQuery hook - add tests for create_external_table me… URL: https://github.com/apache/airflow/pull/6973#issuecomment-569874615 @kaxil sure, I will close it for now and create fresh one when finish all tests for `BigQueryBaseCursor` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] TobKed closed pull request #6972: [AIRFLOW-6408] BigQuery hook - add tests for patch_table method
TobKed closed pull request #6972: [AIRFLOW-6408] BigQuery hook - add tests for patch_table method URL: https://github.com/apache/airflow/pull/6972 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] TobKed commented on issue #6972: [AIRFLOW-6408] BigQuery hook - add tests for patch_table method
TobKed commented on issue #6972: [AIRFLOW-6408] BigQuery hook - add tests for patch_table method URL: https://github.com/apache/airflow/pull/6972#issuecomment-569874584 @kaxil sure, I will close it for now and create fresh one when finish all tests for `BigQueryBaseCursor` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6968: [Airflow-6110] [AIP-21] Rename GCP natural language operators
codecov-io edited a comment on issue #6968: [Airflow-6110] [AIP-21] Rename GCP natural language operators URL: https://github.com/apache/airflow/pull/6968#issuecomment-569869985 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=h1) Report > Merging [#6968](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/bd319e8a86fe007ec660764bfa0220611727c4c5?src=pr=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `84%`. [](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=tree) ```diff @@Coverage Diff@@ ## master#6968 +/- ## = - Coverage 84.81% 84.52% -0.3% = Files 679 679 Lines 3848938505 +16 = - Hits3264632547 -99 - Misses 5843 5958+115 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=tree) | Coverage Δ | | |---|---|---| | [...gle/cloud/example\_dags/example\_natural\_language.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX25hdHVyYWxfbGFuZ3VhZ2UucHk=) | `0% <0%> (ø)` | :arrow_up: | | [...oviders/google/cloud/operators/natural\_language.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9uYXR1cmFsX2xhbmd1YWdlLnB5) | `100% <100%> (ø)` | :arrow_up: | | [...contrib/operators/gcp\_natural\_language\_operator.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfbmF0dXJhbF9sYW5ndWFnZV9vcGVyYXRvci5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.43% <0%> (-1.45%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=footer). Last update [bd319e8...cc9cb14](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6968: [Airflow-6110] [AIP-21] Rename GCP natural language operators
codecov-io edited a comment on issue #6968: [Airflow-6110] [AIP-21] Rename GCP natural language operators URL: https://github.com/apache/airflow/pull/6968#issuecomment-569869985 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=h1) Report > Merging [#6968](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/bd319e8a86fe007ec660764bfa0220611727c4c5?src=pr=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `84%`. [](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=tree) ```diff @@Coverage Diff@@ ## master#6968 +/- ## = - Coverage 84.81% 84.52% -0.3% = Files 679 679 Lines 3848938505 +16 = - Hits3264632547 -99 - Misses 5843 5958+115 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=tree) | Coverage Δ | | |---|---|---| | [...gle/cloud/example\_dags/example\_natural\_language.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX25hdHVyYWxfbGFuZ3VhZ2UucHk=) | `0% <0%> (ø)` | :arrow_up: | | [...oviders/google/cloud/operators/natural\_language.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9uYXR1cmFsX2xhbmd1YWdlLnB5) | `100% <100%> (ø)` | :arrow_up: | | [...contrib/operators/gcp\_natural\_language\_operator.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfbmF0dXJhbF9sYW5ndWFnZV9vcGVyYXRvci5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.43% <0%> (-1.45%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=footer). Last update [bd319e8...cc9cb14](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6968: [Airflow-6110] [AIP-21] Rename GCP natural language operators
codecov-io commented on issue #6968: [Airflow-6110] [AIP-21] Rename GCP natural language operators URL: https://github.com/apache/airflow/pull/6968#issuecomment-569869985 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=h1) Report > Merging [#6968](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/bd319e8a86fe007ec660764bfa0220611727c4c5?src=pr=desc) will **decrease** coverage by `0.29%`. > The diff coverage is `84%`. [](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=tree) ```diff @@Coverage Diff@@ ## master#6968 +/- ## = - Coverage 84.81% 84.52% -0.3% = Files 679 679 Lines 3848938505 +16 = - Hits3264632547 -99 - Misses 5843 5958+115 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=tree) | Coverage Δ | | |---|---|---| | [...gle/cloud/example\_dags/example\_natural\_language.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2V4YW1wbGVfZGFncy9leGFtcGxlX25hdHVyYWxfbGFuZ3VhZ2UucHk=) | `0% <0%> (ø)` | :arrow_up: | | [...oviders/google/cloud/operators/natural\_language.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9uYXR1cmFsX2xhbmd1YWdlLnB5) | `100% <100%> (ø)` | :arrow_up: | | [...contrib/operators/gcp\_natural\_language\_operator.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfbmF0dXJhbF9sYW5ndWFnZV9vcGVyYXRvci5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6968/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.43% <0%> (-1.45%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=footer). Last update [bd319e8...cc9cb14](https://codecov.io/gh/apache/airflow/pull/6968?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005910#comment-17005910
]
zorro.shi commented on AIRFLOW-2516:
Hi [~higrys] I had a similar issue happening at 1.10.6. The following is a
deadlock log that may help you
LATEST DETECTED DEADLOCK
2019-12-31 01:00:35 0x7fa024527700
*** (1) TRANSACTION:
TRANSACTION 12953238, ACTIVE 0 sec starting index read
mysql tables in use 3, locked 3
LOCK WAIT 16 lock struct(s), heap size 1136, 26 row lock(s)
MySQL thread id 4441025, OS thread handle 140324389902080, query id 132956673
iZwz9ds6obifnanjsnb2gqZ 172.18.38.223 airflow Sending data
UPDATE task_instance, (SELECT task_instance.try_number AS try_number,
task_instance.task_id AS task_id, task_instance.dag_id AS dag_id,
task_instance.execution_date AS execution_date, task_instance.start_date AS
start_date, task_instance.end_date AS end_date, task_instance.duration AS
duration, task_instance.state AS state, task_instance.max_tries AS max_tries,
task_instance.hostname AS hostname, task_instance.unixname AS unixname,
task_instance.job_id AS job_id, task_instance.pool AS pool, task_instance.queue
AS queue, task_instance.priority_weight AS priority_weight,
task_instance.operator AS operator, task_instance.queued_dttm AS queued_dttm,
task_instance.pid AS pid, task_instance.executor_config AS executor_config
FROM task_instance LEFT OUTER JOIN dag_run ON task_instance.dag_id =
dag_run.dag_id AND task_instance.execution_date = dag_run.execution_date
WHERE task_in
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 122 page no 39 n bits 160 index PRIMARY of table
`airflow`.`task_instance` trx id 12953238 lock_mode X locks rec but not gap
waiting
Record lock, heap no 88 PHYSICAL RECORD: n_fields 21; compact format; info bits 0
0: len 24; hex 776d735f6e6b635f7072696e74696e675f686973746f7279; asc
wms_nkc_printing_history;;
1: len 19; hex 776d735f617263686976652e776d735f6e6b63; asc
wms_archive.wms_nkc;;
2: len 7; hex 5e08db9000; asc ^ ;;
3: len 6; hex 00c5a694; asc ;;
4: len 7; hex 3b01870928; asc ; (;;
5: len 7; hex 5e0a2d22050d7b; asc ^ -" {;;
6: len 7; hex 5e0a2d22050d88; asc ^ -" ;;
7: SQL NULL;
8: len 6; hex 717565756564; asc queued;;
9: len 4; hex 8000; asc ;;
10: len 0; hex ; asc ;;
11: len 4; hex 726f6f74; asc root;;
12: SQL NULL;
13: len 12; hex 64656661756c745f706f6f6c; asc default_pool;;
14: len 7; hex 64656661756c74; asc default;;
15: len 4; hex 8002; asc ;;
16: SQL NULL;
17: len 7; hex 5e0a2d33061021; asc ^ -3 !;;
18: SQL NULL;
19: len 4; hex 8000; asc ;;
20: len 5; hex 80057d942e; asc } .;;
*** (2) TRANSACTION:
TRANSACTION 12953236, ACTIVE 0 sec updating or deleting, thread declared inside
InnoDB 4999
mysql tables in use 1, locked 1
3 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 1
MySQL thread id 4445271, OS thread handle 140325780879104, query id 132956674
iZwz9ds6obifnanjsnb2gqZ 172.18.38.223 airflow updating
UPDATE task_instance SET state='queued', queued_dttm='2019-12-30
17:00:35.397345' WHERE task_instance.task_id = 'wms_nkc_printing_history' AND
task_instance.dag_id = 'wms_archive.wms_nkc' AND task_instance.execution_date =
'2019-12-29 17:00:00'
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 122 page no 39 n bits 160 index PRIMARY of table
`airflow`.`task_instance` trx id 12953236 lock_mode X locks rec but not gap
Record lock, heap no 88 PHYSICAL RECORD: n_fields 21; compact format; info bits 0
0: len 24; hex 776d735f6e6b635f7072696e74696e675f686973746f7279; asc
wms_nkc_printing_history;;
1: len 19; hex 776d735f617263686976652e776d735f6e6b63; asc
wms_archive.wms_nkc;;
2: len 7; hex 5e08db9000; asc ^ ;;
3: len 6; hex 00c5a694; asc ;;
4: len 7; hex 3b01870928; asc ; (;;
5: len 7; hex 5e0a2d22050d7b; asc ^ -" {;;
6: len 7; hex 5e0a2d22050d88; asc ^ -" ;;
7: SQL NULL;
8: len 6; hex 717565756564; asc queued;;
9: len 4; hex 8000; asc ;;
10: len 0; hex ; asc ;;
11: len 4; hex 726f6f74; asc root;;
12: SQL NULL;
13: len 12; hex 64656661756c745f706f6f6c; asc default_pool;;
14: len 7; hex 64656661756c74; asc default;;
15: len 4; hex 8002; asc ;;
16: SQL NULL;
17: len 7; hex 5e0a2d33061021; asc ^ -3 !;;
18: SQL NULL;
19: len 4; hex 8000; asc ;;
20: len 5; hex 80057d942e; asc } .;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 122 page no 17 n bits 320 index ti_dag_state of table
`airflow`.`task_instance` trx id 12953236 lock_mode X locks rec but not gap
waiting
Record lock, heap no 235 PHYSICAL RECORD: n_fields 4; compact format; info bits 0
0: len 19; hex 776d735f617263686976652e776d735f6e6b63; asc
wms_archive.wms_nkc;;
1: len 9; hex 7363686564756c6564; asc scheduled;;
2: len 24; hex 776d735f6e6b635f7072696e74696e675f686973746f7279; asc
wms_nkc_printing_history;;
3: len
[jira] [Comment Edited] (AIRFLOW-3819) k8s executor - Allow the configuration of a global default for pod resource request/limits
[
https://issues.apache.org/jira/browse/AIRFLOW-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005898#comment-17005898
]
Lihan Li edited comment on AIRFLOW-3819 at 12/31/19 2:38 AM:
-
Also just hit by this issue.
I think this better resolved within Airflow, it is a very small addition; Also,
Mesos has task_cpu within Airflow.
[~aizhamal]
was (Author: lihan):
I think this better resolved within Airflow, it is a very small addition; Also,
Mesos has task_cpu within Airflow.
> k8s executor - Allow the configuration of a global default for pod resource
> request/limits
> ---
>
> Key: AIRFLOW-3819
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3819
> Project: Apache Airflow
> Issue Type: Improvement
> Components: executors
>Affects Versions: 1.10.2
>Reporter: afusr
>Priority: Minor
> Labels: kubernetes
>
> Currently the kubernetes executor allows you to specify pod resources
> requests and limits (cpu and memory). For example:
> {noformat}
> # Limit resources on this operator/task with node affinity & tolerations
> three_task = PythonOperator(
> task_id="three_task", python_callable=print_stuff, dag=dag,
> executor_config={
> "KubernetesExecutor": {"request_memory": "128Mi",
>"limit_memory": "128Mi",
>"tolerations": tolerations,
>"affinity": affinity}}
> )
> {noformat}
> These values are used by kubernetes when making scaling decisions. It would
> be nice to be able to specify a global default for these values, to ensure
> that each pod airflow creates has a value specified for these properties.
> There is still the requirement to override these values on a dag by dag
> basis.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-3819) k8s executor - Allow the configuration of a global default for pod resource request/limits
[
https://issues.apache.org/jira/browse/AIRFLOW-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005898#comment-17005898
]
Lihan Li commented on AIRFLOW-3819:
---
I think this better resolved within Airflow, it is a very small addition; Also,
Mesos has task_cpu within Airflow.
> k8s executor - Allow the configuration of a global default for pod resource
> request/limits
> ---
>
> Key: AIRFLOW-3819
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3819
> Project: Apache Airflow
> Issue Type: Improvement
> Components: executors
>Affects Versions: 1.10.2
>Reporter: afusr
>Priority: Minor
> Labels: kubernetes
>
> Currently the kubernetes executor allows you to specify pod resources
> requests and limits (cpu and memory). For example:
> {noformat}
> # Limit resources on this operator/task with node affinity & tolerations
> three_task = PythonOperator(
> task_id="three_task", python_callable=print_stuff, dag=dag,
> executor_config={
> "KubernetesExecutor": {"request_memory": "128Mi",
>"limit_memory": "128Mi",
>"tolerations": tolerations,
>"affinity": affinity}}
> )
> {noformat}
> These values are used by kubernetes when making scaling decisions. It would
> be nice to be able to specify a global default for these values, to ensure
> that each pod airflow creates has a value specified for these properties.
> There is still the requirement to override these values on a dag by dag
> basis.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] lihan commented on issue #4660: [AIRFLOW-3819] - Allow the configuration of a global default for work…
lihan commented on issue #4660: [AIRFLOW-3819] - Allow the configuration of a global default for work… URL: https://github.com/apache/airflow/pull/4660#issuecomment-569850777 Same problem here, pretty surprised this has to solved with Kubernetes. Can we add the features to Airflow? Also, this same functionality exists in Mesos Configuration within Airflow. @atrbgithub This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005871#comment-17005871
]
Zhidong commented on AIRFLOW-2516:
--
Thanks a lot [~higrys]! Those excerpts are from [Locks Set by Different SQL
Statements in
InnoDB|https://dev.mysql.com/doc/refman/8.0/en/innodb-locks-set.html].
> Deadlock found when trying to update task_instance table
>
>
> Key: AIRFLOW-2516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2516
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
>Affects Versions: 1.8.0
>Reporter: Jeff Liu
>Priority: Major
> Attachments: Screenshot 2019-12-30 at 10.42.52.png,
> image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png, jobs.py
>
>
>
>
> {code:java}
> [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask:
> [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue:
> airflow run production_wipeout_wipe_manager.Carat Carat_20180227
> 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py
> [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback
> (most recent call last):
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/bin/airflow", line 27, in
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> args.func(args)
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> pool=args.pool,
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in
> wrapper
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result =
> func(*args, **kwargs)
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in
> _run_raw_task
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> self.handle_failure(e, test_mode, context)
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in
> handle_failure
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> session.merge(self)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1920, in merge
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask:
> _resolve_conflict_map=_resolve_conflict_map)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1974, in _merge
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged =
> self.query(mapper.class_).get(key[1])
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882,
> in get
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident,
> loading.load_on_pk_identity)
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952,
> in _get_impl
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> db_load_fn(self, primary_key_identity)
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247,
> in load_on_pk_i
> dentity
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> q.one()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884,
> in one
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> self.one_or_none()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854,
> in one_or_none
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> list(self)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925,
> in __iter__
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return
> self._execute_and_instances(context)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946,
> in
[GitHub] [airflow] mikolaje commented on issue #6795: Adjust the MASTER_URL of spark-submit in SparkSubmitHook
mikolaje commented on issue #6795: Adjust the MASTER_URL of spark-submit in SparkSubmitHook URL: https://github.com/apache/airflow/pull/6795#issuecomment-569843149 @albertusk95 yes, do I have to open a new PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil merged pull request #6974: [AIRFLOW-XXXX] Add Documentation for check_slas flag
kaxil merged pull request #6974: [AIRFLOW-] Add Documentation for check_slas flag URL: https://github.com/apache/airflow/pull/6974 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #6974: [AIRFLOW-XXXX] Add Documentation for check_slas flag
kaxil commented on a change in pull request #6974: [AIRFLOW-] Add Documentation for check_slas flag URL: https://github.com/apache/airflow/pull/6974#discussion_r362116083 ## File path: docs/concepts.rst ## @@ -805,6 +805,14 @@ In addition to sending alerts to the addresses specified in a task's ``email`` p the ``sla_miss_callback`` specifies an additional ``Callable`` object to be invoked when the SLA is not met. +If you don't want to check SLAs, you can disable globally (all the DAGs) by +setting ``check_slas=False`` under ``[core]`` section in ``airflow.cfg`` file: + +```ini Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6974: [AIRFLOW-XXXX] Add Documentation for check_slas flag
mik-laj commented on a change in pull request #6974: [AIRFLOW-] Add Documentation for check_slas flag URL: https://github.com/apache/airflow/pull/6974#discussion_r362115294 ## File path: docs/concepts.rst ## @@ -805,6 +805,14 @@ In addition to sending alerts to the addresses specified in a task's ``email`` p the ``sla_miss_callback`` specifies an additional ``Callable`` object to be invoked when the SLA is not met. +If you don't want to check SLAs, you can disable globally (all the DAGs) by +setting ``check_slas=False`` under ``[core]`` section in ``airflow.cfg`` file: + +```ini Review comment: We use restructure(.rst) markup. It's markdown (.md). Can you use `code-block` directive? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6974: [AIRFLOW-XXXX] Add Documentation for check_slas flag
mik-laj commented on a change in pull request #6974: [AIRFLOW-] Add Documentation for check_slas flag URL: https://github.com/apache/airflow/pull/6974#discussion_r362115294 ## File path: docs/concepts.rst ## @@ -805,6 +805,14 @@ In addition to sending alerts to the addresses specified in a task's ``email`` p the ``sla_miss_callback`` specifies an additional ``Callable`` object to be invoked when the SLA is not met. +If you don't want to check SLAs, you can disable globally (all the DAGs) by +setting ``check_slas=False`` under ``[core]`` section in ``airflow.cfg`` file: + +```ini Review comment: We use restructure markup. It's markdown. Can you use `code-block` directive? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6923: [AIRFLOW-5149] - skip SLA checks config
mik-laj commented on a change in pull request #6923: [AIRFLOW-5149] - skip SLA checks config URL: https://github.com/apache/airflow/pull/6923#discussion_r362115161 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -176,6 +176,9 @@ store_serialized_dags = False # Updating serialized DAG can not be faster than a minimum interval to reduce database write rate. min_serialized_dag_update_interval = 30 +# On each dagrun check against defined SLAs Review comment: @kaxil Fantastic! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #6923: [AIRFLOW-5149] - skip SLA checks config
kaxil commented on a change in pull request #6923: [AIRFLOW-5149] - skip SLA checks config URL: https://github.com/apache/airflow/pull/6923#discussion_r362113797 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -176,6 +176,9 @@ store_serialized_dags = False # Updating serialized DAG can not be faster than a minimum interval to reduce database write rate. min_serialized_dag_update_interval = 30 +# On each dagrun check against defined SLAs Review comment: Created at https://github.com/apache/airflow/pull/6974 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil opened a new pull request #6974: [AIRFLOW-XXXX] Add Documentation for check_slas flag
kaxil opened a new pull request #6974: [AIRFLOW-] Add Documentation for check_slas flag URL: https://github.com/apache/airflow/pull/6974 Based on discussion in https://github.com/apache/airflow/pull/6923 - [x] Description above provides context of the change - [ ] Commit message contains [\[AIRFLOW-\]](https://issues.apache.org/jira/browse/AIRFLOW-) or `[AIRFLOW-]` for document-only changes - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #6972: [AIRFLOW-6408] BigQuery hook - add tests for patch_table method
kaxil commented on issue #6972: [AIRFLOW-6408] BigQuery hook - add tests for patch_table method URL: https://github.com/apache/airflow/pull/6972#issuecomment-569815962 Can you create a single PR with all the BigQueryHook tests, please? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #6973: [AIRFLOW-6407] BigQuery hook - add tests for create_external_table me…
kaxil commented on issue #6973: [AIRFLOW-6407] BigQuery hook - add tests for create_external_table me… URL: https://github.com/apache/airflow/pull/6973#issuecomment-569815921 Can you create a single PR with all the BigQueryHook tests, please? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6912: [AIRFLOW-6352] security - ui - add login timeout
mik-laj commented on a change in pull request #6912: [AIRFLOW-6352] security - ui - add login timeout URL: https://github.com/apache/airflow/pull/6912#discussion_r362112340 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -370,10 +370,13 @@ default_wrap = False # on webserver startup update_fab_perms = True +# Minutes of non-activity before logged out from UI +# 0 means never get forcibly logged out Review comment: Related discussion: https://github.com/apache/airflow/pull/6952#discussion_r362102402 I think, we can add short description in 'security.rst' file or a new file in `howto` directory. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6952: [AIRFLOW-4445] mushroom cloud errors too verbose
mik-laj commented on a change in pull request #6952: [AIRFLOW-4445] mushroom cloud errors too verbose URL: https://github.com/apache/airflow/pull/6952#discussion_r362111591 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -324,6 +324,12 @@ error_logfile = - # Expose the configuration file in the web server expose_config = False +# Expose hostname in the web server +expose_hostname = True Review comment: A similar situation took place with the guides for operators. Initially, we only had one file that contained a description of all operators. However, at some point, we refactored the documentation to create a new quality. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6952: [AIRFLOW-4445] mushroom cloud errors too verbose
mik-laj commented on a change in pull request #6952: [AIRFLOW-4445] mushroom cloud errors too verbose URL: https://github.com/apache/airflow/pull/6952#discussion_r362111344 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -324,6 +324,12 @@ error_logfile = - # Expose the configuration file in the web server expose_config = False +# Expose hostname in the web server +expose_hostname = True Review comment: I answered most of your questions in a different thread.: https://github.com/apache/airflow/pull/6923#discussion_r362110552 I think we can add new section in `security.rst`file. This file is long and suitable for refactoring, but I think we should do it in a separate PR. Please note that refactoring documentation, which is even a collection of random guides, is simpler than writing all documentation from scratch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6923: [AIRFLOW-5149] - skip SLA checks config
mik-laj commented on a change in pull request #6923: [AIRFLOW-5149] - skip SLA checks config URL: https://github.com/apache/airflow/pull/6923#discussion_r362110552 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -176,6 +176,9 @@ store_serialized_dags = False # Updating serialized DAG can not be faster than a minimum interval to reduce database write rate. min_serialized_dag_update_interval = 30 +# On each dagrun check against defined SLAs Review comment: > and where all the previous options are documented? We have documentation gaps, but we try to develop documentation all the time. There is no precise documentation for the SLA mechanism, but this does not mean that it is impossible to write a few sentences and put them in the documentation. Another person who wants to describe the wider documentation will have less work. I think that these two sentences describing this parameter may be placed in a new document in the howto directory. I will give an example of documentation that describes the configuration parameter. https://github.com/apache/airflow/pull/5850/files This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6923: [AIRFLOW-5149] - skip SLA checks config
mik-laj commented on a change in pull request #6923: [AIRFLOW-5149] - skip SLA checks config URL: https://github.com/apache/airflow/pull/6923#discussion_r362110552 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -176,6 +176,9 @@ store_serialized_dags = False # Updating serialized DAG can not be faster than a minimum interval to reduce database write rate. min_serialized_dag_update_interval = 30 +# On each dagrun check against defined SLAs Review comment: > and where all the previous options are documented? We have documentation gaps, but we try to develop documentation all the time. There is no precise documentation for the SLA mechanism, but this does not mean that it is impossible to write a few sentences and put them in the documentation. Another person who wants to describe the wider documentation will have less work. I think that these two sentences describing this parameter may be placed in a new document in the howto directory. I will give an example of documentation that describes the configuration parameter. https://github.com/apache/airflow/pull/5850/files This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] tooptoop4 commented on issue #6909: [AIRFLOW-6359] spark_submit_hook.py status polling interval config
tooptoop4 commented on issue #6909: [AIRFLOW-6359] spark_submit_hook.py status polling interval config URL: https://github.com/apache/airflow/pull/6909#issuecomment-569813419 @nuclearpinguin can u pls merge? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #6912: [AIRFLOW-6352] security - ui - add login timeout
potiuk commented on a change in pull request #6912: [AIRFLOW-6352] security - ui - add login timeout URL: https://github.com/apache/airflow/pull/6912#discussion_r362108831 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -370,10 +370,13 @@ default_wrap = False # on webserver startup update_fab_perms = True +# Minutes of non-activity before logged out from UI +# 0 means never get forcibly logged out Review comment: @mik-laj - > where do you think the documentation should be added ? Maybe you can provide some pointers? Most of the config options are documented in comments of the default_airlfow.cfg. Any other specific place you think this documentation should be added (and where all the previous options are documented?) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #6952: [AIRFLOW-4445] mushroom cloud errors too verbose
potiuk commented on a change in pull request #6952: [AIRFLOW-4445] mushroom cloud errors too verbose URL: https://github.com/apache/airflow/pull/6952#discussion_r362108536 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -324,6 +324,12 @@ error_logfile = - # Expose the configuration file in the web server expose_config = False +# Expose hostname in the web server +expose_hostname = True Review comment: @mik-laj - > where do you think the documentation should be added ? Maybe you can provide some pointers? Most of the config options are documented in comments of the default_airlfow.cfg. Any other specific place you think this documentation should be added (and where all the previous options are documented?) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #6923: [AIRFLOW-5149] - skip SLA checks config
potiuk commented on a change in pull request #6923: [AIRFLOW-5149] - skip SLA checks config URL: https://github.com/apache/airflow/pull/6923#discussion_r362108489 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -176,6 +176,9 @@ store_serialized_dags = False # Updating serialized DAG can not be faster than a minimum interval to reduce database write rate. min_serialized_dag_update_interval = 30 +# On each dagrun check against defined SLAs Review comment: @mik-laj - > where do you think the documentation should be added ? Maybe you can provide some pointers? Most of the config options are documented in comments of the default_airlfow.cfg. Any other specific place you think this documentation should be added (and where all the previous options are documented?) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3225) Display a user customized airflow-webserver-name string on the top banner.
[ https://issues.apache.org/jira/browse/AIRFLOW-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005835#comment-17005835 ] Kamil Bregula commented on AIRFLOW-3225: Are you still working on this change? > Display a user customized airflow-webserver-name string on the top banner. > --- > > Key: AIRFLOW-3225 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3225 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Affects Versions: 2.0.0 >Reporter: lufeng >Assignee: lufeng >Priority: Minor > > Airflow webUI doesn't provide a convenient way for users to name their > Airflow cluster, we propose to introduce the following webserver related > config ([webserver]web_server_name = FOO) and display the web_server_name on > the top right banner next to clock. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-1396) Transfer data from BigQuery to MySQL operator
[
https://issues.apache.org/jira/browse/AIRFLOW-1396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005834#comment-17005834
]
Kamil Bregula commented on AIRFLOW-1396:
We have BigQueryToMySqlOperator operators. Is it looks good for you?
> Transfer data from BigQuery to MySQL operator
> -
>
> Key: AIRFLOW-1396
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1396
> Project: Apache Airflow
> Issue Type: New Feature
> Components: contrib, gcp, operators
>Reporter: Yu Ishikawa
>Priority: Major
>
> We should have an operator to transfer queried data from bigquery to mysql
> like {{HiveToMySqlTransfer}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5462) Google Authentication redirection page crashes with KeyError: 'login'
[
https://issues.apache.org/jira/browse/AIRFLOW-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005833#comment-17005833
]
Kamil Bregula commented on AIRFLOW-5462:
This ticket was on the exclusion list.
https://issues.apache.org/jira/browse/AIRFLOW-1607?
> Google Authentication redirection page crashes with KeyError: 'login'
> -
>
> Key: AIRFLOW-5462
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5462
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication
>Affects Versions: 1.10.3, 1.10.4, 1.10.5
>Reporter: Qian Yu
>Priority: Minor
> Fix For: 2.0.0
>
>
> When user is logged out and try to access the URL of a DAG's TreeView, he's
> redirected to login page. However, after he logs in with google
> authentication, the page redirects to the oauth-authorized/login page and
> crashes with a mushroom cloud.
>
> Expected:
> Redirects back to the original URL the user wanted to access
> Actual:
> Crash with mushroom cloud
>
> Configuration:
> {code:python}
> [webserver]
> authenticate = True
> auth_backend = airflow.contrib.auth.backends.google_auth
> [google]
> client_id = google_client_id
> client_secret = google_client_secret
> oauth_callback_route = /oauth2callback
> domain = "example1.com,example2.com"
> {code}
>
> The issue looks similar to the one described here:
> [https://github.com/apache/incubator-superset/issues/7739]
> {code:python}
> / ( () ) \___
> /( ( ( ) _)) ) )\
>(( ( )() ) ( ) )
> ((/ ( _( ) ( _) ) ( () ) )
> ( ( ( (_) ((( ) .((_ ) . )_
>( ( )( ( )) ) . ) ( )
> ( ( ( ( ) ( _ ( _) ). ) . ) ) ( )
> ( ( ( ) ( ) ( )) ) _)( ) ) )
> ( ( ( \ ) ((_ ( ) ( ) ) ) ) )) ( )
> ( ( ( ( (_ ( ) ( _) ) ( ) ) )
> ( ( ( ( ( ) (_ ) ) ) _) ) _( ( )
> (( ( )(( _) _) _(_ ( (_ )
>(_((__(_(__(( ( ( | ) ) ) )_))__))_)___)
>((__)\\||lll|l||/// \_))
> ( /(/ ( ) ) )\ )
> (( ( ( | | ) ) )\ )
>( /(| / ( )) ) ) )) )
> ( ( _(|)_) )
> ( ||\(|(|)|/|| )
> (|(||(||))
> ( //|/l|||)|\\ \ )
> (/ / // /|//\\ \ \ \ _)
> ---
> Node: ...
> ---
> Traceback (most recent call last):
> File "/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
> response = self.full_dispatch_request()
> File "/lib/python3.6/site-packages/flask/app.py", line 1951, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/lib/python3.6/site-packages/flask/app.py", line 1820, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
> raise value
> File "/lib/python3.6/site-packages/flask/app.py", line 1949, in
> full_dispatch_request
> rv = self.dispatch_request()
> File "/lib/python3.6/site-packages/flask/app.py", line 1935, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/lib/python3.6/site-packages/flask_appbuilder/security/views.py",
> line 633, in oauth_authorized
> resp = self.appbuilder.sm.oauth_remotes[provider].authorized_response()
> KeyError: 'login'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5462) Google Authentication redirection page crashes with KeyError: 'login'
[
https://issues.apache.org/jira/browse/AIRFLOW-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005832#comment-17005832
]
Kamil Bregula commented on AIRFLOW-5462:
This does not cause problems for our client. We can fix it, but there is a
small chance that it would be used in Cloud Composer. In previous work (fixing
JIRA) we have intentionally excluded OAuth tickets as not related to our client.
> Google Authentication redirection page crashes with KeyError: 'login'
> -
>
> Key: AIRFLOW-5462
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5462
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication
>Affects Versions: 1.10.3, 1.10.4, 1.10.5
>Reporter: Qian Yu
>Priority: Minor
> Fix For: 2.0.0
>
>
> When user is logged out and try to access the URL of a DAG's TreeView, he's
> redirected to login page. However, after he logs in with google
> authentication, the page redirects to the oauth-authorized/login page and
> crashes with a mushroom cloud.
>
> Expected:
> Redirects back to the original URL the user wanted to access
> Actual:
> Crash with mushroom cloud
>
> Configuration:
> {code:python}
> [webserver]
> authenticate = True
> auth_backend = airflow.contrib.auth.backends.google_auth
> [google]
> client_id = google_client_id
> client_secret = google_client_secret
> oauth_callback_route = /oauth2callback
> domain = "example1.com,example2.com"
> {code}
>
> The issue looks similar to the one described here:
> [https://github.com/apache/incubator-superset/issues/7739]
> {code:python}
> / ( () ) \___
> /( ( ( ) _)) ) )\
>(( ( )() ) ( ) )
> ((/ ( _( ) ( _) ) ( () ) )
> ( ( ( (_) ((( ) .((_ ) . )_
>( ( )( ( )) ) . ) ( )
> ( ( ( ( ) ( _ ( _) ). ) . ) ) ( )
> ( ( ( ) ( ) ( )) ) _)( ) ) )
> ( ( ( \ ) ((_ ( ) ( ) ) ) ) )) ( )
> ( ( ( ( (_ ( ) ( _) ) ( ) ) )
> ( ( ( ( ( ) (_ ) ) ) _) ) _( ( )
> (( ( )(( _) _) _(_ ( (_ )
>(_((__(_(__(( ( ( | ) ) ) )_))__))_)___)
>((__)\\||lll|l||/// \_))
> ( /(/ ( ) ) )\ )
> (( ( ( | | ) ) )\ )
>( /(| / ( )) ) ) )) )
> ( ( _(|)_) )
> ( ||\(|(|)|/|| )
> (|(||(||))
> ( //|/l|||)|\\ \ )
> (/ / // /|//\\ \ \ \ _)
> ---
> Node: ...
> ---
> Traceback (most recent call last):
> File "/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
> response = self.full_dispatch_request()
> File "/lib/python3.6/site-packages/flask/app.py", line 1951, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/lib/python3.6/site-packages/flask/app.py", line 1820, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
> raise value
> File "/lib/python3.6/site-packages/flask/app.py", line 1949, in
> full_dispatch_request
> rv = self.dispatch_request()
> File "/lib/python3.6/site-packages/flask/app.py", line 1935, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/lib/python3.6/site-packages/flask_appbuilder/security/views.py",
> line 633, in oauth_authorized
> resp = self.appbuilder.sm.oauth_remotes[provider].authorized_response()
> KeyError: 'login'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mik-laj commented on a change in pull request #6952: [AIRFLOW-4445] mushroom cloud errors too verbose
mik-laj commented on a change in pull request #6952: [AIRFLOW-4445] mushroom cloud errors too verbose URL: https://github.com/apache/airflow/pull/6952#discussion_r362102402 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -324,6 +324,12 @@ error_logfile = - # Expose the configuration file in the web server expose_config = False +# Expose hostname in the web server +expose_hostname = True Review comment: Can you add some documentation about this option? If there is no information about this feature in the documentation, very few people will be able to use it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6923: [AIRFLOW-5149] - skip SLA checks config
mik-laj commented on a change in pull request #6923: [AIRFLOW-5149] - skip SLA checks config URL: https://github.com/apache/airflow/pull/6923#discussion_r362102159 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -176,6 +176,9 @@ store_serialized_dags = False # Updating serialized DAG can not be faster than a minimum interval to reduce database write rate. min_serialized_dag_update_interval = 30 +# On each dagrun check against defined SLAs Review comment: Can you add some documentation about this option? If there is no information about this feature in the documentation, very few people will be able to use it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6912: [AIRFLOW-6352] security - ui - add login timeout
mik-laj commented on a change in pull request #6912: [AIRFLOW-6352] security - ui - add login timeout URL: https://github.com/apache/airflow/pull/6912#discussion_r362102039 ## File path: airflow/config_templates/default_airflow.cfg ## @@ -370,10 +370,13 @@ default_wrap = False # on webserver startup update_fab_perms = True +# Minutes of non-activity before logged out from UI +# 0 means never get forcibly logged out Review comment: Can you add some documentation about this option? If there is no information about this feature in the documentation, very few people will be able to use it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6763: [AIRFLOW-6205] The 'airflow db shell' works on mysql without port/passwd
potiuk commented on issue #6763: [AIRFLOW-6205] The 'airflow db shell' works on mysql without port/passwd URL: https://github.com/apache/airflow/pull/6763#issuecomment-569801893 Not really . Have another, more important stuff to work on. Care to take over? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5462) Google Authentication redirection page crashes with KeyError: 'login'
[
https://issues.apache.org/jira/browse/AIRFLOW-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005816#comment-17005816
]
Jarek Potiuk commented on AIRFLOW-5462:
---
[~kamil.bregula] [~turbaszek] - should we look at this?
> Google Authentication redirection page crashes with KeyError: 'login'
> -
>
> Key: AIRFLOW-5462
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5462
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication
>Affects Versions: 1.10.3, 1.10.4, 1.10.5
>Reporter: Qian Yu
>Priority: Minor
> Fix For: 2.0.0
>
>
> When user is logged out and try to access the URL of a DAG's TreeView, he's
> redirected to login page. However, after he logs in with google
> authentication, the page redirects to the oauth-authorized/login page and
> crashes with a mushroom cloud.
>
> Expected:
> Redirects back to the original URL the user wanted to access
> Actual:
> Crash with mushroom cloud
>
> Configuration:
> {code:python}
> [webserver]
> authenticate = True
> auth_backend = airflow.contrib.auth.backends.google_auth
> [google]
> client_id = google_client_id
> client_secret = google_client_secret
> oauth_callback_route = /oauth2callback
> domain = "example1.com,example2.com"
> {code}
>
> The issue looks similar to the one described here:
> [https://github.com/apache/incubator-superset/issues/7739]
> {code:python}
> / ( () ) \___
> /( ( ( ) _)) ) )\
>(( ( )() ) ( ) )
> ((/ ( _( ) ( _) ) ( () ) )
> ( ( ( (_) ((( ) .((_ ) . )_
>( ( )( ( )) ) . ) ( )
> ( ( ( ( ) ( _ ( _) ). ) . ) ) ( )
> ( ( ( ) ( ) ( )) ) _)( ) ) )
> ( ( ( \ ) ((_ ( ) ( ) ) ) ) )) ( )
> ( ( ( ( (_ ( ) ( _) ) ( ) ) )
> ( ( ( ( ( ) (_ ) ) ) _) ) _( ( )
> (( ( )(( _) _) _(_ ( (_ )
>(_((__(_(__(( ( ( | ) ) ) )_))__))_)___)
>((__)\\||lll|l||/// \_))
> ( /(/ ( ) ) )\ )
> (( ( ( | | ) ) )\ )
>( /(| / ( )) ) ) )) )
> ( ( _(|)_) )
> ( ||\(|(|)|/|| )
> (|(||(||))
> ( //|/l|||)|\\ \ )
> (/ / // /|//\\ \ \ \ _)
> ---
> Node: ...
> ---
> Traceback (most recent call last):
> File "/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
> response = self.full_dispatch_request()
> File "/lib/python3.6/site-packages/flask/app.py", line 1951, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/lib/python3.6/site-packages/flask/app.py", line 1820, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
> raise value
> File "/lib/python3.6/site-packages/flask/app.py", line 1949, in
> full_dispatch_request
> rv = self.dispatch_request()
> File "/lib/python3.6/site-packages/flask/app.py", line 1935, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/lib/python3.6/site-packages/flask_appbuilder/security/views.py",
> line 633, in oauth_authorized
> resp = self.appbuilder.sm.oauth_remotes[provider].authorized_response()
> KeyError: 'login'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mik-laj commented on issue #6763: [AIRFLOW-6205] The 'airflow db shell' works on mysql without port/passwd
mik-laj commented on issue #6763: [AIRFLOW-6205] The 'airflow db shell' works on mysql without port/passwd URL: https://github.com/apache/airflow/pull/6763#issuecomment-569799020 @potiuk Any progress on it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6905: [AIRFLOW-6361] Run LocalTaskJob directly in Celery task
mik-laj commented on issue #6905: [AIRFLOW-6361] Run LocalTaskJob directly in Celery task URL: https://github.com/apache/airflow/pull/6905#issuecomment-569797159 @ashb I added note about plugins in `celery.rst` file. Although I will try to remove this limitation in the future. It doesn't seem difficult. The next step is to decouple the executor from the CLI. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6409) BigQuery hook - add tests for run_query method
[ https://issues.apache.org/jira/browse/AIRFLOW-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005810#comment-17005810 ] ASF subversion and git services commented on AIRFLOW-6409: -- Commit 179ddfd937748f017b2c1d4ce9f481627e896a91 in airflow's branch refs/heads/master from TobKed [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=179ddfd ] [AIRFLOW-6409] BigQuery hook - add tests for run_query method (#6971) > BigQuery hook - add tests for run_query method > -- > > Key: AIRFLOW-6409 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6409 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, hooks, tests >Affects Versions: 1.10.7 >Reporter: Tobiasz Kedzierski >Assignee: Tobiasz Kedzierski >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6409) BigQuery hook - add tests for run_query method
[ https://issues.apache.org/jira/browse/AIRFLOW-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005809#comment-17005809 ] ASF GitHub Bot commented on AIRFLOW-6409: - potiuk commented on pull request #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method URL: https://github.com/apache/airflow/pull/6971 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > BigQuery hook - add tests for run_query method > -- > > Key: AIRFLOW-6409 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6409 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, hooks, tests >Affects Versions: 1.10.7 >Reporter: Tobiasz Kedzierski >Assignee: Tobiasz Kedzierski >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method
potiuk merged pull request #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method URL: https://github.com/apache/airflow/pull/6971 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method
codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method URL: https://github.com/apache/airflow/pull/6971#issuecomment-569783872 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=h1) Report > Merging [#6971](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/fd345914c39b9e3534ee2986eef3498c24aff57a?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6971 +/- ## == - Coverage84.8% 84.52% -0.29% == Files 679 679 Lines 3848938489 == - Hits3264132532 -109 - Misses 5848 5957 +109 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.72% <0%> (-1.16%)` | :arrow_down: | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `77.07% <0%> (+0.75%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=footer). Last update [fd34591...8646e54](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method
codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method URL: https://github.com/apache/airflow/pull/6971#issuecomment-569783872 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=h1) Report > Merging [#6971](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/fd345914c39b9e3534ee2986eef3498c24aff57a?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6971 +/- ## == - Coverage84.8% 84.52% -0.29% == Files 679 679 Lines 3848938489 == - Hits3264132532 -109 - Misses 5848 5957 +109 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.72% <0%> (-1.16%)` | :arrow_down: | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `77.07% <0%> (+0.75%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=footer). Last update [fd34591...8646e54](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method
codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method URL: https://github.com/apache/airflow/pull/6971#issuecomment-569783872 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=h1) Report > Merging [#6971](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/fd345914c39b9e3534ee2986eef3498c24aff57a?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6971 +/- ## == - Coverage84.8% 84.52% -0.29% == Files 679 679 Lines 3848938489 == - Hits3264132532 -109 - Misses 5848 5957 +109 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.72% <0%> (-1.16%)` | :arrow_down: | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `77.07% <0%> (+0.75%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=footer). Last update [fd34591...8646e54](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method
codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method URL: https://github.com/apache/airflow/pull/6971#issuecomment-569783872 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=h1) Report > Merging [#6971](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/fd345914c39b9e3534ee2986eef3498c24aff57a?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6971 +/- ## == - Coverage84.8% 84.52% -0.29% == Files 679 679 Lines 3848938489 == - Hits3264132532 -109 - Misses 5848 5957 +109 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.72% <0%> (-1.16%)` | :arrow_down: | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `77.07% <0%> (+0.75%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=footer). Last update [fd34591...8646e54](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method
codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method URL: https://github.com/apache/airflow/pull/6971#issuecomment-569783872 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=h1) Report > Merging [#6971](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/fd345914c39b9e3534ee2986eef3498c24aff57a?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6971 +/- ## == - Coverage84.8% 84.52% -0.29% == Files 679 679 Lines 3848938489 == - Hits3264132532 -109 - Misses 5848 5957 +109 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.72% <0%> (-1.16%)` | :arrow_down: | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `77.07% <0%> (+0.75%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=footer). Last update [fd34591...8646e54](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method
codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method URL: https://github.com/apache/airflow/pull/6971#issuecomment-569783872 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=h1) Report > Merging [#6971](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/fd345914c39b9e3534ee2986eef3498c24aff57a?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6971 +/- ## == - Coverage84.8% 84.52% -0.29% == Files 679 679 Lines 3848938489 == - Hits3264132532 -109 - Misses 5848 5957 +109 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.72% <0%> (-1.16%)` | :arrow_down: | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `77.07% <0%> (+0.75%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=footer). Last update [fd34591...8646e54](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method
codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method URL: https://github.com/apache/airflow/pull/6971#issuecomment-569783872 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=h1) Report > Merging [#6971](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/fd345914c39b9e3534ee2986eef3498c24aff57a?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6971 +/- ## == - Coverage84.8% 84.52% -0.29% == Files 679 679 Lines 3848938489 == - Hits3264132532 -109 - Misses 5848 5957 +109 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.72% <0%> (-1.16%)` | :arrow_down: | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `77.07% <0%> (+0.75%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=footer). Last update [fd34591...8646e54](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method
codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method URL: https://github.com/apache/airflow/pull/6971#issuecomment-569783872 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=h1) Report > Merging [#6971](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/fd345914c39b9e3534ee2986eef3498c24aff57a?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6971 +/- ## == - Coverage84.8% 84.52% -0.29% == Files 679 679 Lines 3848938489 == - Hits3264132532 -109 - Misses 5848 5957 +109 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.72% <0%> (-1.16%)` | :arrow_down: | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `77.07% <0%> (+0.75%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=footer). Last update [fd34591...8646e54](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method
codecov-io edited a comment on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method URL: https://github.com/apache/airflow/pull/6971#issuecomment-569783872 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=h1) Report > Merging [#6971](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/fd345914c39b9e3534ee2986eef3498c24aff57a?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6971 +/- ## == - Coverage84.8% 84.52% -0.29% == Files 679 679 Lines 3848938489 == - Hits3264132532 -109 - Misses 5848 5957 +109 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.72% <0%> (-1.16%)` | :arrow_down: | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `77.07% <0%> (+0.75%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=footer). Last update [fd34591...8646e54](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method
codecov-io commented on issue #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method URL: https://github.com/apache/airflow/pull/6971#issuecomment-569783872 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=h1) Report > Merging [#6971](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/fd345914c39b9e3534ee2986eef3498c24aff57a?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6971 +/- ## == - Coverage84.8% 84.52% -0.29% == Files 679 679 Lines 3848938489 == - Hits3264132532 -109 - Misses 5848 5957 +109 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/jobs/backfill\_job.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2JhY2tmaWxsX2pvYi5weQ==) | `90.72% <0%> (-1.16%)` | :arrow_down: | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/6971/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `77.07% <0%> (+0.75%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=footer). Last update [fd34591...8646e54](https://codecov.io/gh/apache/airflow/pull/6971?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6407) BigQuery hook - add tests for create_external_table method
[ https://issues.apache.org/jira/browse/AIRFLOW-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005780#comment-17005780 ] ASF GitHub Bot commented on AIRFLOW-6407: - TobKed commented on pull request #6973: [AIRFLOW-6407] BigQuery hook - add tests for create_external_table me… URL: https://github.com/apache/airflow/pull/6973 …thod - [ ] Description above provides context of the change - [ ] Commit message contains [\[AIRFLOW-6407\]](https://issues.apache.org/jira/browse/AIRFLOW-6407) or `[AIRFLOW-]` for document-only changes - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > BigQuery hook - add tests for create_external_table method > -- > > Key: AIRFLOW-6407 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6407 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, hooks, tests >Affects Versions: 1.10.7 >Reporter: Tobiasz Kedzierski >Assignee: Tobiasz Kedzierski >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] TobKed opened a new pull request #6973: [AIRFLOW-6407] BigQuery hook - add tests for create_external_table me…
TobKed opened a new pull request #6973: [AIRFLOW-6407] BigQuery hook - add tests for create_external_table me… URL: https://github.com/apache/airflow/pull/6973 …thod - [ ] Description above provides context of the change - [ ] Commit message contains [\[AIRFLOW-6407\]](https://issues.apache.org/jira/browse/AIRFLOW-6407) or `[AIRFLOW-]` for document-only changes - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6408) BigQuery hook - add tests for patch_table method
[ https://issues.apache.org/jira/browse/AIRFLOW-6408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005779#comment-17005779 ] ASF GitHub Bot commented on AIRFLOW-6408: - TobKed commented on pull request #6972: [AIRFLOW-6408] BigQuery hook - add tests for patch_table method URL: https://github.com/apache/airflow/pull/6972 - [ ] Description above provides context of the change - [ ] Commit message contains [\[AIRFLOW-6408\]](https://issues.apache.org/jira/browse/AIRFLOW-6408) or `[AIRFLOW-6408]` for document-only changes - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > BigQuery hook - add tests for patch_table method > > > Key: AIRFLOW-6408 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6408 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, hooks, tests >Affects Versions: 1.10.7 >Reporter: Tobiasz Kedzierski >Assignee: Tobiasz Kedzierski >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] TobKed opened a new pull request #6972: [AIRFLOW-6408] BigQuery hook - add tests for patch_table method
TobKed opened a new pull request #6972: [AIRFLOW-6408] BigQuery hook - add tests for patch_table method URL: https://github.com/apache/airflow/pull/6972 - [ ] Description above provides context of the change - [ ] Commit message contains [\[AIRFLOW-6408\]](https://issues.apache.org/jira/browse/AIRFLOW-6408) or `[AIRFLOW-6408]` for document-only changes - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] TobKed opened a new pull request #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method
TobKed opened a new pull request #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method URL: https://github.com/apache/airflow/pull/6971 - [ ] Description above provides context of the change - [ ] Commit message contains [\[AIRFLOW-6409\]](https://issues.apache.org/jira/browse/AIRFLOW-6409) or `[AIRFLOW-6409]` for document-only changes - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6409) BigQuery hook - add tests for run_query method
[ https://issues.apache.org/jira/browse/AIRFLOW-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005778#comment-17005778 ] ASF GitHub Bot commented on AIRFLOW-6409: - TobKed commented on pull request #6971: [AIRFLOW-6409] BigQuery hook - add tests for run_query method URL: https://github.com/apache/airflow/pull/6971 - [ ] Description above provides context of the change - [ ] Commit message contains [\[AIRFLOW-6409\]](https://issues.apache.org/jira/browse/AIRFLOW-6409) or `[AIRFLOW-6409]` for document-only changes - [ ] Unit tests coverage for changes (not needed for documentation changes) - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [ ] Relevant documentation is updated including usage instructions. - [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > BigQuery hook - add tests for run_query method > -- > > Key: AIRFLOW-6409 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6409 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, hooks, tests >Affects Versions: 1.10.7 >Reporter: Tobiasz Kedzierski >Assignee: Tobiasz Kedzierski >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6408) BigQuery hook - add tests for patch_table method
Tobiasz Kedzierski created AIRFLOW-6408: --- Summary: BigQuery hook - add tests for patch_table method Key: AIRFLOW-6408 URL: https://issues.apache.org/jira/browse/AIRFLOW-6408 Project: Apache Airflow Issue Type: Improvement Components: gcp, hooks, tests Affects Versions: 1.10.7 Reporter: Tobiasz Kedzierski Assignee: Tobiasz Kedzierski -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6409) BigQuery hook - add tests for run_query method
Tobiasz Kedzierski created AIRFLOW-6409: --- Summary: BigQuery hook - add tests for run_query method Key: AIRFLOW-6409 URL: https://issues.apache.org/jira/browse/AIRFLOW-6409 Project: Apache Airflow Issue Type: Improvement Components: gcp, hooks, tests Affects Versions: 1.10.7 Reporter: Tobiasz Kedzierski Assignee: Tobiasz Kedzierski -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6407) BigQuery hook - add tests for create_external_table method
Tobiasz Kedzierski created AIRFLOW-6407: --- Summary: BigQuery hook - add tests for create_external_table method Key: AIRFLOW-6407 URL: https://issues.apache.org/jira/browse/AIRFLOW-6407 Project: Apache Airflow Issue Type: Improvement Components: gcp, hooks, tests Affects Versions: 1.10.7 Reporter: Tobiasz Kedzierski Assignee: Tobiasz Kedzierski -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6398) improve flakey test test_mark_success_no_kill
[
https://issues.apache.org/jira/browse/AIRFLOW-6398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005770#comment-17005770
]
ASF subversion and git services commented on AIRFLOW-6398:
--
Commit fed4c72cca7c256d140b993315919ec7a0536145 in airflow's branch
refs/heads/v1-10-test from Daniel Standish
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=fed4c72 ]
[AIRFLOW-6398] improve flakey test test_mark_success_no_kill (#6959)
(cherry picked from commit fd345914c39b9e3534ee2986eef3498c24aff57a)
> improve flakey test test_mark_success_no_kill
> -
>
> Key: AIRFLOW-6398
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6398
> Project: Apache Airflow
> Issue Type: Bug
> Components: tests
>Affects Versions: 1.10.7
>Reporter: Daniel Standish
>Assignee: Daniel Standish
>Priority: Major
>
> test test_mark_success_no_kill fails regularly
> part of the problem is that it depends on timing of a subprocess and database
> operations
> we can reduce complexity by using python operator instead of bash operator.
> currently it uses bash operator to call {{sleep 600}}
> this introduces an unnecessary layer of multiprocessing. there is also a bug
> in bash operator which was exacerbating the problem [AIRFLOW-6397]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4464) Fix case-insensitive id columns in mysql
[ https://issues.apache.org/jira/browse/AIRFLOW-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005764#comment-17005764 ] t oo commented on AIRFLOW-4464: --- ran into this. https://github.com/apache/airflow/blob/1.10.6/airflow/models/dagrun.py#L392-L399 is where error was raised IntegrityError: (MySQLdb._exceptions.IntegrityError) (1062, "Duplicate entry File "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/jobs/scheduler_job.py", line 157, in _run_file_processor pickle_dags) File "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/utils/db.py", line 74, in wrapper return func(*args, **kwargs) File "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/jobs/scheduler_job.py", line 1591, in process_file self._process_dags(dagbag, dags, ti_keys_to_schedule) File "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/jobs/scheduler_job.py", line 1276, in _process_dags self._process_task_instances(dag, tis_out) File "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/utils/db.py", line 74, in wrapper return func(*args, **kwargs) File "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/jobs/scheduler_job.py", line 761, in _process_task_instances run.verify_integrity(session=session) File "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/utils/db.py", line 70, in wrapper return func(*args, **kwargs) File "/home/ec2-user/venv/local/lib/python2.7/site-packages/airflow/models/dagrun.py", line 399, in verify_integrity session.commit() > Fix case-insensitive id columns in mysql > > > Key: AIRFLOW-4464 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4464 > Project: Apache Airflow > Issue Type: Improvement > Components: database >Reporter: Joshua Carp >Assignee: Joshua Carp >Priority: Minor > Labels: mysql > > By default, string comparisons in mysql are case-insensitive, so the task ids > "foo" and "FOO" are treated as identical. This means that a dag with those > task ids will fail to schedule with a sqlalchemy `IntegrityError` using > mysql, but not postgres or sqlite. This situation probably doesn't happen > often, and users probably shouldn't use task ids that are identical except > for case, but I think we should improve the behavior here. A few options: > > * Configure sqlalchemy to use a binary collation for string id columns under > mysql so that string comparisons are case-sensitive. > * Require dag and task ids to be unique regardless of case. This would be a > breaking change. > * Document that mysql users should configure mysql to use binary collations > for string types by default. This would still show users a 500 if the > database isn't configured correctly. > > I'll submit a pull request with a failing unit test to describe the issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk commented on issue #6950: [AIRFLOW-6392] Remove cyclic dependency baseoperator <-> helpers
potiuk commented on issue #6950: [AIRFLOW-6392] Remove cyclic dependency baseoperator <-> helpers URL: https://github.com/apache/airflow/pull/6950#issuecomment-569765487 @mik-laj ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6950: [AIRFLOW-6392] Remove cyclic dependency baseoperator <-> helpers
codecov-io edited a comment on issue #6950: [AIRFLOW-6392] Remove cyclic dependency baseoperator <-> helpers URL: https://github.com/apache/airflow/pull/6950#issuecomment-569523812 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=h1) Report > Merging [#6950](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3813143957002ac2750fee17a3fb17be0e324205?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6950 +/- ## == - Coverage84.8% 84.52% -0.29% == Files 679 679 Lines 3848938489 == - Hits3264132533 -108 - Misses 5848 5956 +108 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/helpers.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `81.69% <ø> (-1.06%)` | :arrow_down: | | [...low/example\_dags/example\_short\_circuit\_operator.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zaG9ydF9jaXJjdWl0X29wZXJhdG9yLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.25% <100%> (+0.17%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=footer). Last update [3813143...5594ddf](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5462) Google Authentication redirection page crashes with KeyError: 'login'
[
https://issues.apache.org/jira/browse/AIRFLOW-5462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005514#comment-17005514
]
Bruno Faustino commented on AIRFLOW-5462:
-
Issue is still happening in Airflow 1.10.7 even bumping Flask-AppBuilder to
2.2.1.
It is preventing us from using Google Authentication as an OAuth.
> Google Authentication redirection page crashes with KeyError: 'login'
> -
>
> Key: AIRFLOW-5462
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5462
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication
>Affects Versions: 1.10.3, 1.10.4, 1.10.5
>Reporter: Qian Yu
>Priority: Minor
> Fix For: 2.0.0
>
>
> When user is logged out and try to access the URL of a DAG's TreeView, he's
> redirected to login page. However, after he logs in with google
> authentication, the page redirects to the oauth-authorized/login page and
> crashes with a mushroom cloud.
>
> Expected:
> Redirects back to the original URL the user wanted to access
> Actual:
> Crash with mushroom cloud
>
> Configuration:
> {code:python}
> [webserver]
> authenticate = True
> auth_backend = airflow.contrib.auth.backends.google_auth
> [google]
> client_id = google_client_id
> client_secret = google_client_secret
> oauth_callback_route = /oauth2callback
> domain = "example1.com,example2.com"
> {code}
>
> The issue looks similar to the one described here:
> [https://github.com/apache/incubator-superset/issues/7739]
> {code:python}
> / ( () ) \___
> /( ( ( ) _)) ) )\
>(( ( )() ) ( ) )
> ((/ ( _( ) ( _) ) ( () ) )
> ( ( ( (_) ((( ) .((_ ) . )_
>( ( )( ( )) ) . ) ( )
> ( ( ( ( ) ( _ ( _) ). ) . ) ) ( )
> ( ( ( ) ( ) ( )) ) _)( ) ) )
> ( ( ( \ ) ((_ ( ) ( ) ) ) ) )) ( )
> ( ( ( ( (_ ( ) ( _) ) ( ) ) )
> ( ( ( ( ( ) (_ ) ) ) _) ) _( ( )
> (( ( )(( _) _) _(_ ( (_ )
>(_((__(_(__(( ( ( | ) ) ) )_))__))_)___)
>((__)\\||lll|l||/// \_))
> ( /(/ ( ) ) )\ )
> (( ( ( | | ) ) )\ )
>( /(| / ( )) ) ) )) )
> ( ( _(|)_) )
> ( ||\(|(|)|/|| )
> (|(||(||))
> ( //|/l|||)|\\ \ )
> (/ / // /|//\\ \ \ \ _)
> ---
> Node: ...
> ---
> Traceback (most recent call last):
> File "/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
> response = self.full_dispatch_request()
> File "/lib/python3.6/site-packages/flask/app.py", line 1951, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/lib/python3.6/site-packages/flask/app.py", line 1820, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
> raise value
> File "/lib/python3.6/site-packages/flask/app.py", line 1949, in
> full_dispatch_request
> rv = self.dispatch_request()
> File "/lib/python3.6/site-packages/flask/app.py", line 1935, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/lib/python3.6/site-packages/flask_appbuilder/security/views.py",
> line 633, in oauth_authorized
> resp = self.appbuilder.sm.oauth_remotes[provider].authorized_response()
> KeyError: 'login'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6950: [AIRFLOW-6392] Remove cyclic dependency baseoperator <-> helpers
codecov-io edited a comment on issue #6950: [AIRFLOW-6392] Remove cyclic dependency baseoperator <-> helpers URL: https://github.com/apache/airflow/pull/6950#issuecomment-569523812 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=h1) Report > Merging [#6950](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3813143957002ac2750fee17a3fb17be0e324205?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6950 +/- ## == - Coverage84.8% 84.52% -0.29% == Files 679 679 Lines 3848938489 == - Hits3264132533 -108 - Misses 5848 5956 +108 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/helpers.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `81.69% <ø> (-1.06%)` | :arrow_down: | | [...low/example\_dags/example\_short\_circuit\_operator.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zaG9ydF9jaXJjdWl0X29wZXJhdG9yLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.25% <100%> (+0.17%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=footer). Last update [3813143...5594ddf](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6950: [AIRFLOW-6392] Remove cyclic dependency baseoperator <-> helpers
codecov-io edited a comment on issue #6950: [AIRFLOW-6392] Remove cyclic dependency baseoperator <-> helpers URL: https://github.com/apache/airflow/pull/6950#issuecomment-569523812 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=h1) Report > Merging [#6950](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3813143957002ac2750fee17a3fb17be0e324205?src=pr=desc) will **decrease** coverage by `0.28%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6950 +/- ## == - Coverage84.8% 84.52% -0.29% == Files 679 679 Lines 3848938489 == - Hits3264132533 -108 - Misses 5848 5956 +108 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/helpers.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `81.69% <ø> (-1.06%)` | :arrow_down: | | [...low/example\_dags/example\_short\_circuit\_operator.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zaG9ydF9jaXJjdWl0X29wZXJhdG9yLnB5) | `100% <100%> (ø)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.25% <100%> (+0.17%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6950/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=footer). Last update [3813143...5594ddf](https://codecov.io/gh/apache/airflow/pull/6950?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6398) improve flakey test test_mark_success_no_kill
[
https://issues.apache.org/jira/browse/AIRFLOW-6398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005497#comment-17005497
]
ASF GitHub Bot commented on AIRFLOW-6398:
-
potiuk commented on pull request #6959: [AIRFLOW-6398] improve flakey test
test_mark_success_no_kill
URL: https://github.com/apache/airflow/pull/6959
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> improve flakey test test_mark_success_no_kill
> -
>
> Key: AIRFLOW-6398
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6398
> Project: Apache Airflow
> Issue Type: Bug
> Components: tests
>Affects Versions: 1.10.7
>Reporter: Daniel Standish
>Assignee: Daniel Standish
>Priority: Major
>
> test test_mark_success_no_kill fails regularly
> part of the problem is that it depends on timing of a subprocess and database
> operations
> we can reduce complexity by using python operator instead of bash operator.
> currently it uses bash operator to call {{sleep 600}}
> this introduces an unnecessary layer of multiprocessing. there is also a bug
> in bash operator which was exacerbating the problem [AIRFLOW-6397]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6398) improve flakey test test_mark_success_no_kill
[
https://issues.apache.org/jira/browse/AIRFLOW-6398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005498#comment-17005498
]
ASF subversion and git services commented on AIRFLOW-6398:
--
Commit fd345914c39b9e3534ee2986eef3498c24aff57a in airflow's branch
refs/heads/master from Daniel Standish
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=fd34591 ]
[AIRFLOW-6398] improve flakey test test_mark_success_no_kill (#6959)
> improve flakey test test_mark_success_no_kill
> -
>
> Key: AIRFLOW-6398
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6398
> Project: Apache Airflow
> Issue Type: Bug
> Components: tests
>Affects Versions: 1.10.7
>Reporter: Daniel Standish
>Assignee: Daniel Standish
>Priority: Major
>
> test test_mark_success_no_kill fails regularly
> part of the problem is that it depends on timing of a subprocess and database
> operations
> we can reduce complexity by using python operator instead of bash operator.
> currently it uses bash operator to call {{sleep 600}}
> this introduces an unnecessary layer of multiprocessing. there is also a bug
> in bash operator which was exacerbating the problem [AIRFLOW-6397]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] potiuk commented on issue #6959: [AIRFLOW-6398] improve flakey test test_mark_success_no_kill
potiuk commented on issue #6959: [AIRFLOW-6398] improve flakey test test_mark_success_no_kill URL: https://github.com/apache/airflow/pull/6959#issuecomment-569749869 Thanks @dstandish ! one less flaky at a time! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk merged pull request #6959: [AIRFLOW-6398] improve flakey test test_mark_success_no_kill
potiuk merged pull request #6959: [AIRFLOW-6398] improve flakey test test_mark_success_no_kill URL: https://github.com/apache/airflow/pull/6959 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #6950: [AIRFLOW-6392] Remove cyclic dependency baseoperator <-> helpers
potiuk commented on a change in pull request #6950: [AIRFLOW-6392] Remove cyclic dependency baseoperator <-> helpers URL: https://github.com/apache/airflow/pull/6950#discussion_r362053242 ## File path: UPDATING.md ## @@ -57,6 +57,31 @@ https://developers.google.com/style/inclusive-documentation --> +### Chain and cross_downstream moved from helpers to BaseOperator + +The chain and cross_downstream methods are now moved to airflow.models.baseoperator module from Review comment: It's already solved as well :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dimberman commented on issue #5788: [POC] multi-threading using asyncio
dimberman commented on issue #5788: [POC] multi-threading using asyncio URL: https://github.com/apache/airflow/pull/5788#issuecomment-569744458 Is this necessary? Asyncio doesn’t do multi-threading. It uses a single thread but switches off what coroutine uses that thread based on readiness. This is actually a pretty important distinction when it comes to parallel processing in python. @ash recently made some changes to the LocalExecutor to use os.fork instead of processes, this makes the multithreading in the LocalExecutor much faster and lighter weight. I don’t think that there’s really that much blocking in our operators outside of specific ones that are making SQL calls etc. I don’t think this would be more performant than the current LocalExecutor but would be glad to discuss potential benefits I’m not seeing/if you’ve done speed comparisons. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx=10.0.32=10.14.5=email_footer_2] On Sun, Dec 29, 2019 at 5:56 PM, Darren Weber wrote: Tagging this as related to * https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-28%3A+Add+AsyncExecutor+option [https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-28%3A+Add+AsyncExecutor+option] * https://issues.apache.org/jira/browse/AIRFLOW-6395 [https://issues.apache.org/jira/browse/AIRFLOW-6395] — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [https://github.com/apache/airflow/pull/5788?email_source=notifications_token=AAUFRAX635KVKD3QBFCES5DQ3FIONA5CNFSM4ILDNB72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHZNMGY#issuecomment-569562651] , or unsubscribe [https://github.com/notifications/unsubscribe-auth/AAUFRAVWOHYCRQB6N456YUTQ3FIONANCNFSM4ILDNB7Q] . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] stale[bot] commented on issue #6270: [AIRFLOW-5566] Fix false positive deprecation warning
stale[bot] commented on issue #6270: [AIRFLOW-5566] Fix false positive deprecation warning URL: https://github.com/apache/airflow/pull/6270#issuecomment-569742297 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005486#comment-17005486
]
Jarek Potiuk commented on AIRFLOW-2516:
---
I applied it and here you have modified [^jobs.py] -> it should be enough to
replace it in your Airflow installation.
> Deadlock found when trying to update task_instance table
>
>
> Key: AIRFLOW-2516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2516
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
>Affects Versions: 1.8.0
>Reporter: Jeff Liu
>Priority: Major
> Attachments: Screenshot 2019-12-30 at 10.42.52.png,
> image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png, jobs.py
>
>
>
>
> {code:java}
> [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask:
> [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue:
> airflow run production_wipeout_wipe_manager.Carat Carat_20180227
> 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py
> [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback
> (most recent call last):
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/bin/airflow", line 27, in
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> args.func(args)
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> pool=args.pool,
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in
> wrapper
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result =
> func(*args, **kwargs)
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in
> _run_raw_task
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> self.handle_failure(e, test_mode, context)
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in
> handle_failure
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> session.merge(self)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1920, in merge
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask:
> _resolve_conflict_map=_resolve_conflict_map)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1974, in _merge
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged =
> self.query(mapper.class_).get(key[1])
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882,
> in get
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident,
> loading.load_on_pk_identity)
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952,
> in _get_impl
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> db_load_fn(self, primary_key_identity)
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247,
> in load_on_pk_i
> dentity
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> q.one()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884,
> in one
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> self.one_or_none()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854,
> in one_or_none
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> list(self)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925,
> in __iter__
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return
> self._execute_and_instances(context)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946,
> in _execute_and_instances
>
[jira] [Updated] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarek Potiuk updated AIRFLOW-2516:
--
Attachment: jobs.py
> Deadlock found when trying to update task_instance table
>
>
> Key: AIRFLOW-2516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2516
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
>Affects Versions: 1.8.0
>Reporter: Jeff Liu
>Priority: Major
> Attachments: Screenshot 2019-12-30 at 10.42.52.png,
> image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png, jobs.py
>
>
>
>
> {code:java}
> [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask:
> [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue:
> airflow run production_wipeout_wipe_manager.Carat Carat_20180227
> 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py
> [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback
> (most recent call last):
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/bin/airflow", line 27, in
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> args.func(args)
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> pool=args.pool,
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in
> wrapper
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result =
> func(*args, **kwargs)
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in
> _run_raw_task
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> self.handle_failure(e, test_mode, context)
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in
> handle_failure
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> session.merge(self)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1920, in merge
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask:
> _resolve_conflict_map=_resolve_conflict_map)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1974, in _merge
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged =
> self.query(mapper.class_).get(key[1])
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882,
> in get
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident,
> loading.load_on_pk_identity)
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952,
> in _get_impl
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> db_load_fn(self, primary_key_identity)
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247,
> in load_on_pk_i
> dentity
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> q.one()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884,
> in one
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> self.one_or_none()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854,
> in one_or_none
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> list(self)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925,
> in __iter__
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: return
> self._execute_and_instances(context)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2946,
> in _execute_and_instances
> [2018-05-23 17:59:57,240] {base_task_runner.py:98} INFO - Subtask:
> close_with_result=True)
> [2018-05-23 17:59:57,240] {base_task_runner.py:98}
[jira] [Commented] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005482#comment-17005482
]
Jarek Potiuk commented on AIRFLOW-2516:
---
Yep - that's exactly the story I figured as well.
Where do you have the doc excerpt from :)? I'd love to read more. I think that
might be good for you indeed to try it out:
[https://github.com/apache/airflow/commit/26911864fecb8cf160db732605b66e3243b10436]
it is rather easy to apply it - it's rather isolated commit. I can apply it and
send you the modified files.
> Deadlock found when trying to update task_instance table
>
>
> Key: AIRFLOW-2516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2516
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
>Affects Versions: 1.8.0
>Reporter: Jeff Liu
>Priority: Major
> Attachments: Screenshot 2019-12-30 at 10.42.52.png,
> image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png
>
>
>
>
> {code:java}
> [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask:
> [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue:
> airflow run production_wipeout_wipe_manager.Carat Carat_20180227
> 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py
> [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback
> (most recent call last):
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/bin/airflow", line 27, in
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> args.func(args)
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> pool=args.pool,
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in
> wrapper
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result =
> func(*args, **kwargs)
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in
> _run_raw_task
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> self.handle_failure(e, test_mode, context)
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in
> handle_failure
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> session.merge(self)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1920, in merge
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask:
> _resolve_conflict_map=_resolve_conflict_map)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1974, in _merge
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged =
> self.query(mapper.class_).get(key[1])
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882,
> in get
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident,
> loading.load_on_pk_identity)
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952,
> in _get_impl
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> db_load_fn(self, primary_key_identity)
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247,
> in load_on_pk_i
> dentity
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> q.one()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2884,
> in one
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> self.one_or_none()
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2854,
> in one_or_none
> [2018-05-23 17:59:57,238] {base_task_runner.py:98} INFO - Subtask: ret =
> list(self)
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2925,
> in __iter__
> [2018-05-23 17:59:57,239] {base_task_runner.py:98} INFO - Subtask:
[GitHub] [airflow] nuclearpinguin commented on issue #6761: [AIRFLOW-6204] Add GCP system tests helper
nuclearpinguin commented on issue #6761: [AIRFLOW-6204] Add GCP system tests helper URL: https://github.com/apache/airflow/pull/6761#issuecomment-569737632 I decided to use argparse instead of click. Mainly because `click.command` decorator returns an object not the decorated function, thus decorated function can not be used in a pytes fixture. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2516) Deadlock found when trying to update task_instance table
[
https://issues.apache.org/jira/browse/AIRFLOW-2516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005465#comment-17005465
]
Zhidong commented on AIRFLOW-2516:
--
The execution plan is different on mariadb-5.5.60 from yours, *QUERY ONE*
actually using 'ti_state' index to scan matched rows on my machine.
As MySQL document puts
{code:java}
1. If a secondary index is used in a search and index record locks to be set
are exclusive, InnoDB also retrieves the corresponding clustered index records
and sets locks on them.
2. UPDATE ... WHERE ... sets an exclusive next-key lock on every record the
search encounters.
3. When UPDATE modifies a clustered index record, implicit locks are taken on
affected secondary index records.
{code}
The story might be:
# *QUERY TWO* find a *row(A)* first by 'PRIMARY' index, set lock on 'PRIMARY'
index.
# *QUERY ONE* also find the *row(A)* via secondary index lock('ti_state'),
and set lock on 'ti_state' index.
# *QUERY ONE* then tries to lock *row(A)* on 'PRIMARY' index, find that it's
already been locked.
# *QUERY TWO* finished updating state field, and tries to aquire 'ti_state'
index lock on *row(A)* .
# Result in deadlock.
[~higrys] , how about just replacing *QUERY ONE* with the new one from V1.10?
This minimize the side effects could be introduced by version update.
> Deadlock found when trying to update task_instance table
>
>
> Key: AIRFLOW-2516
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2516
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
>Affects Versions: 1.8.0
>Reporter: Jeff Liu
>Priority: Major
> Attachments: Screenshot 2019-12-30 at 10.42.52.png,
> image-2019-12-30-10-48-41-313.png, image-2019-12-30-10-58-02-610.png
>
>
>
>
> {code:java}
> [2018-05-23 17:59:57,218] {base_task_runner.py:98} INFO - Subtask:
> [2018-05-23 17:59:57,217] {base_executor.py:49} INFO - Adding to queue:
> airflow run production_wipeout_wipe_manager.Carat Carat_20180227
> 2018-05-23T17:41:18.815809 --local -sd DAGS_FOLDER/wipeout/wipeout.py
> [2018-05-23 17:59:57,231] {base_task_runner.py:98} INFO - Subtask: Traceback
> (most recent call last):
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/bin/airflow", line 27, in
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> args.func(args)
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
> [2018-05-23 17:59:57,232] {base_task_runner.py:98} INFO - Subtask:
> pool=args.pool,
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in
> wrapper
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: result =
> func(*args, **kwargs)
> [2018-05-23 17:59:57,233] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1532, in
> _run_raw_task
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> self.handle_failure(e, test_mode, context)
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1641, in
> handle_failure
> [2018-05-23 17:59:57,234] {base_task_runner.py:98} INFO - Subtask:
> session.merge(self)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1920, in merge
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask:
> _resolve_conflict_map=_resolve_conflict_map)
> [2018-05-23 17:59:57,235] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line
> 1974, in _merge
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: merged =
> self.query(mapper.class_).get(key[1])
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 882,
> in get
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: ident,
> loading.load_on_pk_identity)
> [2018-05-23 17:59:57,236] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 952,
> in _get_impl
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: return
> db_load_fn(self, primary_key_identity)
> [2018-05-23 17:59:57,237] {base_task_runner.py:98} INFO - Subtask: File
> "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 247,
> in load_on_pk_i
> dentity
> [2018-05-23 17:59:57,237] {base_task_runner.py:98}
[GitHub] [airflow] codecov-io edited a comment on issue #6905: [AIRFLOW-6361] Run LocalTaskJob directly in Celery task
codecov-io edited a comment on issue #6905: [AIRFLOW-6361] Run LocalTaskJob directly in Celery task URL: https://github.com/apache/airflow/pull/6905#issuecomment-569145791 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=h1) Report > Merging [#6905](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7d97d16c47ba181839fd583b8653d2872dc4290b?src=pr=desc) will **decrease** coverage by `0.17%`. > The diff coverage is `83.33%`. [](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6905 +/- ## == - Coverage84.7% 84.52% -0.18% == Files 680 679 -1 Lines 3855638492 -64 == - Hits3265932536 -123 - Misses 5897 5956 +59 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `96.7% <100%> (+2.65%)` | :arrow_up: | | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `88.11% <33.33%> (-0.33%)` | :arrow_down: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | ... and [21 more](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=footer). Last update [7d97d16...a992c13](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6905: [AIRFLOW-6361] Run LocalTaskJob directly in Celery task
codecov-io edited a comment on issue #6905: [AIRFLOW-6361] Run LocalTaskJob directly in Celery task URL: https://github.com/apache/airflow/pull/6905#issuecomment-569145791 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=h1) Report > Merging [#6905](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7d97d16c47ba181839fd583b8653d2872dc4290b?src=pr=desc) will **decrease** coverage by `0.17%`. > The diff coverage is `83.33%`. [](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6905 +/- ## == - Coverage84.7% 84.52% -0.18% == Files 680 679 -1 Lines 3855638492 -64 == - Hits3265932536 -123 - Misses 5897 5956 +59 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `96.7% <100%> (+2.65%)` | :arrow_up: | | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `88.11% <33.33%> (-0.33%)` | :arrow_down: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | ... and [21 more](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=footer). Last update [7d97d16...a992c13](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6905: [AIRFLOW-6361] Run LocalTaskJob directly in Celery task
codecov-io edited a comment on issue #6905: [AIRFLOW-6361] Run LocalTaskJob directly in Celery task URL: https://github.com/apache/airflow/pull/6905#issuecomment-569145791 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=h1) Report > Merging [#6905](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7d97d16c47ba181839fd583b8653d2872dc4290b?src=pr=desc) will **decrease** coverage by `0.17%`. > The diff coverage is `83.33%`. [](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6905 +/- ## == - Coverage84.7% 84.52% -0.18% == Files 680 679 -1 Lines 3855638492 -64 == - Hits3265932536 -123 - Misses 5897 5956 +59 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `96.7% <100%> (+2.65%)` | :arrow_up: | | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `88.11% <33.33%> (-0.33%)` | :arrow_down: | | [airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | ... and [21 more](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=footer). Last update [7d97d16...a992c13](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6905: [AIRFLOW-6361] Run LocalTaskJob directly in Celery task
codecov-io edited a comment on issue #6905: [AIRFLOW-6361] Run LocalTaskJob directly in Celery task URL: https://github.com/apache/airflow/pull/6905#issuecomment-569145791 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=h1) Report > Merging [#6905](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7d97d16c47ba181839fd583b8653d2872dc4290b?src=pr=desc) will **decrease** coverage by `0.17%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6905 +/- ## == - Coverage84.7% 84.52% -0.18% == Files 680 679 -1 Lines 3855638492 -64 == - Hits3265932536 -123 - Misses 5897 5956 +59 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `88.11% <100%> (-0.33%)` | :arrow_down: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `96.7% <100%> (+2.65%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.75% <0%> (-20%)` | :arrow_down: | | [airflow/utils/cli.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9jbGkucHk=) | `68.75% <0%> (-2.95%)` | :arrow_down: | | [airflow/cli/commands/dag\_command.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGFnX2NvbW1hbmQucHk=) | `86.2% <0%> (-0.51%)` | :arrow_down: | | [airflow/cli/commands/task\_command.py](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvdGFza19jb21tYW5kLnB5) | `69.93% <0%> (-0.29%)` | :arrow_down: | | ... and [7 more](https://codecov.io/gh/apache/airflow/pull/6905/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=footer). Last update [7d97d16...a992c13](https://codecov.io/gh/apache/airflow/pull/6905?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5790) Migrate AwsHook to /providers/aws
[ https://issues.apache.org/jira/browse/AIRFLOW-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005437#comment-17005437 ] ASF GitHub Bot commented on AIRFLOW-5790: - stale[bot] commented on pull request #6480: [AIRFLOW-5790] Move AWS Hook to /providers/aws URL: https://github.com/apache/airflow/pull/6480 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Migrate AwsHook to /providers/aws > - > > Key: AIRFLOW-5790 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5790 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] stale[bot] closed pull request #6480: [AIRFLOW-5790] Move AWS Hook to /providers/aws
stale[bot] closed pull request #6480: [AIRFLOW-5790] Move AWS Hook to /providers/aws URL: https://github.com/apache/airflow/pull/6480 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
