[GitHub] [airflow] yuqian90 commented on issue #7139: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker
yuqian90 commented on issue #7139: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker URL: https://github.com/apache/airflow/pull/7139#issuecomment-577994008 > Merged to v1-10-test in [14111b4](https://github.com/apache/airflow/commit/14111b4bddbaa7c7085a25cdfe923f88219e1582) Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6608) Change logging level to DEBUG for PythonOperator Env exports
[
https://issues.apache.org/jira/browse/AIRFLOW-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022683#comment-17022683
]
ASF subversion and git services commented on AIRFLOW-6608:
--
Commit 23c5a58b83d2f2febc9945d50824f79e6ff9f510 in airflow's branch
refs/heads/master from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=23c5a58 ]
[AIRFLOW-6608] Change logging level for PythonOperator Env exports (#7246)
> Change logging level to DEBUG for PythonOperator Env exports
>
>
> Key: AIRFLOW-6608
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6608
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Affects Versions: 1.10.7
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 1.10.8
>
>
> The following logs just create noise and add nothing for the end-users. We
> should change this to DEBUG
> {noformat}
> [2020-01-20 17:58:27,364] {python_operator.py:105} INFO - Exporting the
> following env vars:
> AIRFLOW_CTX_DAG_EMAIL=a...@mail.com
> AIRFLOW_CTX_DAG_OWNER=me
> AIRFLOW_CTX_DAG_ID=my_dag_id
> AIRFLOW_CTX_TASK_ID=my_task_id
> AIRFLOW_CTX_EXECUTION_DATE=2020-01-20T12:27:48.408593+00:00
> AIRFLOW_CTX_DAG_RUN_ID=manual__2020-01-20T12:27:48.408593+00:00
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6608) Change logging level to DEBUG for PythonOperator Env exports
[
https://issues.apache.org/jira/browse/AIRFLOW-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022682#comment-17022682
]
ASF GitHub Bot commented on AIRFLOW-6608:
-
kaxil commented on pull request #7246: [AIRFLOW-6608] Change logging level for
PythonOperator Env exports
URL: https://github.com/apache/airflow/pull/7246
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Change logging level to DEBUG for PythonOperator Env exports
>
>
> Key: AIRFLOW-6608
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6608
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Affects Versions: 1.10.7
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 1.10.8
>
>
> The following logs just create noise and add nothing for the end-users. We
> should change this to DEBUG
> {noformat}
> [2020-01-20 17:58:27,364] {python_operator.py:105} INFO - Exporting the
> following env vars:
> AIRFLOW_CTX_DAG_EMAIL=a...@mail.com
> AIRFLOW_CTX_DAG_OWNER=me
> AIRFLOW_CTX_DAG_ID=my_dag_id
> AIRFLOW_CTX_TASK_ID=my_task_id
> AIRFLOW_CTX_EXECUTION_DATE=2020-01-20T12:27:48.408593+00:00
> AIRFLOW_CTX_DAG_RUN_ID=manual__2020-01-20T12:27:48.408593+00:00
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] kaxil merged pull request #7246: [AIRFLOW-6608] Change logging level for PythonOperator Env exports
kaxil merged pull request #7246: [AIRFLOW-6608] Change logging level for PythonOperator Env exports URL: https://github.com/apache/airflow/pull/7246 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow-on-k8s-operator] kaxil merged pull request #6: Get e2e tests working
kaxil merged pull request #6: Get e2e tests working URL: https://github.com/apache/airflow-on-k8s-operator/pull/6 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[airflow-on-k8s-operator] branch master updated: Get e2e tests working (#6)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow-on-k8s-operator.git
The following commit(s) were added to refs/heads/master by this push:
new dd430ea Get e2e tests working (#6)
dd430ea is described below
commit dd430eaa7c3add972f410e058a2fe0a4a58eeb64
Author: barney-s <6457279+barne...@users.noreply.github.com>
AuthorDate: Thu Jan 23 20:41:17 2020 -0800
Get e2e tests working (#6)
- Cleanup Application scheme registration.
- Fix test sample paths
- Set controller synctime to 2 mins
- Cleanup Scheme registration in controller-reconciler
- Use custom codec that is created from custom scheme (kb v2 uses custom
scheme)
---
controllers/airflowbase_controller.go | 2 +-
controllers/airflowcluster_controller.go | 2 +-
main.go| 5 +
test/e2e/base/base_test.go | 2 +-
test/e2e/cluster/cluster_test.go | 2 +-
test/e2e/gcp/gcp_test.go | 2 +-
.../pkg/genericreconciler/genericreconciler.go | 11 ---
.../pkg/reconciler/manager/k8s/manager.go | 23 +++---
.../controller-reconciler/pkg/test/framework.go| 6 +++---
9 files changed, 33 insertions(+), 22 deletions(-)
diff --git a/controllers/airflowbase_controller.go
b/controllers/airflowbase_controller.go
index bb78421..d451a09 100644
--- a/controllers/airflowbase_controller.go
+++ b/controllers/airflowbase_controller.go
@@ -69,6 +69,7 @@ func (r *AirflowBaseReconciler) Reconcile(req ctrl.Request)
(ctrl.Result, error)
// SetupWithManager - called by main
func (r *AirflowBaseReconciler) SetupWithManager(mgr ctrl.Manager) error {
+ _ = app.AddToScheme(r.Scheme)
return ctrl.NewControllerManagedBy(mgr).
For({}).
Complete(abReconciler(mgr))
@@ -86,7 +87,6 @@ func abReconciler(mgr manager.Manager) *gr.Reconciler {
WithErrorHandler(abHandleError).
WithValidator(validate).
WithDefaulter(applyDefaults).
- RegisterSchemeBuilder(app.SchemeBuilder).
Build()
}
diff --git a/controllers/airflowcluster_controller.go
b/controllers/airflowcluster_controller.go
index aadb4bc..0852937 100644
--- a/controllers/airflowcluster_controller.go
+++ b/controllers/airflowcluster_controller.go
@@ -90,6 +90,7 @@ func (r *AirflowClusterReconciler) Reconcile(req
ctrl.Request) (ctrl.Result, err
// SetupWithManager - called by main
func (r *AirflowClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
+ _ = app.AddToScheme(r.Scheme)
return ctrl.NewControllerManagedBy(mgr).
For({}).
Complete(acReconciler(mgr))
@@ -110,7 +111,6 @@ func acReconciler(mgr manager.Manager) *gr.Reconciler {
WithErrorHandler(acHandleError).
WithValidator(acValidate).
WithDefaulter(acApplyDefaults).
- RegisterSchemeBuilder(app.SchemeBuilder).
Build()
}
diff --git a/main.go b/main.go
index c083011..d22f412 100644
--- a/main.go
+++ b/main.go
@@ -18,6 +18,7 @@ package main
import (
"flag"
"os"
+ "time"
airflowv1alpha1 "github.com/apache/airflow-on-k8s-operator/api/v1alpha1"
"github.com/apache/airflow-on-k8s-operator/controllers"
@@ -38,6 +39,7 @@ func init() {
_ = clientgoscheme.AddToScheme(scheme)
_ = airflowv1alpha1.AddToScheme(scheme)
+
// +kubebuilder:scaffold:scheme
}
@@ -53,11 +55,14 @@ func main() {
o.Development = true
}))
+ syncperiod := time.Minute * 2
+
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
Scheme: scheme,
MetricsBindAddress: metricsAddr,
LeaderElection: enableLeaderElection,
Port: 9443,
+ SyncPeriod: ,
})
if err != nil {
setupLog.Error(err, "unable to start manager")
diff --git a/test/e2e/base/base_test.go b/test/e2e/base/base_test.go
index 740389b..1dbfe69 100644
--- a/test/e2e/base/base_test.go
+++ b/test/e2e/base/base_test.go
@@ -27,7 +27,7 @@ import (
const (
CRName= "AirflowBase"
- SampleDir = "../../hack/sample/"
+ SampleDir = "../../../hack/sample/"
)
var f *test.Framework
diff --git a/test/e2e/cluster/cluster_test.go b/test/e2e/cluster/cluster_test.go
index 2d56a68..48eac95 100644
--- a/test/e2e/cluster/cluster_test.go
+++ b/test/e2e/cluster/cluster_test.go
@@ -27,7 +27,7 @@ import (
const (
CRName= "AirflowCluster"
- SampleDir = "../../hack/sample/"
+ SampleDir = "../../../hack/sample/"
)
var f *test.Framework
diff --git a/test/e2e/gcp/gcp_test.go
[GitHub] [airflow] TeddyHartanto commented on issue #5308: [AIRFLOW-4549] skipped tasks should be ok for wait_for_downstream
TeddyHartanto commented on issue #5308: [AIRFLOW-4549] skipped tasks should be ok for wait_for_downstream URL: https://github.com/apache/airflow/pull/5308#issuecomment-577974864 @kaxil no objection too. Just the static checks failure as mentioned by @potiuk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7249: [AIRFLOW-6626] Add email on failure or retry to default config
codecov-io commented on issue #7249: [AIRFLOW-6626] Add email on failure or retry to default config URL: https://github.com/apache/airflow/pull/7249#issuecomment-577972968 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=h1) Report > Merging [#7249](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/21a46518d513f4b66ae1be748328b02903034f38?src=pr=desc) will **decrease** coverage by `0.27%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7249 +/- ## == - Coverage 85.34% 85.07% -0.28% == Files 791 791 Lines 4012440135 +11 == - Hits3424334143 -100 - Misses 5881 5992 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.28% <ø> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `89.34% <0%> (ø)` | :arrow_up: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <0%> (ø)` | :arrow_up: | | [airflow/ti\_deps/deps/runnable\_exec\_date\_dep.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy90aV9kZXBzL2RlcHMvcnVubmFibGVfZXhlY19kYXRlX2RlcC5weQ==) | `100% <0%> (ø)` | :arrow_up: | | [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.62% <0%> (+0.02%)` | :arrow_up: | | ... and [2 more](https://codecov.io/gh/apache/airflow/pull/7249/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=footer). Last update [21a4651...44ad8f4](https://codecov.io/gh/apache/airflow/pull/7249?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] stale[bot] commented on issue #6458: [AIRFLOW-4707](https://issues.apache.org/jira/browse/AIRFLOW-4708) tests/plugins made pylint compliant
stale[bot] commented on issue #6458: [AIRFLOW-4707](https://issues.apache.org/jira/browse/AIRFLOW-4708) tests/plugins made pylint compliant URL: https://github.com/apache/airflow/pull/6458#issuecomment-577971373 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] stale[bot] commented on issue #5550: AIRFLOW-4912: Support to update nested keys in emr_defaults conn extras.
stale[bot] commented on issue #5550: AIRFLOW-4912: Support to update nested keys in emr_defaults conn extras. URL: https://github.com/apache/airflow/pull/5550#issuecomment-577971376 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] stale[bot] commented on issue #6597: [AIRFLOW-6005] Common classes should always be imported from airflow, Depends on [AIRFLOW-6128]
stale[bot] commented on issue #6597: [AIRFLOW-6005] Common classes should always be imported from airflow, Depends on [AIRFLOW-6128] URL: https://github.com/apache/airflow/pull/6597#issuecomment-577971380 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6626) Add email on failure or retry to default config
[ https://issues.apache.org/jira/browse/AIRFLOW-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022651#comment-17022651 ] ASF GitHub Bot commented on AIRFLOW-6626: - xinbinhuang commented on pull request #7249: [AIRFLOW-6626] Add email on failure or retry to default config URL: https://github.com/apache/airflow/pull/7249 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add email on failure or retry to default config > --- > > Key: AIRFLOW-6626 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6626 > Project: Apache Airflow > Issue Type: Task > Components: configuration >Affects Versions: 1.10.7 >Reporter: Xinbin Huang >Assignee: Xinbin Huang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] xinbinhuang opened a new pull request #7249: [AIRFLOW-6626] Add email on failure or retry to default config
xinbinhuang opened a new pull request #7249: [AIRFLOW-6626] Add email on failure or retry to default config URL: https://github.com/apache/airflow/pull/7249 --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6626) Add email on failure or retry to default config
Xinbin Huang created AIRFLOW-6626: - Summary: Add email on failure or retry to default config Key: AIRFLOW-6626 URL: https://issues.apache.org/jira/browse/AIRFLOW-6626 Project: Apache Airflow Issue Type: Task Components: configuration Affects Versions: 1.10.7 Reporter: Xinbin Huang Assignee: Xinbin Huang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io commented on issue #7248: [AIRFLOW-4204] Update super() calls
codecov-io commented on issue #7248: [AIRFLOW-4204] Update super() calls URL: https://github.com/apache/airflow/pull/7248#issuecomment-577959634 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7248?src=pr=h1) Report > Merging [#7248](https://codecov.io/gh/apache/airflow/pull/7248?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/21a46518d513f4b66ae1be748328b02903034f38?src=pr=desc) will **decrease** coverage by `0.27%`. > The diff coverage is `77.77%`. [](https://codecov.io/gh/apache/airflow/pull/7248?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7248 +/- ## == - Coverage 85.34% 85.07% -0.28% == Files 791 791 Lines 4012440135 +11 == - Hits3424334143 -100 - Misses 5881 5992 +111 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7248?src=pr=tree) | Coverage Δ | | |---|---|---| | [...w/providers/snowflake/operators/s3\_to\_snowflake.py](https://codecov.io/gh/apache/airflow/pull/7248/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvc25vd2ZsYWtlL29wZXJhdG9ycy9zM190b19zbm93Zmxha2UucHk=) | `0% <0%> (ø)` | :arrow_up: | | [airflow/providers/snowflake/operators/snowflake.py](https://codecov.io/gh/apache/airflow/pull/7248/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvc25vd2ZsYWtlL29wZXJhdG9ycy9zbm93Zmxha2UucHk=) | `0% <0%> (ø)` | :arrow_up: | | [airflow/contrib/operators/dynamodb\_to\_s3.py](https://codecov.io/gh/apache/airflow/pull/7248/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9keW5hbW9kYl90b19zMy5weQ==) | `88.67% <100%> (ø)` | :arrow_up: | | [airflow/providers/mysql/hooks/mysql.py](https://codecov.io/gh/apache/airflow/pull/7248/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvaG9va3MvbXlzcWwucHk=) | `92.77% <100%> (ø)` | :arrow_up: | | [airflow/operators/bigquery\_to\_mysql.py](https://codecov.io/gh/apache/airflow/pull/7248/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvYmlncXVlcnlfdG9fbXlzcWwucHk=) | `74% <100%> (ø)` | :arrow_up: | | [airflow/operators/google\_api\_to\_s3\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7248/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ29vZ2xlX2FwaV90b19zM190cmFuc2Zlci5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/gcp/sensors/gcs.py](https://codecov.io/gh/apache/airflow/pull/7248/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avc2Vuc29ycy9nY3MucHk=) | `97.14% <100%> (ø)` | :arrow_up: | | [airflow/gcp/hooks/discovery\_api.py](https://codecov.io/gh/apache/airflow/pull/7248/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvZGlzY292ZXJ5X2FwaS5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/operators/sql\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7248/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvc3FsX3RvX2djcy5weQ==) | `92.07% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7248/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | ... and [10 more](https://codecov.io/gh/apache/airflow/pull/7248/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7248?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7248?src=pr=footer). Last update [21a4651...a2514aa](https://codecov.io/gh/apache/airflow/pull/7248?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4204) Update super() calls
[
https://issues.apache.org/jira/browse/AIRFLOW-4204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022627#comment-17022627
]
ASF GitHub Bot commented on AIRFLOW-4204:
-
xinbinhuang commented on pull request #7248: [AIRFLOW-4204] Update super() calls
URL: https://github.com/apache/airflow/pull/7248
Update super() calls
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Update super() calls
> -
>
> Key: AIRFLOW-4204
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4204
> Project: Apache Airflow
> Issue Type: Sub-task
> Components: core
>Reporter: Fokko Driesprong
>Assignee: Chao-Han Tsai
>Priority: Major
> Fix For: 2.0.0
>
>
> In all classes, replace {{super(__class__, self).__init__(...)}} by
> {{super().__init__(...)}}
> Similarly for any other {{super}} calls for other methods.
> (In Python 3 {{super(__class__, self) == super()}})
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] xinbinhuang opened a new pull request #7248: [AIRFLOW-4204] Update super() calls
xinbinhuang opened a new pull request #7248: [AIRFLOW-4204] Update super() calls URL: https://github.com/apache/airflow/pull/7248 Update super() calls --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7234: [AIRFLOW-6493] Add SSL configuration to Redis hook connections
codecov-io edited a comment on issue #7234: [AIRFLOW-6493] Add SSL configuration to Redis hook connections URL: https://github.com/apache/airflow/pull/7234#issuecomment-576896420 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=h1) Report > Merging [#7234](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/059eda05f82fefce4410f44f761f945a27d83daf?src=pr=desc) will **decrease** coverage by `0.71%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7234 +/- ## == - Coverage 85.34% 84.62% -0.72% == Files 791 791 Lines 4012440137 +13 == - Hits3424233966 -276 - Misses 5882 6171 +289 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/providers/redis/hooks/redis.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvaG9va3MvcmVkaXMucHk=) | `100% <100%> (ø)` | :arrow_up: | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0%> (-54.93%)` | :arrow_down: | | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50% <0%> (-50%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0%> (-46.67%)` | :arrow_down: | | [airflow/executors/celery\_executor.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvY2VsZXJ5X2V4ZWN1dG9yLnB5) | `49.65% <0%> (-38.78%)` | :arrow_down: | | [airflow/providers/redis/sensors/redis\_key.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvc2Vuc29ycy9yZWRpc19rZXkucHk=) | `61.53% <0%> (-38.47%)` | :arrow_down: | | ... and [11 more](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=footer). Last update [059eda0...65247f7](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7234: [AIRFLOW-6493] Add SSL configuration to Redis hook connections
codecov-io edited a comment on issue #7234: [AIRFLOW-6493] Add SSL configuration to Redis hook connections URL: https://github.com/apache/airflow/pull/7234#issuecomment-576896420 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=h1) Report > Merging [#7234](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/059eda05f82fefce4410f44f761f945a27d83daf?src=pr=desc) will **decrease** coverage by `0.9%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7234 +/- ## == - Coverage 85.34% 84.43% -0.91% == Files 791 791 Lines 4012440137 +13 == - Hits3424233890 -352 - Misses 5882 6247 +365 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/providers/redis/hooks/redis.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvaG9va3MvcmVkaXMucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0%> (-54.93%)` | :arrow_down: | | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50% <0%> (-50%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0%> (-46.67%)` | :arrow_down: | | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55% <0%> (-45%)` | :arrow_down: | | ... and [17 more](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=footer). Last update [059eda0...65247f7](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7234: [AIRFLOW-6493] Add SSL configuration to Redis hook connections
codecov-io edited a comment on issue #7234: [AIRFLOW-6493] Add SSL configuration to Redis hook connections URL: https://github.com/apache/airflow/pull/7234#issuecomment-576896420 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=h1) Report > Merging [#7234](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/059eda05f82fefce4410f44f761f945a27d83daf?src=pr=desc) will **decrease** coverage by `1.15%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7234 +/- ## == - Coverage 85.34% 84.18% -1.16% == Files 791 791 Lines 4012440137 +13 == - Hits3424233790 -452 - Misses 5882 6347 +465 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/providers/redis/hooks/redis.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvaG9va3MvcmVkaXMucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0%> (-54.93%)` | :arrow_down: | | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50% <0%> (-50%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | ... and [23 more](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=footer). Last update [059eda0...65247f7](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7234: [AIRFLOW-6493] Add SSL configuration to Redis hook connections
codecov-io edited a comment on issue #7234: [AIRFLOW-6493] Add SSL configuration to Redis hook connections URL: https://github.com/apache/airflow/pull/7234#issuecomment-576896420 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=h1) Report > Merging [#7234](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/059eda05f82fefce4410f44f761f945a27d83daf?src=pr=desc) will **decrease** coverage by `1.15%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7234 +/- ## == - Coverage 85.34% 84.18% -1.16% == Files 791 791 Lines 4012440137 +13 == - Hits3424233790 -452 - Misses 5882 6347 +465 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/providers/redis/hooks/redis.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvaG9va3MvcmVkaXMucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0%> (-54.93%)` | :arrow_down: | | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50% <0%> (-50%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | ... and [23 more](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=footer). Last update [059eda0...65247f7](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7234: [AIRFLOW-6493] Add SSL configuration to Redis hook connections
codecov-io edited a comment on issue #7234: [AIRFLOW-6493] Add SSL configuration to Redis hook connections URL: https://github.com/apache/airflow/pull/7234#issuecomment-576896420 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=h1) Report > Merging [#7234](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/059eda05f82fefce4410f44f761f945a27d83daf?src=pr=desc) will **decrease** coverage by `1.15%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7234 +/- ## == - Coverage 85.34% 84.18% -1.16% == Files 791 791 Lines 4012440137 +13 == - Hits3424233790 -452 - Misses 5882 6347 +465 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/providers/redis/hooks/redis.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvaG9va3MvcmVkaXMucHk=) | `100% <100%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/api/auth/backend/kerberos\_auth.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvYXV0aC9iYWNrZW5kL2tlcmJlcm9zX2F1dGgucHk=) | `28.16% <0%> (-54.93%)` | :arrow_down: | | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50% <0%> (-50%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | ... and [23 more](https://codecov.io/gh/apache/airflow/pull/7234/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=footer). Last update [059eda0...65247f7](https://codecov.io/gh/apache/airflow/pull/7234?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7247: [AIRFLOW-6625] Explicitly log using utf-8 encoding
codecov-io commented on issue #7247: [AIRFLOW-6625] Explicitly log using utf-8 encoding URL: https://github.com/apache/airflow/pull/7247#issuecomment-577901642 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7247?src=pr=h1) Report > Merging [#7247](https://codecov.io/gh/apache/airflow/pull/7247?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/61201a5e5cf2ece1253fccdde0dd5833267d9ee2?src=pr=desc) will **decrease** coverage by `0.43%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7247?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7247 +/- ## == - Coverage 85.34% 84.91% -0.44% == Files 791 791 Lines 4013540135 == - Hits3425534079 -176 - Misses 5880 6056 +176 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7247?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/utils/log/file\_task\_handler.py](https://codecov.io/gh/apache/airflow/pull/7247/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZmlsZV90YXNrX2hhbmRsZXIucHk=) | `88.76% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7247/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7247/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7247/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/7247/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `56% <0%> (-44%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7247/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7247/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/7247/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `80.95% <0%> (-7.24%)` | :arrow_down: | | [airflow/utils/helpers.py](https://codecov.io/gh/apache/airflow/pull/7247/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `75.62% <0%> (-6.88%)` | :arrow_down: | | [airflow/jobs/scheduler\_job.py](https://codecov.io/gh/apache/airflow/pull/7247/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL3NjaGVkdWxlcl9qb2IucHk=) | `88.75% <0%> (-0.59%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7247?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7247?src=pr=footer). Last update [61201a5...730a13b](https://codecov.io/gh/apache/airflow/pull/7247?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] davlum commented on issue #7168: [AIRFLOW-6561] Add possibility to specify default resources for airfl…
davlum commented on issue #7168: [AIRFLOW-6561] Add possibility to specify
default resources for airfl…
URL: https://github.com/apache/airflow/pull/7168#issuecomment-577890923
The JSON you are passing will not turn into a K8S resource object. I would
suggest deserializing into a `kubernetes.client.models.V1ResourceRequirements`
object in the `WorkerConfiguration` class.
Hypothetically you could pass JSON which conforms to the K8S api;
`{"limits": {"cpu":"foo", "memory":"bar"}, "requests": {"cpu":"foo",
"memory":"bar"}}`, but I think deserializing to the object is the better choice.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] davlum commented on a change in pull request #7168: [AIRFLOW-6561] Add possibility to specify default resources for airfl…
davlum commented on a change in pull request #7168: [AIRFLOW-6561] Add
possibility to specify default resources for airfl…
URL: https://github.com/apache/airflow/pull/7168#discussion_r370363281
##
File path: airflow/executors/kubernetes_executor.py
##
@@ -110,6 +110,12 @@ def __init__(self): # pylint: disable=too-many-statements
self.worker_run_as_user = self._get_security_context_val('run_as_user')
self.worker_fs_group = self._get_security_context_val('fs_group')
+kube_worker_resources = conf.get(self.kubernetes_section,
'worker_resources')
+if kube_worker_resources:
+self.worker_resources = json.loads(kube_worker_resources)
+else:
+self.worker_resources = None
Review comment:
```suggestion
kube_worker_resources = conf.get(self.kubernetes_section,
'worker_resources', fallback=None)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] davlum commented on a change in pull request #7168: [AIRFLOW-6561] Add possibility to specify default resources for airfl…
davlum commented on a change in pull request #7168: [AIRFLOW-6561] Add
possibility to specify default resources for airfl…
URL: https://github.com/apache/airflow/pull/7168#discussion_r370363281
##
File path: airflow/executors/kubernetes_executor.py
##
@@ -110,6 +110,12 @@ def __init__(self): # pylint: disable=too-many-statements
self.worker_run_as_user = self._get_security_context_val('run_as_user')
self.worker_fs_group = self._get_security_context_val('fs_group')
+kube_worker_resources = conf.get(self.kubernetes_section,
'worker_resources')
+if kube_worker_resources:
+self.worker_resources = json.loads(kube_worker_resources)
+else:
+self.worker_resources = None
Review comment:
```suggestion
kube_worker_resources = conf.get(self.kubernetes_section,
'worker_resources', fallback=None)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] s-mcqueen opened a new pull request #7247: [AIRFLOW-6625] Explicitly log using utf-8 encoding
s-mcqueen opened a new pull request #7247: [AIRFLOW-6625] Explicitly log using utf-8 encoding URL: https://github.com/apache/airflow/pull/7247 This is the standard encoding supported by Airflow. We should be explicit about logging using this encoding. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [X] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6625) Airflow should log explicitly in utf-8
[ https://issues.apache.org/jira/browse/AIRFLOW-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022511#comment-17022511 ] ASF GitHub Bot commented on AIRFLOW-6625: - s-mcqueen commented on pull request #7247: [AIRFLOW-6625] Explicitly log using utf-8 encoding URL: https://github.com/apache/airflow/pull/7247 This is the standard encoding supported by Airflow. We should be explicit about logging using this encoding. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [X] Unit tests coverage for changes (not needed for documentation changes) - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [X] Relevant documentation is updated including usage instructions. - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Airflow should log explicitly in utf-8 > -- > > Key: AIRFLOW-6625 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6625 > Project: Apache Airflow > Issue Type: Task > Components: core >Affects Versions: 1.10.7 >Reporter: Sean McQueen >Assignee: Sean McQueen >Priority: Minor > > Currently in file_task_logger, we use a logging.FileHandler without a > specified encoding. Because Airflow supports utf-8, we should be explicit > here. This can prevent some environment issues where Airflow will fail in > logging because of implicit encoding at this layer -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] boring-cyborg[bot] commented on issue #7247: [AIRFLOW-6625] Explicitly log using utf-8 encoding
boring-cyborg[bot] commented on issue #7247: [AIRFLOW-6625] Explicitly log using utf-8 encoding URL: https://github.com/apache/airflow/pull/7247#issuecomment-577883090 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github .com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://apache-airflow-slack.herokuapp.com/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6625) Airflow should log explicitly in utf-8
Sean McQueen created AIRFLOW-6625: - Summary: Airflow should log explicitly in utf-8 Key: AIRFLOW-6625 URL: https://issues.apache.org/jira/browse/AIRFLOW-6625 Project: Apache Airflow Issue Type: Task Components: core Affects Versions: 1.10.7 Reporter: Sean McQueen Assignee: Sean McQueen Currently in file_task_logger, we use a logging.FileHandler without a specified encoding. Because Airflow supports utf-8, we should be explicit here. This can prevent some environment issues where Airflow will fail in logging because of implicit encoding at this layer -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow-on-k8s-operator] barney-s commented on issue #6: Get e2e tests working
barney-s commented on issue #6: Get e2e tests working
URL:
https://github.com/apache/airflow-on-k8s-operator/pull/6#issuecomment-577875276
## Tests passed
```shell
go test -v -timeout 20m test/e2e/base/base_test.go --namespace
airflowop-system
=== RUN Test
Running Suite: AirflowBase Suite
Random Seed: 1579811766
Will run 2 of 2 specs
• [SLOW TEST:185.490 seconds]
AirflowBase controller tests
/usr/local/google/home/barni/workspace/src/github.com/apache/airflow-on-k8s-operator/test/e2e/base/base_test.go:70
creating a AirflowBase with mysql
/usr/local/google/home/barni/workspace/src/github.com/apache/airflow-on-k8s-operator/test/e2e/base/base_test.go:76
--
• [SLOW TEST:35.350 seconds]
AirflowBase controller tests
/usr/local/google/home/barni/workspace/src/github.com/apache/airflow-on-k8s-operator/test/e2e/base/base_test.go:70
creating a AirflowBase with postgres
/usr/local/google/home/barni/workspace/src/github.com/apache/airflow-on-k8s-operator/test/e2e/base/base_test.go:88
--
[GitHub] [airflow-on-k8s-operator] barney-s opened a new pull request #6: Get e2e tests working
barney-s opened a new pull request #6: Get e2e tests working URL: https://github.com/apache/airflow-on-k8s-operator/pull/6 - Cleanup Application scheme registration. - Fix test sample paths - Set controller synctime to 2 mins - Cleanup Scheme registration in controller-reconciler - Use custom codec that is created from custom scheme (kb v2 uses custom scheme) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6620) Mock celery in worker cli test
[ https://issues.apache.org/jira/browse/AIRFLOW-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022457#comment-17022457 ] ASF subversion and git services commented on AIRFLOW-6620: -- Commit e1018fab9e8cda6f899a9e78ce9b31675e5b5398 in airflow's branch refs/heads/v1-10-test from Tomek Urbaszek [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e1018fa ] [AIRFLOW-6620] Mock celery in worker cli test (#7243) Actual tests start celery worker and doesn't stop it. Sometimes it result in timeout of a test. > Mock celery in worker cli test > -- > > Key: AIRFLOW-6620 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6620 > Project: Apache Airflow > Issue Type: Improvement > Components: celery, tests >Affects Versions: 1.10.8 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 1.10.8 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6608) Change logging level to DEBUG for PythonOperator Env exports
[
https://issues.apache.org/jira/browse/AIRFLOW-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022444#comment-17022444
]
ASF subversion and git services commented on AIRFLOW-6608:
--
Commit 7d00479402905e991b55ae26c4b04b44daa0192e in airflow's branch
refs/heads/v1-10-test from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=7d00479 ]
fixup! [AIRFLOW-6608] Change logging level for Bash & PyOperator Env exports
> Change logging level to DEBUG for PythonOperator Env exports
>
>
> Key: AIRFLOW-6608
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6608
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Affects Versions: 1.10.7
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 1.10.8
>
>
> The following logs just create noise and add nothing for the end-users. We
> should change this to DEBUG
> {noformat}
> [2020-01-20 17:58:27,364] {python_operator.py:105} INFO - Exporting the
> following env vars:
> AIRFLOW_CTX_DAG_EMAIL=a...@mail.com
> AIRFLOW_CTX_DAG_OWNER=me
> AIRFLOW_CTX_DAG_ID=my_dag_id
> AIRFLOW_CTX_TASK_ID=my_task_id
> AIRFLOW_CTX_EXECUTION_DATE=2020-01-20T12:27:48.408593+00:00
> AIRFLOW_CTX_DAG_RUN_ID=manual__2020-01-20T12:27:48.408593+00:00
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #7246: [AIRFLOW-6608] Change logging level for PythonOperator Env exports
codecov-io edited a comment on issue #7246: [AIRFLOW-6608] Change logging level for PythonOperator Env exports URL: https://github.com/apache/airflow/pull/7246#issuecomment-577852270 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=h1) Report > Merging [#7246](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/af91d9f4f08a6bf308cb10418669d17d1ae6cc44?src=pr=desc) will **decrease** coverage by `1.16%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7246 +/- ## == - Coverage 85.34% 84.17% -1.17% == Files 791 791 Lines 4012840135 +7 == - Hits3424733784 -463 - Misses 5881 6351 +470 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.81% <ø> (ø)` | :arrow_up: | | [airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=) | `52.32% <100%> (-0.18%)` | :arrow_down: | | [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.62% <100%> (+0.02%)` | :arrow_up: | | [airflow/operators/python.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcHl0aG9uLnB5) | `95.13% <100%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | ... and [22 more](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=footer). Last update [af91d9f...3353bfe](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #7246: [AIRFLOW-6608] Change logging level for PythonOperator Env exports
codecov-io edited a comment on issue #7246: [AIRFLOW-6608] Change logging level for PythonOperator Env exports URL: https://github.com/apache/airflow/pull/7246#issuecomment-577852270 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=h1) Report > Merging [#7246](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/af91d9f4f08a6bf308cb10418669d17d1ae6cc44?src=pr=desc) will **decrease** coverage by `1.16%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7246 +/- ## == - Coverage 85.34% 84.17% -1.17% == Files 791 791 Lines 4012840135 +7 == - Hits3424733784 -463 - Misses 5881 6351 +470 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.81% <ø> (ø)` | :arrow_up: | | [airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=) | `52.32% <100%> (-0.18%)` | :arrow_down: | | [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.62% <100%> (+0.02%)` | :arrow_up: | | [airflow/operators/python.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcHl0aG9uLnB5) | `95.13% <100%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | ... and [22 more](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=footer). Last update [af91d9f...3353bfe](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #7169: [AIRFLOW-6555] mushroom cloud error when clicking 'task instance details' from graph view of dag that has never been run yet
kaxil commented on issue #7169: [AIRFLOW-6555] mushroom cloud error when clicking 'task instance details' from graph view of dag that has never been run yet URL: https://github.com/apache/airflow/pull/7169#issuecomment-577852483 @KevinYang21 What do you think of this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7246: [AIRFLOW-6608] Change logging level for PythonOperator Env exports
codecov-io commented on issue #7246: [AIRFLOW-6608] Change logging level for PythonOperator Env exports URL: https://github.com/apache/airflow/pull/7246#issuecomment-577852270 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=h1) Report > Merging [#7246](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/af91d9f4f08a6bf308cb10418669d17d1ae6cc44?src=pr=desc) will **decrease** coverage by `1.16%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7246 +/- ## == - Coverage 85.34% 84.17% -1.17% == Files 791 791 Lines 4012840135 +7 == - Hits3424733784 -463 - Misses 5881 6351 +470 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `94.73% <ø> (ø)` | :arrow_up: | | [airflow/gcp/hooks/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvaG9va3MvYmlncXVlcnkucHk=) | `91.81% <ø> (ø)` | :arrow_up: | | [airflow/cli/commands/celery\_command.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvY2VsZXJ5X2NvbW1hbmQucHk=) | `52.32% <100%> (-0.18%)` | :arrow_down: | | [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.62% <100%> (+0.02%)` | :arrow_up: | | [airflow/operators/python.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcHl0aG9uLnB5) | `95.13% <100%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/generic\_transfer.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvZ2VuZXJpY190cmFuc2Zlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.51% <0%> (-72.16%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | ... and [22 more](https://codecov.io/gh/apache/airflow/pull/7246/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=footer). Last update [af91d9f...3353bfe](https://codecov.io/gh/apache/airflow/pull/7246?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] tooptoop4 commented on issue #7169: [AIRFLOW-6555] mushroom cloud error when clicking 'task instance details' from graph view of dag that has never been run yet
tooptoop4 commented on issue #7169: [AIRFLOW-6555] mushroom cloud error when clicking 'task instance details' from graph view of dag that has never been run yet URL: https://github.com/apache/airflow/pull/7169#issuecomment-577847502 @kaxil pls merge This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6620) Mock celery in worker cli test
[ https://issues.apache.org/jira/browse/AIRFLOW-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022427#comment-17022427 ] ASF GitHub Bot commented on AIRFLOW-6620: - kaxil commented on pull request #7243: [AIRFLOW-6620] Mock celery in worker cli test URL: https://github.com/apache/airflow/pull/7243 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Mock celery in worker cli test > -- > > Key: AIRFLOW-6620 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6620 > Project: Apache Airflow > Issue Type: Improvement > Components: celery, tests >Affects Versions: 1.10.8 >Reporter: Tomasz Urbaszek >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6620) Mock celery in worker cli test
[ https://issues.apache.org/jira/browse/AIRFLOW-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-6620. - Fix Version/s: 1.10.8 Resolution: Fixed > Mock celery in worker cli test > -- > > Key: AIRFLOW-6620 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6620 > Project: Apache Airflow > Issue Type: Improvement > Components: celery, tests >Affects Versions: 1.10.8 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 1.10.8 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6620) Mock celery in worker cli test
[ https://issues.apache.org/jira/browse/AIRFLOW-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022428#comment-17022428 ] ASF subversion and git services commented on AIRFLOW-6620: -- Commit 61201a5e5cf2ece1253fccdde0dd5833267d9ee2 in airflow's branch refs/heads/master from Tomek Urbaszek [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=61201a5 ] [AIRFLOW-6620] Mock celery in worker cli test (#7243) Actual tests start celery worker and doesn't stop it. Sometimes it result in timeout of a test. > Mock celery in worker cli test > -- > > Key: AIRFLOW-6620 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6620 > Project: Apache Airflow > Issue Type: Improvement > Components: celery, tests >Affects Versions: 1.10.8 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 1.10.8 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil merged pull request #7243: [AIRFLOW-6620] Mock celery in worker cli test
kaxil merged pull request #7243: [AIRFLOW-6620] Mock celery in worker cli test URL: https://github.com/apache/airflow/pull/7243 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7243: [AIRFLOW-6620] Mock celery in worker cli test
codecov-io commented on issue #7243: [AIRFLOW-6620] Mock celery in worker cli test URL: https://github.com/apache/airflow/pull/7243#issuecomment-577843198 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7243?src=pr=h1) Report > Merging [#7243](https://codecov.io/gh/apache/airflow/pull/7243?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/d5386794948f6fd9a07cb8969f05ecbc194b8170?src=pr=desc) will **decrease** coverage by `0.27%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/7243?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7243 +/- ## == - Coverage 85.34% 85.07% -0.28% == Files 791 791 Lines 4013440135 +1 == - Hits3425434143 -111 - Misses 5880 5992 +112 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7243?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7243/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7243/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7243/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/7243/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/7243/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `76.47% <0%> (-21.18%)` | :arrow_down: | | [airflow/gcp/operators/bigquery.py](https://codecov.io/gh/apache/airflow/pull/7243/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2JpZ3F1ZXJ5LnB5) | `91.62% <0%> (+0.02%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7243?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7243?src=pr=footer). Last update [d538679...d8b1c0b](https://codecov.io/gh/apache/airflow/pull/7243?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] abdulbasitds commented on issue #6007: Improved and Fixed Apache Glue hook/operator/sensor from [AIRFLOW-2310]
abdulbasitds commented on issue #6007: Improved and Fixed Apache Glue hook/operator/sensor from [AIRFLOW-2310] URL: https://github.com/apache/airflow/pull/6007#issuecomment-577842896 @potiuk thank you for writing the details about the rebase, it seems very helpful. I will try it tommorow and will let you know This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2279) Clearing Tasks Across DAGs
[ https://issues.apache.org/jira/browse/AIRFLOW-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022417#comment-17022417 ] ASF GitHub Bot commented on AIRFLOW-2279: - kaxil commented on pull request #7139: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker URL: https://github.com/apache/airflow/pull/7139 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Clearing Tasks Across DAGs > -- > > Key: AIRFLOW-2279 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2279 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Achal Soni >Assignee: Qian Yu >Priority: Major > Fix For: 1.10.8 > > Attachments: cross_dag_ui_screenshot.png > > > At Stripe, we commonly have discrete dags that depend on each other by > leveraging ExternalTaskSensors. We also find ourselves routinely wanting to > not only clear tasks and their downstream tasks in a particular dag, but also > their downstream tasks in their dependent dags (linked by > ExternalTaskSensors). > We currently have extended Airflow to handle this by modifying the webapp and > cli tool to optionally clear dependent tasks across multiple dags (see > attached screenshot). > We want to open the floor for discussion with the larger Airflow community > about the usage of ExternalTaskSensors and specifically how to handle > clearing across dags. We are interested in learning more about the accepted > practices in this regard, and are very open/willing to contribute in this > area if there is interest! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] kaxil closed pull request #7139: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker
kaxil closed pull request #7139: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker URL: https://github.com/apache/airflow/pull/7139 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #7139: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker
kaxil commented on issue #7139: [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker URL: https://github.com/apache/airflow/pull/7139#issuecomment-577841070 Merged to v1-10-test in 14111b4bd This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6522) Sensors in reschedule mode with S3TaskHandler can cause log duplication
[
https://issues.apache.org/jira/browse/AIRFLOW-6522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022412#comment-17022412
]
ASF subversion and git services commented on AIRFLOW-6522:
--
Commit c7ad2c3f1a5c3fd46edbbc390b1229e01e64 in airflow's branch
refs/heads/v1-10-test from rconroy293
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c7ad2c3 ]
[AIRFLOW-6522] Clear task log file before starting to fix duplication in
S3TaskHandler (#7120)
The same task instance (including try number) can be run on a worker
when using a sensor in "reschedule" mode. Accordingly, this clears the
local log file when re-initializing the logger so that the old log
lines aren't uploaded again when the logger is closed.
(cherry picked from commit 88608caa56bf3621807af860a6a378242220de47)
> Sensors in reschedule mode with S3TaskHandler can cause log duplication
> ---
>
> Key: AIRFLOW-6522
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6522
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.10.6
>Reporter: Robert Conroy
>Assignee: Robert Conroy
>Priority: Minor
> Fix For: 1.10.8
>
>
> With sensors using {{reschedule}} mode and {{S3TaskHandler}} for logging, the
> task instance log gets a bunch of duplicate messages. I believe this is
> happening because contents of the local log file are appended to what's
> already in S3. The local log file may contain log messages that have already
> been uploaded to S3 if the task is sent back to a worker that had already
> processed a poke for that task instance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5501) in_cluster default value in KubernetesPodOperator overwrites configuration
[
https://issues.apache.org/jira/browse/AIRFLOW-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022409#comment-17022409
]
ASF subversion and git services commented on AIRFLOW-5501:
--
Commit 645e021303ad460ebca213ae5de56ba787c328f4 in airflow's branch
refs/heads/v1-10-test from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=645e021 ]
[AIRFLOW-5501] Make default `in_cluster` value in KubernetesPodOperator respect
config (#6124)
(cherry-picked from e54fba5b4)
> in_cluster default value in KubernetesPodOperator overwrites configuration
> --
>
> Key: AIRFLOW-5501
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5501
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 1.10.5
>Reporter: Quentin Lemaire
>Priority: Major
> Fix For: 1.10.8
>
>
> Hi!
> The default value of the parameter *in_cluster* of the
> *kube_client.get_kube_client* function is
> *in_cluster=conf.getboolean('kubernetes', 'in_cluster').* Therefore, the
> expected behavior is that when, *in_cluster* is not set, it takes the value
> in the configuration file.
> However, the default value of *in_cluster* in *KubernetesPodOperator.py* is
> False and *in_cluster* is passed as a parameter when calling the
> *kube_client.get_kube_client* function. Therefore, it changes the expecting
> behavior by overwritting the default value. When *in_cluster* is not set when
> initializing *KubernetesPodOperator*, the value of *in_cluster* in
> *kube_client.get_kube_client* is False and not the value which is in the
> configuration file.
> It is quite confusing because it can feel like the value in the configuration
> file is not working properly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7234: [AIRFLOW-6493] Add SSL configuration to Redis hook connections
nuclearpinguin commented on a change in pull request #7234: [AIRFLOW-6493] Add
SSL configuration to Redis hook connections
URL: https://github.com/apache/airflow/pull/7234#discussion_r370297981
##
File path: airflow/providers/redis/hooks/redis.py
##
@@ -54,6 +58,13 @@ def get_conn(self):
self.password = None if str(conn.password).lower() in ['none',
'false', ''] else conn.password
self.db = conn.extra_dejson.get('db', None)
+# check for ssl parameters in conn.extra
+ssl_args = {}
+for arg_name, arg_val in conn.extra_dejson.items():
+if arg_name in ["ssl", "ssl_cert_reqs", "ssl_ca_certs",
"ssl_keyfile", "ssl_cert_file",
+"ssl_check_hostname"]:
+ssl_args[arg_name] = arg_val
Review comment:
```python
ssl_args_names = ["ssl", "ssl_cert_reqs", "ssl_ca_certs", "ssl_keyfile",
"ssl_cert_file", "ssl_check_hostname"]
ssl_args = {n: v for n, v conn.extra_dejson.items() if n in ssl_args_names}
```
What do you think about simplifying this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-6608) Change logging level to DEBUG for PythonOperator Env exports
[
https://issues.apache.org/jira/browse/AIRFLOW-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022399#comment-17022399
]
ASF GitHub Bot commented on AIRFLOW-6608:
-
kaxil commented on pull request #7246: [AIRFLOW-6608] Change logging level for
PythonOperator Env exports
URL: https://github.com/apache/airflow/pull/7246
While working on #7229 during resolving conflicts, I updated it for
BashOperator instead of PythonOperator.
---
Issue link: WILL BE INSERTED BY
[boring-cyborg](https://github.com/kaxil/boring-cyborg)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- =
JIRA ID*
- [x] Unit tests coverage for changes (not needed for documentation changes)
- [x] Commits follow "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [x] I will engage committers as explained in [Contribution Workflow
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
* For document-only changes commit message can start with
`[AIRFLOW-]`.
---
In case of fundamental code change, Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
for more information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Change logging level to DEBUG for PythonOperator Env exports
>
>
> Key: AIRFLOW-6608
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6608
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Affects Versions: 1.10.7
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 1.10.8
>
>
> The following logs just create noise and add nothing for the end-users. We
> should change this to DEBUG
> {noformat}
> [2020-01-20 17:58:27,364] {python_operator.py:105} INFO - Exporting the
> following env vars:
> AIRFLOW_CTX_DAG_EMAIL=a...@mail.com
> AIRFLOW_CTX_DAG_OWNER=me
> AIRFLOW_CTX_DAG_ID=my_dag_id
> AIRFLOW_CTX_TASK_ID=my_task_id
> AIRFLOW_CTX_EXECUTION_DATE=2020-01-20T12:27:48.408593+00:00
> AIRFLOW_CTX_DAG_RUN_ID=manual__2020-01-20T12:27:48.408593+00:00
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] kaxil opened a new pull request #7246: [AIRFLOW-6608] Change logging level for PythonOperator Env exports
kaxil opened a new pull request #7246: [AIRFLOW-6608] Change logging level for PythonOperator Env exports URL: https://github.com/apache/airflow/pull/7246 While working on #7229 during resolving conflicts, I updated it for BashOperator instead of PythonOperator. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #7245: [AIRFLOW-6624] Improve webserver command with pidfile checking
codecov-io commented on issue #7245: [AIRFLOW-6624] Improve webserver command with pidfile checking URL: https://github.com/apache/airflow/pull/7245#issuecomment-577833662 # [Codecov](https://codecov.io/gh/apache/airflow/pull/7245?src=pr=h1) Report > Merging [#7245](https://codecov.io/gh/apache/airflow/pull/7245?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/af91d9f4f08a6bf308cb10418669d17d1ae6cc44?src=pr=desc) will **decrease** coverage by `52.79%`. > The diff coverage is `20.58%`. [](https://codecov.io/gh/apache/airflow/pull/7245?src=pr=tree) ```diff @@Coverage Diff @@ ## master#7245 +/- ## == - Coverage 85.34% 32.54% -52.8% == Files 791 790 -1 Lines 4012840125 -3 == - Hits3424713059 -21188 - Misses 588127066 +21185 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7245?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/cli/commands/webserver\_command.py](https://codecov.io/gh/apache/airflow/pull/7245/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvd2Vic2VydmVyX2NvbW1hbmQucHk=) | `15.49% <15%> (-39.85%)` | :arrow_down: | | [airflow/utils/process\_utils.py](https://codecov.io/gh/apache/airflow/pull/7245/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9wcm9jZXNzX3V0aWxzLnB5) | `32.14% <28.57%> (-67.86%)` | :arrow_down: | | [...low/contrib/operators/wasb\_delete\_blob\_operator.py](https://codecov.io/gh/apache/airflow/pull/7245/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy93YXNiX2RlbGV0ZV9ibG9iX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [...flow/contrib/example\_dags/example\_qubole\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7245/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9zZW5zb3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [...ample\_dags/example\_emr\_job\_flow\_automatic\_steps.py](https://codecov.io/gh/apache/airflow/pull/7245/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX2Vtcl9qb2JfZmxvd19hdXRvbWF0aWNfc3RlcHMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [...contrib/example\_dags/example\_papermill\_operator.py](https://codecov.io/gh/apache/airflow/pull/7245/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3BhcGVybWlsbF9vcGVyYXRvci5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/hooks/mssql\_hook.py](https://codecov.io/gh/apache/airflow/pull/7245/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9tc3NxbF9ob29rLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/hooks/webhdfs\_hook.py](https://codecov.io/gh/apache/airflow/pull/7245/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy93ZWJoZGZzX2hvb2sucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/contrib/sensors/emr\_base\_sensor.py](https://codecov.io/gh/apache/airflow/pull/7245/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvZW1yX2Jhc2Vfc2Vuc29yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/python\_operator.py](https://codecov.io/gh/apache/airflow/pull/7245/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcHl0aG9uX29wZXJhdG9yLnB5) | `0% <0%> (-100%)` | :arrow_down: | | ... and [649 more](https://codecov.io/gh/apache/airflow/pull/7245/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7245?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7245?src=pr=footer). Last update [af91d9f...d44accd](https://codecov.io/gh/apache/airflow/pull/7245?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (AIRFLOW-6460) Reduce timeout in pytest
[ https://issues.apache.org/jira/browse/AIRFLOW-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik closed AIRFLOW-6460. --- Resolution: Fixed > Reduce timeout in pytest > > > Key: AIRFLOW-6460 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6460 > Project: Apache Airflow > Issue Type: Improvement > Components: tests >Affects Versions: 1.10.7 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6608) Change logging level to DEBUG for PythonOperator Env exports
[
https://issues.apache.org/jira/browse/AIRFLOW-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022397#comment-17022397
]
ASF subversion and git services commented on AIRFLOW-6608:
--
Commit 83447d31562e7bb527cbf47774d5dc84fd6cbd0a in airflow's branch
refs/heads/v1-10-test from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=83447d3 ]
[AIRFLOW-6608] Change logging level for Bash & PyOperator Env exports
> Change logging level to DEBUG for PythonOperator Env exports
>
>
> Key: AIRFLOW-6608
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6608
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
>Affects Versions: 1.10.7
>Reporter: Kaxil Naik
>Assignee: Kaxil Naik
>Priority: Minor
> Fix For: 1.10.8
>
>
> The following logs just create noise and add nothing for the end-users. We
> should change this to DEBUG
> {noformat}
> [2020-01-20 17:58:27,364] {python_operator.py:105} INFO - Exporting the
> following env vars:
> AIRFLOW_CTX_DAG_EMAIL=a...@mail.com
> AIRFLOW_CTX_DAG_OWNER=me
> AIRFLOW_CTX_DAG_ID=my_dag_id
> AIRFLOW_CTX_TASK_ID=my_task_id
> AIRFLOW_CTX_EXECUTION_DATE=2020-01-20T12:27:48.408593+00:00
> AIRFLOW_CTX_DAG_RUN_ID=manual__2020-01-20T12:27:48.408593+00:00
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2279) Clearing Tasks Across DAGs
[ https://issues.apache.org/jira/browse/AIRFLOW-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022350#comment-17022350 ] ASF subversion and git services commented on AIRFLOW-2279: -- Commit 14111b4bddbaa7c7085a25cdfe923f88219e1582 in airflow's branch refs/heads/v1-10-test from Qian Yu [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=14111b4 ] [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker (#6633) > Clearing Tasks Across DAGs > -- > > Key: AIRFLOW-2279 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2279 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Achal Soni >Assignee: Qian Yu >Priority: Major > Fix For: 1.10.8 > > Attachments: cross_dag_ui_screenshot.png > > > At Stripe, we commonly have discrete dags that depend on each other by > leveraging ExternalTaskSensors. We also find ourselves routinely wanting to > not only clear tasks and their downstream tasks in a particular dag, but also > their downstream tasks in their dependent dags (linked by > ExternalTaskSensors). > We currently have extended Airflow to handle this by modifying the webapp and > cli tool to optionally clear dependent tasks across multiple dags (see > attached screenshot). > We want to open the floor for discussion with the larger Airflow community > about the usage of ExternalTaskSensors and specifically how to handle > clearing across dags. We are interested in learning more about the accepted > practices in this regard, and are very open/willing to contribute in this > area if there is interest! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7245: [AIRFLOW-6624] Improve webserver command with pidfile checking
nuclearpinguin commented on a change in pull request #7245: [AIRFLOW-6624]
Improve webserver command with pidfile checking
URL: https://github.com/apache/airflow/pull/7245#discussion_r370287859
##
File path: airflow/cli/commands/webserver_command.py
##
@@ -195,15 +197,11 @@ def webserver(args):
port=args.port, host=args.hostname,
ssl_context=(ssl_cert, ssl_key) if ssl_cert and ssl_key else
None)
else:
-os.environ['SKIP_DAGS_PARSING'] = 'True'
-app = cached_app(None)
-pid, stdout, stderr, log_file = setup_locations(
+pid_file, stdout, stderr, log_file = setup_locations(
"webserver", args.pid, args.stdout, args.stderr, args.log_file)
-os.environ.pop('SKIP_DAGS_PARSING')
Review comment:
The `app` was unused and I see no reason for those env changes when we
remove the app.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7245: [AIRFLOW-6624] Improve webserver command with pidfile checking
nuclearpinguin commented on a change in pull request #7245: [AIRFLOW-6624]
Improve webserver command with pidfile checking
URL: https://github.com/apache/airflow/pull/7245#discussion_r370288287
##
File path: airflow/cli/commands/webserver_command.py
##
@@ -263,40 +264,31 @@ def monitor_gunicorn(gunicorn_master_proc):
sys.exit(gunicorn_master_proc.returncode)
if args.daemon:
-base, ext = os.path.splitext(pid)
-ctx = daemon.DaemonContext(
-pidfile=TimeoutPIDLockFile(base + "-monitor" + ext, -1),
-files_preserve=[handle],
-stdout=stdout,
-stderr=stderr,
-signal_map={
-signal.SIGINT: kill_proc,
-signal.SIGTERM: kill_proc
-},
Review comment:
I moved the signal handling to `monitor_gunicorn` function.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin merged pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker
nuclearpinguin merged pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker URL: https://github.com/apache/airflow/pull/7206 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-6342) EmrAddStepsOperator broken reference and faulty test
[ https://issues.apache.org/jira/browse/AIRFLOW-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-6342: Fix Version/s: (was: 1.10.8) > EmrAddStepsOperator broken reference and faulty test > > > Key: AIRFLOW-6342 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6342 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Affects Versions: 1.10.7 >Reporter: Aviem Zur >Assignee: Aviem Zur >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6460) Reduce timeout in pytest
[ https://issues.apache.org/jira/browse/AIRFLOW-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-6460: Fix Version/s: (was: 1.10.8) > Reduce timeout in pytest > > > Key: AIRFLOW-6460 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6460 > Project: Apache Airflow > Issue Type: Improvement > Components: tests >Affects Versions: 1.10.7 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] nuclearpinguin opened a new pull request #7245: [AIRFLOW-6624] Improve webserver command with pidfile checking
nuclearpinguin opened a new pull request #7245: [AIRFLOW-6624] Improve webserver command with pidfile checking URL: https://github.com/apache/airflow/pull/7245 When running webserver as daemon it can happen that it will not start due to existing pidfile. This PR improves whole webserver command and adds pidfile checking. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Reopened] (AIRFLOW-6460) Reduce timeout in pytest
[ https://issues.apache.org/jira/browse/AIRFLOW-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik reopened AIRFLOW-6460: - > Reduce timeout in pytest > > > Key: AIRFLOW-6460 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6460 > Project: Apache Airflow > Issue Type: Improvement > Components: tests >Affects Versions: 1.10.7 >Reporter: Kamil Bregula >Assignee: Kamil Bregula >Priority: Major > Fix For: 1.10.8 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6591) Add cli option to stop celery worker
[ https://issues.apache.org/jira/browse/AIRFLOW-6591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6591. -- Fix Version/s: 2.0.0 Resolution: Done > Add cli option to stop celery worker > > > Key: AIRFLOW-6591 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6591 > Project: Apache Airflow > Issue Type: New Feature > Components: celery, cli >Affects Versions: 1.10.8 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6467) Remove unnecessary local variable
[ https://issues.apache.org/jira/browse/AIRFLOW-6467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022388#comment-17022388 ] ASF subversion and git services commented on AIRFLOW-6467: -- Commit a8df08cebc9b1ff4ac5cac5290bca67928453a48 in airflow's branch refs/heads/v1-10-test from Mustafa Gök [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=a8df08c ] [AIRFLOW-6467] Use self.dag i/o creating a new one (#7067) (cherry picked from commit f4c7d17902c87bff50bbd8c94028a539e4e1c3a3) > Remove unnecessary local variable > - > > Key: AIRFLOW-6467 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6467 > Project: Apache Airflow > Issue Type: Test > Components: tests >Affects Versions: 1.10.7 >Reporter: Mustafa Gök >Assignee: Mustafa Gök >Priority: Trivial > Fix For: 1.10.8 > > > In test_templated_sensor (tests/sensors/test_external_task_sensor.py) unit > test method, self.dag can be used instead of creating a local variable named > "dag" (line 98). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6490) Improve time delta comparison in local task job tests
[ https://issues.apache.org/jira/browse/AIRFLOW-6490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022387#comment-17022387 ] ASF subversion and git services commented on AIRFLOW-6490: -- Commit e9183a4b3c4d3fc65ba14d18af89391234c0a9b7 in airflow's branch refs/heads/v1-10-test from Tomasz Urbaszek [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e9183a4 ] [AIRFLOW-6490] Improve time delta comparison in local task job tests (#7083) (cherry picked from commit 2754427c844319cf16998eb2b621fbbeed706eb9) > Improve time delta comparison in local task job tests > - > > Key: AIRFLOW-6490 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6490 > Project: Apache Airflow > Issue Type: Test > Components: tests >Affects Versions: 1.10.8 >Reporter: Tomasz Urbaszek >Priority: Minor > Fix For: 1.10.8 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6381) DAG name changes style of some columns but not all
[ https://issues.apache.org/jira/browse/AIRFLOW-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022386#comment-17022386 ] ASF subversion and git services commented on AIRFLOW-6381: -- Commit adc8b0ca5d6c0b6cf6887465f4dc2a87bfe3aa95 in airflow's branch refs/heads/v1-10-test from Jay [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=adc8b0c ] [AIRFLOW-6381] Remove styling based on DAG id from DAGs page (#6985) > DAG name changes style of some columns but not all > -- > > Key: AIRFLOW-6381 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6381 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.2 >Reporter: Jay K >Assignee: Jay K >Priority: Trivial > Fix For: 1.10.8 > > Attachments: image-2019-12-27-22-38-55-209.png > > > !image-2019-12-27-22-38-55-209.png|width=782,height=114! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6514) AIRFLOW-5695 didn't make it into release
[ https://issues.apache.org/jira/browse/AIRFLOW-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-6514. - Resolution: Fixed Backported to old UI too https://github.com/apache/airflow/commit/4bd2f574cd527c634e0ddeb607a35e363d975524 > AIRFLOW-5695 didn't make it into release > > > Key: AIRFLOW-6514 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6514 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Affects Versions: 1.10.7 >Reporter: david liu >Priority: Minor > Fix For: 1.10.8 > > > Ticket AIRFLOW-5695 is listed as being included in the 1.10.7 release in the > changelogs, but appears to have omitted. There's no reference to this fix in > the commits > [https://github.com/apache/airflow/commits/1.10.7/airflow/www/views.py] > And an inspection of the file shows that the master is updated but the > release is not > [https://github.com/apache/airflow/blob/1.10.7/airflow/www/views.py#L1134] > [https://github.com/apache/airflow/blob/master/airflow/www/views.py#L897] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6514) AIRFLOW-5695 didn't make it into release
[ https://issues.apache.org/jira/browse/AIRFLOW-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022389#comment-17022389 ] ASF subversion and git services commented on AIRFLOW-6514: -- Commit 4bd2f574cd527c634e0ddeb607a35e363d975524 in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=4bd2f57 ] [AIRFLOW-6514] use RUNNING_DEPS to check run from UI (#6367) > AIRFLOW-5695 didn't make it into release > > > Key: AIRFLOW-6514 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6514 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Affects Versions: 1.10.7 >Reporter: david liu >Priority: Minor > Fix For: 1.10.8 > > > Ticket AIRFLOW-5695 is listed as being included in the 1.10.7 release in the > changelogs, but appears to have omitted. There's no reference to this fix in > the commits > [https://github.com/apache/airflow/commits/1.10.7/airflow/www/views.py] > And an inspection of the file shows that the master is updated but the > release is not > [https://github.com/apache/airflow/blob/1.10.7/airflow/www/views.py#L1134] > [https://github.com/apache/airflow/blob/master/airflow/www/views.py#L897] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6414) Create a configurations page in airflow documentation
[ https://issues.apache.org/jira/browse/AIRFLOW-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-6414: Fix Version/s: (was: 1.10.8) > Create a configurations page in airflow documentation > - > > Key: AIRFLOW-6414 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6414 > Project: Apache Airflow > Issue Type: Improvement > Components: documentation >Affects Versions: 1.10.7 >Reporter: Daniel Imberman >Assignee: Daniel Imberman >Priority: Major > > Many questions on the airflow slack and the airflow mailing list come down to > people not knowing which configurations can best help their airflow clusters > run optimally. There appears to be no single hub of all configuration options > and users sometimes find themselves going into airflow source code to find > configurations. > This will place all configurations in configurations.rst in the > airflow/docs/configurations.rst directory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (AIRFLOW-4242) Missing description of statsd metric keys
[ https://issues.apache.org/jira/browse/AIRFLOW-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik reopened AIRFLOW-4242: - > Missing description of statsd metric keys > - > > Key: AIRFLOW-4242 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4242 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation >Reporter: Kamil Bregula >Priority: Trivial > Fix For: 1.10.8 > > > Missing keys: `local_task_job_heartbeat_failure`, > `local_task_job_prolonged_heartbeat_failure`, > `task_removed_from_dag.`, `task_restored_to_dag.`, > `task_instance_created-`, `dag_file_refresh_error` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] nuclearpinguin merged pull request #7242: [AIRFLOW-6619] Add cluster_fields to BigQueryCreateEmptyTableOperator
nuclearpinguin merged pull request #7242: [AIRFLOW-6619] Add cluster_fields to BigQueryCreateEmptyTableOperator URL: https://github.com/apache/airflow/pull/7242 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6619) Add cluster_fields to BigQueryCreateEmptyTableOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022376#comment-17022376 ] ASF subversion and git services commented on AIRFLOW-6619: -- Commit a8307392e13debe659f578f39396767de684e9e8 in airflow's branch refs/heads/master from Kamil Gałuszka [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=a830739 ] [AIRFLOW-6619] Add cluster_fields to BigQueryCreateEmptyTableOperator (#7242) > Add cluster_fields to BigQueryCreateEmptyTableOperator > -- > > Key: AIRFLOW-6619 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6619 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: master >Reporter: Kamil Gałuszka >Assignee: Kamil Gałuszka >Priority: Major > Fix For: 2.0.0 > > Original Estimate: 1.5h > Remaining Estimate: 1.5h > > cluster_fields is available in BigQueryHook but it's unavailable in > BigQueryCreateEmptyTableOperator. > This ticket is to add that functionality into existing operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-6619) Add cluster_fields to BigQueryCreateEmptyTableOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Urbaszek resolved AIRFLOW-6619. -- Resolution: Done > Add cluster_fields to BigQueryCreateEmptyTableOperator > -- > > Key: AIRFLOW-6619 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6619 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: master >Reporter: Kamil Gałuszka >Assignee: Kamil Gałuszka >Priority: Major > Fix For: 2.0.0 > > Original Estimate: 1.5h > Remaining Estimate: 1.5h > > cluster_fields is available in BigQueryHook but it's unavailable in > BigQueryCreateEmptyTableOperator. > This ticket is to add that functionality into existing operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6619) Add cluster_fields to BigQueryCreateEmptyTableOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022375#comment-17022375 ] ASF GitHub Bot commented on AIRFLOW-6619: - nuclearpinguin commented on pull request #7242: [AIRFLOW-6619] Add cluster_fields to BigQueryCreateEmptyTableOperator URL: https://github.com/apache/airflow/pull/7242 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add cluster_fields to BigQueryCreateEmptyTableOperator > -- > > Key: AIRFLOW-6619 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6619 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: master >Reporter: Kamil Gałuszka >Assignee: Kamil Gałuszka >Priority: Major > Fix For: 2.0.0 > > Original Estimate: 1.5h > Remaining Estimate: 1.5h > > cluster_fields is available in BigQueryHook but it's unavailable in > BigQueryCreateEmptyTableOperator. > This ticket is to add that functionality into existing operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6591) Add cli option to stop celery worker
[ https://issues.apache.org/jira/browse/AIRFLOW-6591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022364#comment-17022364 ] ASF subversion and git services commented on AIRFLOW-6591: -- Commit d5386794948f6fd9a07cb8969f05ecbc194b8170 in airflow's branch refs/heads/master from Tomasz Urbaszek [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d538679 ] [AIRFLOW-6591] Add cli option to stop celery worker (#7206) Now users can gracefully stop the Celery worker by sending a SIGTERM signal to Celery main process. > Add cli option to stop celery worker > > > Key: AIRFLOW-6591 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6591 > Project: Apache Airflow > Issue Type: New Feature > Components: celery, cli >Affects Versions: 1.10.8 >Reporter: Tomasz Urbaszek >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6591) Add cli option to stop celery worker
[ https://issues.apache.org/jira/browse/AIRFLOW-6591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022363#comment-17022363 ] ASF GitHub Bot commented on AIRFLOW-6591: - nuclearpinguin commented on pull request #7206: [AIRFLOW-6591] Add cli option to stop celery worker URL: https://github.com/apache/airflow/pull/7206 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add cli option to stop celery worker > > > Key: AIRFLOW-6591 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6591 > Project: Apache Airflow > Issue Type: New Feature > Components: celery, cli >Affects Versions: 1.10.8 >Reporter: Tomasz Urbaszek >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-4242) Missing description of statsd metric keys
[ https://issues.apache.org/jira/browse/AIRFLOW-4242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik closed AIRFLOW-4242. --- Resolution: Fixed > Missing description of statsd metric keys > - > > Key: AIRFLOW-4242 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4242 > Project: Apache Airflow > Issue Type: New Feature > Components: documentation >Reporter: Kamil Bregula >Priority: Trivial > > Missing keys: `local_task_job_heartbeat_failure`, > `local_task_job_prolonged_heartbeat_failure`, > `task_removed_from_dag.`, `task_restored_to_dag.`, > `task_instance_created-`, `dag_file_refresh_error` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6624) Improve webserver command with pidfile checking
[ https://issues.apache.org/jira/browse/AIRFLOW-6624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022360#comment-17022360 ] ASF GitHub Bot commented on AIRFLOW-6624: - nuclearpinguin commented on pull request #7245: [AIRFLOW-6624] Improve webserver command with pidfile checking URL: https://github.com/apache/airflow/pull/7245 When running webserver as daemon it can happen that it will not start due to existing pidfile. This PR improves whole webserver command and adds pidfile checking. --- Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg) Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Commit message/PR title starts with `[AIRFLOW-]`. AIRFLOW- = JIRA ID* - [x] Unit tests coverage for changes (not needed for documentation changes) - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" - [x] Relevant documentation is updated including usage instructions. - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). * For document-only changes commit message can start with `[AIRFLOW-]`. --- In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve webserver command with pidfile checking > --- > > Key: AIRFLOW-6624 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6624 > Project: Apache Airflow > Issue Type: Improvement > Components: cli >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6428) Fix import path for airflow.utils.dates.days_ago in Example DAGs
[ https://issues.apache.org/jira/browse/AIRFLOW-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-6428: Fix Version/s: (was: 1.10.8) > Fix import path for airflow.utils.dates.days_ago in Example DAGs > > > Key: AIRFLOW-6428 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6428 > Project: Apache Airflow > Issue Type: Improvement > Components: utils >Affects Versions: 1.10.7 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Major > Fix For: 2.0.0 > > Attachments: image-2020-01-02-15-33-55-601.png > > > Currently, without the entry in __init__.py, IDEs show that it could not find > the reference to *dates* and hence if you try to find the reference for > *days_ago* function or *dates* modules it can't find it. > Check attachment -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6624) Improve webserver command with pidfile checking
Tomasz Urbaszek created AIRFLOW-6624: Summary: Improve webserver command with pidfile checking Key: AIRFLOW-6624 URL: https://issues.apache.org/jira/browse/AIRFLOW-6624 Project: Apache Airflow Issue Type: Improvement Components: cli Affects Versions: 2.0.0 Reporter: Tomasz Urbaszek -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-2279) Clearing Tasks Across DAGs
[ https://issues.apache.org/jira/browse/AIRFLOW-2279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-2279: Fix Version/s: (was: 2.0.0) 1.10.8 > Clearing Tasks Across DAGs > -- > > Key: AIRFLOW-2279 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2279 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Achal Soni >Assignee: Qian Yu >Priority: Major > Fix For: 1.10.8 > > Attachments: cross_dag_ui_screenshot.png > > > At Stripe, we commonly have discrete dags that depend on each other by > leveraging ExternalTaskSensors. We also find ourselves routinely wanting to > not only clear tasks and their downstream tasks in a particular dag, but also > their downstream tasks in their dependent dags (linked by > ExternalTaskSensors). > We currently have extended Airflow to handle this by modifying the webapp and > cli tool to optionally clear dependent tasks across multiple dags (see > attached screenshot). > We want to open the floor for discussion with the larger Airflow community > about the usage of ExternalTaskSensors and specifically how to handle > clearing across dags. We are interested in learning more about the accepted > practices in this regard, and are very open/willing to contribute in this > area if there is interest! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6359) spark_submit_hook.py polling too frequent
[
https://issues.apache.org/jira/browse/AIRFLOW-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022332#comment-17022332
]
ASF subversion and git services commented on AIRFLOW-6359:
--
Commit 8473c89ca55aa572bcde81bb39f2264a48201e15 in airflow's branch
refs/heads/v1-10-test from tooptoop4
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=8473c89 ]
[AIRFLOW-6359] spark_submit_hook.py status polling interval config (#6909)
* AIRFLOW-6359
* pep checks
* make travis CI refire
* new section = bad
* move poll_interval into the operator instead of .cfg
* review comments
(cherry picked from commit 83ba8e638fb539afd6dac495a8c59012d0aaa9f4)
> spark_submit_hook.py polling too frequent
> -
>
> Key: AIRFLOW-6359
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6359
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.6
>Reporter: t oo
>Assignee: t oo
>Priority: Major
> Fix For: 1.10.8
>
>
> spark polling every second is too frequent, use config to change to desired
> interval, ie 15 seconds
> contrib/hooks/spark_submit_hook.py Poll spark server at a custom interval
> instead of every second
>
> BEFORE
> Sleep for 1 second as we do not want to spam the cluster
> time.sleep(1)
>
> AFTER
> import airflow
> from airflow import configuration as conf
> Sleep for n second as we do not want to spam the cluster
> _poll_interval = conf.getint('sparksubmit', 'poll_interval')
> time.sleep(_poll_interval)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6519) Make TaskInstance logs constants in Webserver configurable
[ https://issues.apache.org/jira/browse/AIRFLOW-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022330#comment-17022330 ] ASF subversion and git services commented on AIRFLOW-6519: -- Commit 9f4908d0f933005f5cf8e676189a9a759403904a in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=9f4908d ] [AIRFLOW-6519] Make TI logs constants in Webserver configurable (#7113) (cherry-picked from c880221d7) > Make TaskInstance logs constants in Webserver configurable > --- > > Key: AIRFLOW-6519 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6519 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Affects Versions: 1.10.7 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Minor > Fix For: 1.10.8 > > > Some of the TaskInstance logs constants in the webserver (in ti_log.html) can > be made configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6327) http_hook: Accept json= parameter for payload
[
https://issues.apache.org/jira/browse/AIRFLOW-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022329#comment-17022329
]
ASF subversion and git services commented on AIRFLOW-6327:
--
Commit c85ea3e38a70c2b524c42caa4d5d88ad1ddd51ca in airflow's branch
refs/heads/v1-10-test from Bjorn Olsen
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c85ea3e ]
[AIRFLOW-6327] http_hook: Accept json= parameter for payload (#6886)
* [AIRFLOW-6327] http_hook: Accept json= parameter for payload
* [AIRFLOW-6327] CODE REVIEW
* [AIRFLOW-6327] CODE REVIEW 2
* [AIRFLOW-6327] CODE REVIEW 3
Co-Authored-By: dstandish
Co-authored-by: dstandish
(cherry picked from commit fd78c65cabae2241a4c1d3a792e00620049cbf3e)
> http_hook: Accept json= parameter for payload
> -
>
> Key: AIRFLOW-6327
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6327
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.6
>Reporter: Bjorn Olsen
>Assignee: Bjorn Olsen
>Priority: Minor
> Fix For: 1.10.8
>
>
> Python requests library allows a user to send json-encoded Python objects by
> making use of the "json=" parameter to a request.
> This handles JSON encoding of the payload and setting the correct content
> type header.
> Example:
> {code:python}
> response = requests.post('https://httpbin.org/post', json={'key':'value'})
> json_response = response.json()
> json_response['data'] '{"key": "value"}'
> json_response['headers']['Content-Type'] 'application/json'
> {code}
> http_hook.run() does not yet have the json= parameter which is inconvenient
> as then we have to pass the "data=" parameter using json.dumps and specify
> the correct headers.
> It would be better if we can just do something like the below and let the
> Requests library ensure the request is valid:
> {code:python}
> obj = {'a':1, 'b': 'abc', 'c': [1, 2, {"d":10}]}
> response = hook.run(
> endpoint,
> json=obj)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6327) http_hook: Accept json= parameter for payload
[
https://issues.apache.org/jira/browse/AIRFLOW-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022325#comment-17022325
]
ASF subversion and git services commented on AIRFLOW-6327:
--
Commit c85ea3e38a70c2b524c42caa4d5d88ad1ddd51ca in airflow's branch
refs/heads/v1-10-test from Bjorn Olsen
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c85ea3e ]
[AIRFLOW-6327] http_hook: Accept json= parameter for payload (#6886)
* [AIRFLOW-6327] http_hook: Accept json= parameter for payload
* [AIRFLOW-6327] CODE REVIEW
* [AIRFLOW-6327] CODE REVIEW 2
* [AIRFLOW-6327] CODE REVIEW 3
Co-Authored-By: dstandish
Co-authored-by: dstandish
(cherry picked from commit fd78c65cabae2241a4c1d3a792e00620049cbf3e)
> http_hook: Accept json= parameter for payload
> -
>
> Key: AIRFLOW-6327
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6327
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.6
>Reporter: Bjorn Olsen
>Assignee: Bjorn Olsen
>Priority: Minor
> Fix For: 1.10.8
>
>
> Python requests library allows a user to send json-encoded Python objects by
> making use of the "json=" parameter to a request.
> This handles JSON encoding of the payload and setting the correct content
> type header.
> Example:
> {code:python}
> response = requests.post('https://httpbin.org/post', json={'key':'value'})
> json_response = response.json()
> json_response['data'] '{"key": "value"}'
> json_response['headers']['Content-Type'] 'application/json'
> {code}
> http_hook.run() does not yet have the json= parameter which is inconvenient
> as then we have to pass the "data=" parameter using json.dumps and specify
> the correct headers.
> It would be better if we can just do something like the below and let the
> Requests library ensure the request is valid:
> {code:python}
> obj = {'a':1, 'b': 'abc', 'c': [1, 2, {"d":10}]}
> response = hook.run(
> endpoint,
> json=obj)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6316) Use sphinx render in tutorial.rst
[ https://issues.apache.org/jira/browse/AIRFLOW-6316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022331#comment-17022331 ] ASF subversion and git services commented on AIRFLOW-6316: -- Commit d27c963f955f740e886263b80507ab61ece0e0ca in airflow's branch refs/heads/v1-10-test from Jiajie Zhong [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d27c963 ] [AIRFLOW-6316] Use exampleinclude directives in tutorial.rst (#6868) (cherry-picked from 072dab62c) > Use sphinx render in tutorial.rst > - > > Key: AIRFLOW-6316 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6316 > Project: Apache Airflow > Issue Type: Improvement > Components: documentation >Affects Versions: 1.10.6 >Reporter: zhongjiajie >Assignee: zhongjiajie >Priority: Major > Fix For: 1.10.8 > > > Recently we hard code in tutorial.rst which > is hard to maintain, such as `set_upstream` > is change to shift in tutorial.py but still > in tutorial.rst. Use sphinx is a better way -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6434) Docker Operator No Longer XComs Result in 1.10.7
[ https://issues.apache.org/jira/browse/AIRFLOW-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022324#comment-17022324 ] ASF subversion and git services commented on AIRFLOW-6434: -- Commit d580172e78e294faf214645fe9efd07a0ae5a054 in airflow's branch refs/heads/v1-10-test from Brian Phillips [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=d580172 ] [AIRFLOW-6434] add return statement back to DockerOperator.execute (#7013) (cherry picked from commit a5030f31b98af24805252f6c47611290aee9c69d) > Docker Operator No Longer XComs Result in 1.10.7 > > > Key: AIRFLOW-6434 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6434 > Project: Apache Airflow > Issue Type: Bug > Components: operators, xcom >Affects Versions: 1.10.7 >Reporter: Brian Phillips >Priority: Trivial > Fix For: 1.10.8 > > > This change > ([https://github.com/apache/airflow/commit/8a6dc6657d2a32ac3979e6478512d518ad5a5212)|https://github.com/apache/airflow/commit/8a6dc6657d2a32ac3979e6478512d518ad5a5212] > introduced a slight (and I believe unintended) change to the Docker Operator > xcom behavior. > Even if xcom_push is True, DockerOperator.execute will not return a value and > thus will not push an xcom value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6238) Dags stats endpoint is slow and returns a large payload
[ https://issues.apache.org/jira/browse/AIRFLOW-6238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022321#comment-17022321 ] ASF subversion and git services commented on AIRFLOW-6238: -- Commit 49fbed898602165c6259cca6e1c0c893e3c75134 in airflow's branch refs/heads/v1-10-test from Robin Edwards [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=49fbed8 ] [AIRFLOW-6238] Filter dags returned by dag_stats Add dag_ids parameter to the dag_stats end point and only request the dags on the current page. This is intended to speed up the response times for systems running a large number of DAGS. (cherry picked from commit 1b6128221536195f912dd813d427cceb3cdb6f7b) (cherry picked from commit 303f9babd2e6781c13cc4c98756b78753aa25480) > Dags stats endpoint is slow and returns a large payload > --- > > Key: AIRFLOW-6238 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6238 > Project: Apache Airflow > Issue Type: Improvement > Components: api, ui >Affects Versions: 2.0.0, 1.10.6 >Reporter: Robin Edwards >Priority: Minor > Fix For: 1.10.8 > > > The dag_stats endpoint returns all dags by default. This can result in an > extremely large payload ~ 3mb and slow response time when you have a lot of > dags (In our case 1500+). > The accompanying pull request adds a dag_ids get parameter to the dag_stats > end point which is populated by the dags present on the page. > Please see previous issue for task_stats AIRFLOW-6095 which has already been > merged with a similar solution -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6359) spark_submit_hook.py polling too frequent
[
https://issues.apache.org/jira/browse/AIRFLOW-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022334#comment-17022334
]
ASF subversion and git services commented on AIRFLOW-6359:
--
Commit 5f819dcc2ccc3f2375954c6202ce587a442963d3 in airflow's branch
refs/heads/v1-10-test from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=5f819dc ]
[AIRFLOW-6359] Make Spark status_poll_interval explicit (#6978)
(cherry picked from commit 6cc1123e6f2ed09b0fa42fa85fcac97a06ba0924)
> spark_submit_hook.py polling too frequent
> -
>
> Key: AIRFLOW-6359
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6359
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.6
>Reporter: t oo
>Assignee: t oo
>Priority: Major
> Fix For: 1.10.8
>
>
> spark polling every second is too frequent, use config to change to desired
> interval, ie 15 seconds
> contrib/hooks/spark_submit_hook.py Poll spark server at a custom interval
> instead of every second
>
> BEFORE
> Sleep for 1 second as we do not want to spam the cluster
> time.sleep(1)
>
> AFTER
> import airflow
> from airflow import configuration as conf
> Sleep for n second as we do not want to spam the cluster
> _poll_interval = conf.getint('sparksubmit', 'poll_interval')
> time.sleep(_poll_interval)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6359) spark_submit_hook.py polling too frequent
[
https://issues.apache.org/jira/browse/AIRFLOW-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022333#comment-17022333
]
ASF subversion and git services commented on AIRFLOW-6359:
--
Commit 8473c89ca55aa572bcde81bb39f2264a48201e15 in airflow's branch
refs/heads/v1-10-test from tooptoop4
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=8473c89 ]
[AIRFLOW-6359] spark_submit_hook.py status polling interval config (#6909)
* AIRFLOW-6359
* pep checks
* make travis CI refire
* new section = bad
* move poll_interval into the operator instead of .cfg
* review comments
(cherry picked from commit 83ba8e638fb539afd6dac495a8c59012d0aaa9f4)
> spark_submit_hook.py polling too frequent
> -
>
> Key: AIRFLOW-6359
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6359
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.6
>Reporter: t oo
>Assignee: t oo
>Priority: Major
> Fix For: 1.10.8
>
>
> spark polling every second is too frequent, use config to change to desired
> interval, ie 15 seconds
> contrib/hooks/spark_submit_hook.py Poll spark server at a custom interval
> instead of every second
>
> BEFORE
> Sleep for 1 second as we do not want to spam the cluster
> time.sleep(1)
>
> AFTER
> import airflow
> from airflow import configuration as conf
> Sleep for n second as we do not want to spam the cluster
> _poll_interval = conf.getint('sparksubmit', 'poll_interval')
> time.sleep(_poll_interval)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6576) Deleting a task with SLA crashes the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-6576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022322#comment-17022322 ] ASF subversion and git services commented on AIRFLOW-6576: -- Commit bb92ff77416abef30cd90d577472972e8385f350 in airflow's branch refs/heads/v1-10-test from QP Hou [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=bb92ff7 ] [AIRFLOW-6576] Fix scheduler crash caused by deleted task with sla misses (#7187) When a task with SLA is deleted from a DAG after the SLA miss is logged but before the notification was sent, scheduler will crash with an AirflowException > Deleting a task with SLA crashes the scheduler > -- > > Key: AIRFLOW-6576 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6576 > Project: Apache Airflow > Issue Type: New Feature > Components: scheduler >Affects Versions: 2.0.0 >Reporter: QP Hou >Assignee: QP Hou >Priority: Major > Fix For: 2.0.0, 1.10.8 > > > When a task with SLA is deleted from a DAG after the SLA miss is logged but > before the notification was sent, scheduler will crash with an > AirflowException -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6261) flower_basic_auth eligible to _cmd
[ https://issues.apache.org/jira/browse/AIRFLOW-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022323#comment-17022323 ] ASF subversion and git services commented on AIRFLOW-6261: -- Commit 1f4af275ea067aae481cd8bd0b8025cacd4c in airflow's branch refs/heads/v1-10-test from Noël Bardelot [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=1f4af27 ] [AIRFLOW-6261] flower_basic_auth eligible to _cmd (#6825) Make the configuration option flower_basic_auth from celery available as the stdout of a command as it contains sensitive information. (cherry picked from commit 683e928e4b76077f9216792b3379bff888b2fd40) > flower_basic_auth eligible to _cmd > -- > > Key: AIRFLOW-6261 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6261 > Project: Apache Airflow > Issue Type: Improvement > Components: configuration >Affects Versions: 1.10.6 >Reporter: Noël BARDELOT >Assignee: Noël BARDELOT >Priority: Trivial > Fix For: 1.10.8 > > > Make the configuration option flower_basic_auth from celery available as the > stdout of a command as it contains sensitive information. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6327) http_hook: Accept json= parameter for payload
[
https://issues.apache.org/jira/browse/AIRFLOW-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022327#comment-17022327
]
ASF subversion and git services commented on AIRFLOW-6327:
--
Commit c85ea3e38a70c2b524c42caa4d5d88ad1ddd51ca in airflow's branch
refs/heads/v1-10-test from Bjorn Olsen
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c85ea3e ]
[AIRFLOW-6327] http_hook: Accept json= parameter for payload (#6886)
* [AIRFLOW-6327] http_hook: Accept json= parameter for payload
* [AIRFLOW-6327] CODE REVIEW
* [AIRFLOW-6327] CODE REVIEW 2
* [AIRFLOW-6327] CODE REVIEW 3
Co-Authored-By: dstandish
Co-authored-by: dstandish
(cherry picked from commit fd78c65cabae2241a4c1d3a792e00620049cbf3e)
> http_hook: Accept json= parameter for payload
> -
>
> Key: AIRFLOW-6327
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6327
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.6
>Reporter: Bjorn Olsen
>Assignee: Bjorn Olsen
>Priority: Minor
> Fix For: 1.10.8
>
>
> Python requests library allows a user to send json-encoded Python objects by
> making use of the "json=" parameter to a request.
> This handles JSON encoding of the payload and setting the correct content
> type header.
> Example:
> {code:python}
> response = requests.post('https://httpbin.org/post', json={'key':'value'})
> json_response = response.json()
> json_response['data'] '{"key": "value"}'
> json_response['headers']['Content-Type'] 'application/json'
> {code}
> http_hook.run() does not yet have the json= parameter which is inconvenient
> as then we have to pass the "data=" parameter using json.dumps and specify
> the correct headers.
> It would be better if we can just do something like the below and let the
> Requests library ensure the request is valid:
> {code:python}
> obj = {'a':1, 'b': 'abc', 'c': [1, 2, {"d":10}]}
> response = hook.run(
> endpoint,
> json=obj)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results
codecov-io edited a comment on issue #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results URL: https://github.com/apache/airflow/pull/6075#issuecomment-561876309 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=h1) Report > Merging [#6075](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/34ed5debc806bc348f82e57359afa01af9ed12ce?src=pr=desc) will **decrease** coverage by `1.06%`. > The diff coverage is `89.66%`. [](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6075 +/- ## == - Coverage 85.18% 84.11% -1.07% == Files 710 791 +81 Lines 3947940138 +659 == + Hits3363033762 +132 - Misses 5849 6376 +527 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=tree) | Coverage Δ | | |---|---|---| | [...flow/contrib/example\_dags/example\_qubole\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9zZW5zb3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | [airflow/gcp/example\_dags/example\_datastore.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZGF0YXN0b3JlLnB5) | `100% <ø> (ø)` | :arrow_up: | | [...ow/gcp/operators/cloud\_storage\_transfer\_service.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2Nsb3VkX3N0b3JhZ2VfdHJhbnNmZXJfc2VydmljZS5weQ==) | `95.63% <ø> (ø)` | :arrow_up: | | [airflow/gcp/example\_dags/example\_dataflow.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZGF0YWZsb3cucHk=) | `100% <ø> (ø)` | :arrow_up: | | [...low/contrib/example\_dags/example\_winrm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3dpbnJtX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/models/pool.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvcG9vbC5weQ==) | `97.36% <ø> (ø)` | :arrow_up: | | [airflow/gcp/example\_dags/example\_compute.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfY29tcHV0ZS5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [...ple\_dags/example\_automl\_vision\_object\_detection.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYXV0b21sX3Zpc2lvbl9vYmplY3RfZGV0ZWN0aW9uLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/gcp/example\_dags/example\_bigquery\_dts.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYmlncXVlcnlfZHRzLnB5) | `100% <ø> (ø)` | :arrow_up: | | [...ders/amazon/aws/example\_dags/example\_datasync\_2.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kYXRhc3luY18yLnB5) | `0% <ø> (ø)` | :arrow_up: | | ... and [363 more](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=footer). Last update [34ed5de...90d3cf9](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6327) http_hook: Accept json= parameter for payload
[
https://issues.apache.org/jira/browse/AIRFLOW-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022328#comment-17022328
]
ASF subversion and git services commented on AIRFLOW-6327:
--
Commit c85ea3e38a70c2b524c42caa4d5d88ad1ddd51ca in airflow's branch
refs/heads/v1-10-test from Bjorn Olsen
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c85ea3e ]
[AIRFLOW-6327] http_hook: Accept json= parameter for payload (#6886)
* [AIRFLOW-6327] http_hook: Accept json= parameter for payload
* [AIRFLOW-6327] CODE REVIEW
* [AIRFLOW-6327] CODE REVIEW 2
* [AIRFLOW-6327] CODE REVIEW 3
Co-Authored-By: dstandish
Co-authored-by: dstandish
(cherry picked from commit fd78c65cabae2241a4c1d3a792e00620049cbf3e)
> http_hook: Accept json= parameter for payload
> -
>
> Key: AIRFLOW-6327
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6327
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.6
>Reporter: Bjorn Olsen
>Assignee: Bjorn Olsen
>Priority: Minor
> Fix For: 1.10.8
>
>
> Python requests library allows a user to send json-encoded Python objects by
> making use of the "json=" parameter to a request.
> This handles JSON encoding of the payload and setting the correct content
> type header.
> Example:
> {code:python}
> response = requests.post('https://httpbin.org/post', json={'key':'value'})
> json_response = response.json()
> json_response['data'] '{"key": "value"}'
> json_response['headers']['Content-Type'] 'application/json'
> {code}
> http_hook.run() does not yet have the json= parameter which is inconvenient
> as then we have to pass the "data=" parameter using json.dumps and specify
> the correct headers.
> It would be better if we can just do something like the below and let the
> Requests library ensure the request is valid:
> {code:python}
> obj = {'a':1, 'b': 'abc', 'c': [1, 2, {"d":10}]}
> response = hook.run(
> endpoint,
> json=obj)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6434) Docker Operator No Longer XComs Result in 1.10.7
[ https://issues.apache.org/jira/browse/AIRFLOW-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022335#comment-17022335 ] ASF subversion and git services commented on AIRFLOW-6434: -- Commit 6d193bfb8f95d3e4a1cc5bdd2b3489a86be1deaa in airflow's branch refs/heads/v1-10-test from Kaxil Naik [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6d193bf ] fixup! [AIRFLOW-6434] add return statement back to DockerOperator.execute (#7013) > Docker Operator No Longer XComs Result in 1.10.7 > > > Key: AIRFLOW-6434 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6434 > Project: Apache Airflow > Issue Type: Bug > Components: operators, xcom >Affects Versions: 1.10.7 >Reporter: Brian Phillips >Priority: Trivial > Fix For: 1.10.8 > > > This change > ([https://github.com/apache/airflow/commit/8a6dc6657d2a32ac3979e6478512d518ad5a5212)|https://github.com/apache/airflow/commit/8a6dc6657d2a32ac3979e6478512d518ad5a5212] > introduced a slight (and I believe unintended) change to the Docker Operator > xcom behavior. > Even if xcom_push is True, DockerOperator.execute will not return a value and > thus will not push an xcom value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6327) http_hook: Accept json= parameter for payload
[
https://issues.apache.org/jira/browse/AIRFLOW-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17022326#comment-17022326
]
ASF subversion and git services commented on AIRFLOW-6327:
--
Commit c85ea3e38a70c2b524c42caa4d5d88ad1ddd51ca in airflow's branch
refs/heads/v1-10-test from Bjorn Olsen
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=c85ea3e ]
[AIRFLOW-6327] http_hook: Accept json= parameter for payload (#6886)
* [AIRFLOW-6327] http_hook: Accept json= parameter for payload
* [AIRFLOW-6327] CODE REVIEW
* [AIRFLOW-6327] CODE REVIEW 2
* [AIRFLOW-6327] CODE REVIEW 3
Co-Authored-By: dstandish
Co-authored-by: dstandish
(cherry picked from commit fd78c65cabae2241a4c1d3a792e00620049cbf3e)
> http_hook: Accept json= parameter for payload
> -
>
> Key: AIRFLOW-6327
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6327
> Project: Apache Airflow
> Issue Type: Improvement
> Components: hooks
>Affects Versions: 1.10.6
>Reporter: Bjorn Olsen
>Assignee: Bjorn Olsen
>Priority: Minor
> Fix For: 1.10.8
>
>
> Python requests library allows a user to send json-encoded Python objects by
> making use of the "json=" parameter to a request.
> This handles JSON encoding of the payload and setting the correct content
> type header.
> Example:
> {code:python}
> response = requests.post('https://httpbin.org/post', json={'key':'value'})
> json_response = response.json()
> json_response['data'] '{"key": "value"}'
> json_response['headers']['Content-Type'] 'application/json'
> {code}
> http_hook.run() does not yet have the json= parameter which is inconvenient
> as then we have to pass the "data=" parameter using json.dumps and specify
> the correct headers.
> It would be better if we can just do something like the below and let the
> Requests library ensure the request is valid:
> {code:python}
> obj = {'a':1, 'b': 'abc', 'c': [1, 2, {"d":10}]}
> response = hook.run(
> endpoint,
> json=obj)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results
codecov-io edited a comment on issue #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results URL: https://github.com/apache/airflow/pull/6075#issuecomment-561876309 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=h1) Report > Merging [#6075](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/34ed5debc806bc348f82e57359afa01af9ed12ce?src=pr=desc) will **decrease** coverage by `1.06%`. > The diff coverage is `89.66%`. [](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6075 +/- ## == - Coverage 85.18% 84.11% -1.07% == Files 710 791 +81 Lines 3947940138 +659 == + Hits3363033762 +132 - Misses 5849 6376 +527 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=tree) | Coverage Δ | | |---|---|---| | [...flow/contrib/example\_dags/example\_qubole\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3F1Ym9sZV9zZW5zb3IucHk=) | `100% <ø> (ø)` | :arrow_up: | | [airflow/gcp/example\_dags/example\_datastore.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZGF0YXN0b3JlLnB5) | `100% <ø> (ø)` | :arrow_up: | | [...ow/gcp/operators/cloud\_storage\_transfer\_service.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9nY3Avb3BlcmF0b3JzL2Nsb3VkX3N0b3JhZ2VfdHJhbnNmZXJfc2VydmljZS5weQ==) | `95.63% <ø> (ø)` | :arrow_up: | | [airflow/gcp/example\_dags/example\_dataflow.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfZGF0YWZsb3cucHk=) | `100% <ø> (ø)` | :arrow_up: | | [...low/contrib/example\_dags/example\_winrm\_operator.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2V4YW1wbGVfZGFncy9leGFtcGxlX3dpbnJtX29wZXJhdG9yLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/models/pool.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvcG9vbC5weQ==) | `97.36% <ø> (ø)` | :arrow_up: | | [airflow/gcp/example\_dags/example\_compute.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfY29tcHV0ZS5weQ==) | `100% <ø> (ø)` | :arrow_up: | | [...ple\_dags/example\_automl\_vision\_object\_detection.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYXV0b21sX3Zpc2lvbl9vYmplY3RfZGV0ZWN0aW9uLnB5) | `100% <ø> (ø)` | :arrow_up: | | [airflow/gcp/example\_dags/example\_bigquery\_dts.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9nY3AvZXhhbXBsZV9kYWdzL2V4YW1wbGVfYmlncXVlcnlfZHRzLnB5) | `100% <ø> (ø)` | :arrow_up: | | [...ders/amazon/aws/example\_dags/example\_datasync\_2.py](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYW1hem9uL2F3cy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kYXRhc3luY18yLnB5) | `0% <ø> (ø)` | :arrow_up: | | ... and [363 more](https://codecov.io/gh/apache/airflow/pull/6075/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=footer). Last update [34ed5de...90d3cf9](https://codecov.io/gh/apache/airflow/pull/6075?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
