date:20190101



[ 
https://issues.apache.org/jira/browse/AIRFLOW-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731812#comment-16731812
 ] 

ASF GitHub Bot commented on AIRFLOW-1921:
-

Fokko commented on pull request #2879: [AIRFLOW-1921] Added support for presto 
https and user auth
URL: https://github.com/apache/incubator-airflow/pull/2879
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> PrestoHook - Add support for TLS and Authentication
> ---
>
> Key: AIRFLOW-1921
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1921
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: hooks
>Affects Versions: 1.8.0
>Reporter: Steve Jacobs
>Priority: Major
> Fix For: 2.0.0
>
>
> The current presto hook does not support authentication, which requires a 
> HTTPS connection to the presto coordinator. The PrestoHook should allow using 
> an authenticated connection over HTTPS using presto's ldap authentication



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] Fokko closed pull request #2879: [AIRFLOW-1921] Added support for presto https and user auth

Fokko closed pull request #2879: [AIRFLOW-1921] Added support for presto https 
and user auth
URL: https://github.com/apache/incubator-airflow/pull/2879
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/hooks/presto_hook.py b/airflow/hooks/presto_hook.py
index d6b5293fc0..e2794df071 100644
--- a/airflow/hooks/presto_hook.py
+++ b/airflow/hooks/presto_hook.py
@@ -21,6 +21,7 @@
 
 from pyhive import presto
 from pyhive.exc import DatabaseError
+from requests.auth import HTTPBasicAuth
 
 from airflow.hooks.dbapi_hook import DbApiHook
 
@@ -45,11 +46,17 @@ class PrestoHook(DbApiHook):
 def get_conn(self):
 """Returns a connection object"""
 db = self.get_connection(self.presto_conn_id)
+reqkwargs = None
+if db.password is not None:
+reqkwargs = {'auth': HTTPBasicAuth(db.login, db.password)}
 return presto.connect(
 host=db.host,
 port=db.port,
 username=db.login,
+source=db.extra_dejson.get('source', 'airflow'),
+protocol=db.extra_dejson.get('protocol', 'http'),
 catalog=db.extra_dejson.get('catalog', 'hive'),
+requests_kwargs=reqkwargs,
 schema=db.schema)
 
 @staticmethod


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4419: [AIRFLOW-3612] Remove incubation/incubator mention

codecov-io commented on issue #4419: [AIRFLOW-3612] Remove incubation/incubator 
mention
URL: 
https://github.com/apache/incubator-airflow/pull/4419#issuecomment-450802992
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4419?src=pr=h1)
 Report
   > Merging 
[#4419](https://codecov.io/gh/apache/incubator-airflow/pull/4419?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/7ee926618aa709bd4d74d55c990db7780de23488?src=pr=desc)
 will **decrease** coverage by `6.83%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4419/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4419?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4419  +/-   ##
   ==
   - Coverage   78.18%   71.35%   -6.84% 
   ==
 Files 204  204  
 Lines   1644518631+2186 
   ==
   + Hits1285813294 +436 
   - Misses   3587 5337+1750
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4419?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/example\_dags/tutorial.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvdHV0b3JpYWwucHk=)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/www/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBwLnB5)
 | `98.96% <ø> (ø)` | :arrow_up: |
   | 
[airflow/operators/slack\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvc2xhY2tfb3BlcmF0b3IucHk=)
 | `97.36% <ø> (ø)` | :arrow_up: |
   | 
[airflow/www\_rbac/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9hcHAucHk=)
 | `97.08% <ø> (ø)` | :arrow_up: |
   | 
[airflow/version.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy92ZXJzaW9uLnB5)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `56.03% <0%> (-17.69%)` | :arrow_down: |
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `75.67% <0%> (-16.89%)` | :arrow_down: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `53.58% <0%> (-16.11%)` | :arrow_down: |
   | 
[airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5)
 | `52.58% <0%> (-12.49%)` | :arrow_down: |
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `77.44% <0%> (+0.73%)` | :arrow_up: |
   | ... and [4 
more](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4419?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4419?src=pr=footer).
 Last update 
[7ee9266...17de469](https://codecov.io/gh/apache/incubator-airflow/pull/4419?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4419: [AIRFLOW-3612] Remove incubation/incubator mention

codecov-io edited a comment on issue #4419: [AIRFLOW-3612] Remove 
incubation/incubator mention
URL: 
https://github.com/apache/incubator-airflow/pull/4419#issuecomment-450802992
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4419?src=pr=h1)
 Report
   > Merging 
[#4419](https://codecov.io/gh/apache/incubator-airflow/pull/4419?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/7ee926618aa709bd4d74d55c990db7780de23488?src=pr=desc)
 will **decrease** coverage by `6.83%`.
   > The diff coverage is `100%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4419/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4419?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4419  +/-   ##
   ==
   - Coverage   78.18%   71.35%   -6.84% 
   ==
 Files 204  204  
 Lines   1644518631+2186 
   ==
   + Hits1285813294 +436 
   - Misses   3587 5337+1750
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4419?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/example\_dags/tutorial.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvdHV0b3JpYWwucHk=)
 | `100% <ø> (ø)` | :arrow_up: |
   | 
[airflow/www/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBwLnB5)
 | `98.96% <ø> (ø)` | :arrow_up: |
   | 
[airflow/operators/slack\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvc2xhY2tfb3BlcmF0b3IucHk=)
 | `97.36% <ø> (ø)` | :arrow_up: |
   | 
[airflow/www\_rbac/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9hcHAucHk=)
 | `97.08% <ø> (ø)` | :arrow_up: |
   | 
[airflow/version.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy92ZXJzaW9uLnB5)
 | `100% <100%> (ø)` | :arrow_up: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `56.03% <0%> (-17.69%)` | :arrow_down: |
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `75.67% <0%> (-16.89%)` | :arrow_down: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `53.58% <0%> (-16.11%)` | :arrow_down: |
   | 
[airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5)
 | `52.58% <0%> (-12.49%)` | :arrow_down: |
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `77.44% <0%> (+0.73%)` | :arrow_up: |
   | ... and [4 
more](https://codecov.io/gh/apache/incubator-airflow/pull/4419/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4419?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4419?src=pr=footer).
 Last update 
[7ee9266...17de469](https://codecov.io/gh/apache/incubator-airflow/pull/4419?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3580) Add tests for HiveToMySqlOperator



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731811#comment-16731811
 ] 

ASF GitHub Bot commented on AIRFLOW-3580:
-

feng-tao commented on pull request #4387: [AIRFLOW-3580] Adds tests for 
HiveToMySqlTransfer
URL: https://github.com/apache/incubator-airflow/pull/4387
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Add tests for HiveToMySqlOperator
> -
>
> Key: AIRFLOW-3580
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3580
> Project: Apache Airflow
>  Issue Type: Test
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao commented on issue #4387: [AIRFLOW-3580] Adds tests for HiveToMySqlTransfer

feng-tao commented on issue #4387: [AIRFLOW-3580] Adds tests for 
HiveToMySqlTransfer
URL: 
https://github.com/apache/incubator-airflow/pull/4387#issuecomment-450802593
 
 
   thanks for adding the tests.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Fokko commented on issue #4247: [AIRFLOW-3402] Support global k8s affinity and toleration configs

Fokko commented on issue #4247: [AIRFLOW-3402] Support global k8s affinity and 
toleration configs
URL: 
https://github.com/apache/incubator-airflow/pull/4247#issuecomment-450802347
 
 
   I've restarted the test, let's see.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao closed pull request #4413: [AIRFLOW-XXX] Adds image code comment

feng-tao closed pull request #4413: [AIRFLOW-XXX] Adds image code comment
URL: https://github.com/apache/incubator-airflow/pull/4413
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/scripts/ci/kubernetes/kube/templates/airflow.template.yaml 
b/scripts/ci/kubernetes/kube/templates/airflow.template.yaml
index 82f7b8b17d..992253a8ab 100644
--- a/scripts/ci/kubernetes/kube/templates/airflow.template.yaml
+++ b/scripts/ci/kubernetes/kube/templates/airflow.template.yaml
@@ -15,6 +15,9 @@
 #  specific language governing permissions and limitations  *
 #  under the License.   *
 
+# Note: The airflow image used in this example is obtained by
+# building the image from the local docker subdirectory.
+
 apiVersion: rbac.authorization.k8s.io/v1beta1
 kind: ClusterRoleBinding
 metadata:


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on issue #4416: [AIRFLOW-3608] Allow devs to submit custom airflow images to k8s CI

feng-tao commented on issue #4416: [AIRFLOW-3608]  Allow devs to submit custom 
airflow images to k8s CI
URL: 
https://github.com/apache/incubator-airflow/pull/4416#issuecomment-450802096
 
 
   lgtm


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3608) Allow users to submit custom airflow images when running Airflow Kubernetes CI



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731806#comment-16731806
 ] 

ASF GitHub Bot commented on AIRFLOW-3608:
-

Fokko commented on pull request #4416: [AIRFLOW-3608]  Allow devs to submit 
custom airflow images to k8s CI
URL: https://github.com/apache/incubator-airflow/pull/4416
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Allow users to submit custom airflow images when running Airflow Kubernetes CI
> --
>
> Key: AIRFLOW-3608
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3608
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: kubernetes
>Reporter: Daniel Imberman
>Assignee: Daniel Imberman
>Priority: Minor
> Fix For: 2.0.0
>
>
> To help move away from Minikube, we need to remove the dependency on
>  a local docker registry and move towards a solution that can be used
>  in any kubernetes cluster. Custom image names allow users to use
>  systems like docker, artifactory and gcr



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] Fokko closed pull request #4416: [AIRFLOW-3608] Allow devs to submit custom airflow images to k8s CI

Fokko closed pull request #4416: [AIRFLOW-3608]  Allow devs to submit custom 
airflow images to k8s CI
URL: https://github.com/apache/incubator-airflow/pull/4416
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/scripts/ci/kubernetes/docker/build.sh 
b/scripts/ci/kubernetes/docker/build.sh
index df755f9659..69126dcd4e 100755
--- a/scripts/ci/kubernetes/docker/build.sh
+++ b/scripts/ci/kubernetes/docker/build.sh
@@ -17,8 +17,8 @@
 #  specific language governing permissions and limitations  *
 #  under the License.   *
 
-IMAGE=${1:-airflow}
-TAG=${2:-latest}
+IMAGE=${IMAGE:-airflow}
+TAG=${TAG:-latest}
 DIRNAME=$(cd "$(dirname "$0")"; pwd)
 AIRFLOW_ROOT="$DIRNAME/../../../.."
 
diff --git a/scripts/ci/kubernetes/kube/deploy.sh 
b/scripts/ci/kubernetes/kube/deploy.sh
index abbf4f8d0f..9bc20e0b78 100755
--- a/scripts/ci/kubernetes/kube/deploy.sh
+++ b/scripts/ci/kubernetes/kube/deploy.sh
@@ -19,8 +19,8 @@
 
 set -x
 
-IMAGE=${1:-airflow/ci}
-TAG=${2:-latest}
+AIRFLOW_IMAGE=${IMAGE:-airflow}
+AIRFLOW_TAG=${TAG:-latest}
 DIRNAME=$(cd "$(dirname "$0")"; pwd)
 TEMPLATE_DIRNAME=${DIRNAME}/templates
 BUILD_DIRNAME=${DIRNAME}/build
@@ -117,6 +117,8 @@ else
   ${SED_COMMAND} -e "/{{INIT_GIT_SYNC}}/{r 
$TEMPLATE_DIRNAME/init_git_sync.template.yaml" -e 'd}' \
   ${TEMPLATE_DIRNAME}/airflow.template.yaml > ${BUILD_DIRNAME}/airflow.yaml
 fi
+${SED_COMMAND} -i "s|{{AIRFLOW_IMAGE}}|$AIRFLOW_IMAGE|g" 
${BUILD_DIRNAME}/airflow.yaml
+${SED_COMMAND} -i "s|{{AIRFLOW_TAG}}|$AIRFLOW_TAG|g" 
${BUILD_DIRNAME}/airflow.yaml
 
 ${SED_COMMAND} -i "s|{{CONFIGMAP_GIT_REPO}}|$CONFIGMAP_GIT_REPO|g" 
${BUILD_DIRNAME}/airflow.yaml
 ${SED_COMMAND} -i "s|{{CONFIGMAP_BRANCH}}|$CONFIGMAP_BRANCH|g" 
${BUILD_DIRNAME}/airflow.yaml
@@ -130,6 +132,10 @@ ${SED_COMMAND} -i 
"s|{{CONFIGMAP_BRANCH}}|$CONFIGMAP_BRANCH|g" ${BUILD_DIRNAME}/
 ${SED_COMMAND} -i 
"s|{{CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT}}|$CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT|g"
 ${BUILD_DIRNAME}/configmaps.yaml
 ${SED_COMMAND} -i 
"s|{{CONFIGMAP_DAGS_VOLUME_CLAIM}}|$CONFIGMAP_DAGS_VOLUME_CLAIM|g" 
${BUILD_DIRNAME}/configmaps.yaml
 
+
+cat ${BUILD_DIRNAME}/airflow.yaml
+cat ${BUILD_DIRNAME}/configmaps.yaml
+
 # Fix file permissions
 if [[ "${TRAVIS}" == true ]]; then
   sudo chown -R travis.travis $HOME/.kube $HOME/.minikube
diff --git a/scripts/ci/kubernetes/kube/templates/airflow.template.yaml 
b/scripts/ci/kubernetes/kube/templates/airflow.template.yaml
index 82f7b8b17d..7c32b43d4e 100644
--- a/scripts/ci/kubernetes/kube/templates/airflow.template.yaml
+++ b/scripts/ci/kubernetes/kube/templates/airflow.template.yaml
@@ -43,7 +43,7 @@ spec:
 spec:
   initContainers:
   - name: "init"
-image: airflow
+image: {{AIRFLOW_IMAGE}}:{{AIRFLOW_TAG}}
 imagePullPolicy: IfNotPresent
 volumeMounts:
 - name: airflow-configmap
@@ -67,7 +67,7 @@ spec:
 {{INIT_GIT_SYNC}}
   containers:
   - name: webserver
-image: airflow
+image: {{AIRFLOW_IMAGE}}:{{AIRFLOW_TAG}}
 imagePullPolicy: IfNotPresent
 ports:
 - name: webserver
@@ -92,7 +92,7 @@ spec:
 - name: airflow-logs
   mountPath: /root/airflow/logs
   - name: scheduler
-image: airflow
+image: {{AIRFLOW_IMAGE}}:{{AIRFLOW_TAG}}
 imagePullPolicy: IfNotPresent
 args: ["scheduler"]
 env:
diff --git a/scripts/ci/kubernetes/setup_kubernetes.sh 
b/scripts/ci/kubernetes/setup_kubernetes.sh
index e0ef216f07..d56cfad12a 100755
--- a/scripts/ci/kubernetes/setup_kubernetes.sh
+++ b/scripts/ci/kubernetes/setup_kubernetes.sh
@@ -31,4 +31,7 @@ fi
 $DIRNAME/minikube/start_minikube.sh
 $DIRNAME/docker/build.sh
 
+
+
+
 echo "Airflow environment on kubernetes is good to go!"


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Closed] (AIRFLOW-3608) Allow users to submit custom airflow images when running Airflow Kubernetes CI

2019-01-01 Thread Fokko Driesprong (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Fokko Driesprong closed AIRFLOW-3608.
-
   Resolution: Fixed
Fix Version/s: 2.0.0

> Allow users to submit custom airflow images when running Airflow Kubernetes CI
> --
>
> Key: AIRFLOW-3608
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3608
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: kubernetes
>Reporter: Daniel Imberman
>Assignee: Daniel Imberman
>Priority: Minor
> Fix For: 2.0.0
>
>
> To help move away from Minikube, we need to remove the dependency on
>  a local docker registry and move towards a solution that can be used
>  in any kubernetes cluster. Custom image names allow users to use
>  systems like docker, artifactory and gcr



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao commented on issue #4419: [AIRFLOW-3612] Remove incubation/incubator mention

feng-tao commented on issue #4419: [AIRFLOW-3612] Remove incubation/incubator 
mention
URL: 
https://github.com/apache/incubator-airflow/pull/4419#issuecomment-450800604
 
 
   @Rotemlofer, I believe it will change once the TLP announcement is official.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on issue #4395: [AIRFLOW-3587] Remove unnecessary condition checks in dag_stats & task_stats

XD-DENG commented on issue #4395: [AIRFLOW-3587] Remove unnecessary condition 
checks in dag_stats & task_stats
URL: 
https://github.com/apache/incubator-airflow/pull/4395#issuecomment-450800234
 
 
   Thanks @feng-tao :-)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on issue #4403: [AIRFLOW-XXX] - Fix missing type in docstring of MySqlToGoogleCloudStorageOperator

feng-tao commented on issue #4403: [AIRFLOW-XXX] - Fix missing type in 
docstring of MySqlToGoogleCloudStorageOperator
URL: 
https://github.com/apache/incubator-airflow/pull/4403#issuecomment-450798951
 
 
   lgtm


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on a change in pull request #4420: [AIRFLOW-3613] Updated ReadMe to add Adobe as an airflow user

feng-tao commented on a change in pull request #4420: [AIRFLOW-3613] Updated 
ReadMe to add Adobe as an airflow user
URL: https://github.com/apache/incubator-airflow/pull/4420#discussion_r244671596
 
 

 ##
 File path: README.md
 ##
 @@ -109,6 +109,7 @@ Currently **officially** using Airflow:
 1. [90 Seconds](https://90seconds.tv/) 
[[@aaronmak](https://github.com/aaronmak)]
 1. [99](https://99taxis.com) [[@fbenevides](https://github.com/fbenevides), 
[@gustavoamigo](https://github.com/gustavoamigo) & 
[@mmmaia](https://github.com/mmmaia)]
 1. [AdBOOST](https://www.adboost.sk) [[AdBOOST](https://github.com/AdBOOST)]
+1. [Adobe](https://www.adobe.com/)
 
 Review comment:
   Please follow the format like other entry.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3587) To remove unnecessary condition checks in dag_stats & task_stats



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731802#comment-16731802
 ] 

ASF GitHub Bot commented on AIRFLOW-3587:
-

feng-tao commented on pull request #4395: [AIRFLOW-3587] Remove unnecessary 
condition checks in dag_stats & task_stats
URL: https://github.com/apache/incubator-airflow/pull/4395
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> To remove unnecessary condition checks in dag_stats & task_stats
> 
>
> Key: AIRFLOW-3587
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3587
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: webapp
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> This is an issue in current master branch.
>  
> There are two unnecessary condition checks in 
> [airflow/www_rbac/views.py|https://github.com/XD-DENG/incubator-airflow/compare/65843a63aaa4...0c50b25a978c#diff-77df5adb32d964f37748c4557ffb3c4c]
>  [dag_stats & 
> task_stats|https://github.com/XD-DENG/incubator-airflow/commit/0c50b25a978c418bb61316a94d28551417d57511].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao closed pull request #4395: [AIRFLOW-3587] Remove unnecessary condition checks in dag_stats & task_stats

feng-tao closed pull request #4395: [AIRFLOW-3587] Remove unnecessary condition 
checks in dag_stats & task_stats
URL: https://github.com/apache/incubator-airflow/pull/4395
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/www_rbac/views.py b/airflow/www_rbac/views.py
index 42328279a3..1ca68368a9 100644
--- a/airflow/www_rbac/views.py
+++ b/airflow/www_rbac/views.py
@@ -297,16 +297,15 @@ def dag_stats(self, session=None):
 filter_dag_ids = [dag_id for dag_id, in 
session.query(models.DagModel.dag_id)]
 
 for dag_id in filter_dag_ids:
-if 'all_dags' in filter_dag_ids or dag_id in filter_dag_ids:
-payload[dag_id] = []
-for state in State.dag_states:
-count = data.get(dag_id, {}).get(state, 0)
-payload[dag_id].append({
-'state': state,
-'count': count,
-'dag_id': dag_id,
-'color': State.color(state)
-})
+payload[dag_id] = []
+for state in State.dag_states:
+count = data.get(dag_id, {}).get(state, 0)
+payload[dag_id].append({
+'state': state,
+'count': count,
+'dag_id': dag_id,
+'color': State.color(state)
+})
 return wwwutils.json_response(payload)
 
 @expose('/task_stats')
@@ -373,16 +372,15 @@ def task_stats(self, session=None):
 if 'all_dags' in filter_dag_ids:
 filter_dag_ids = [dag_id for dag_id, in 
session.query(models.DagModel.dag_id)]
 for dag_id in filter_dag_ids:
-if 'all_dags' in filter_dag_ids or dag_id in filter_dag_ids:
-payload[dag_id] = []
-for state in State.task_states:
-count = data.get(dag_id, {}).get(state, 0)
-payload[dag_id].append({
-'state': state,
-'count': count,
-'dag_id': dag_id,
-'color': State.color(state)
-})
+payload[dag_id] = []
+for state in State.task_states:
+count = data.get(dag_id, {}).get(state, 0)
+payload[dag_id].append({
+'state': state,
+'count': count,
+'dag_id': dag_id,
+'color': State.color(state)
+})
 return wwwutils.json_response(payload)
 
 @expose('/code')


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on issue #4395: [AIRFLOW-3587] Remove unnecessary condition checks in dag_stats & task_stats

feng-tao commented on issue #4395: [AIRFLOW-3587] Remove unnecessary condition 
checks in dag_stats & task_stats
URL: 
https://github.com/apache/incubator-airflow/pull/4395#issuecomment-450798481
 
 
   thanks @XD-DENG for the clarification, I misunderstood that it removed the 
change for checking all-dags . The change lgtm in this case.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ramandumcs opened a new pull request #4420: [AIRFLOW-3613] Updated ReadMe to add Adobe as an airflow user

ramandumcs opened a new pull request #4420: [AIRFLOW-3613] Updated ReadMe to 
add Adobe as an airflow user
URL: https://github.com/apache/incubator-airflow/pull/4420
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] Rotemlofer commented on issue #4419: [AIRFLOW-3612] Remove incubation/incubator mention

Rotemlofer commented on issue #4419: [AIRFLOW-3612] Remove incubation/incubator 
mention
URL: 
https://github.com/apache/incubator-airflow/pull/4419#issuecomment-450796961
 
 
   Does the git-hub repository going to be changed as well 
apache/incubator-airflow   ? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3613) Update Readme to add adobe as airflow user



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731800#comment-16731800
 ] 

ASF GitHub Bot commented on AIRFLOW-3613:
-

ramandumcs commented on pull request #4420: [AIRFLOW-3613] Updated ReadMe to 
add Adobe as an airflow user
URL: https://github.com/apache/incubator-airflow/pull/4420
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Update Readme to add adobe as airflow user
> --
>
> Key: AIRFLOW-3613
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3613
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 1.10.0, 1.10.1
>Reporter: raman
>Assignee: raman
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (AIRFLOW-3613) Update Readme to add adobe as airflow user

2019-01-01 Thread raman (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

raman reassigned AIRFLOW-3613:
--

Assignee: raman

> Update Readme to add adobe as airflow user
> --
>
> Key: AIRFLOW-3613
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3613
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 1.10.0, 1.10.1
>Reporter: raman
>Assignee: raman
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3606) Fix Flake8 test & Fix the Flake8 errors introduced since Flake8 test was broken



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731794#comment-16731794
 ] 

ASF GitHub Bot commented on AIRFLOW-3606:
-

feng-tao commented on pull request #4415: [AIRFLOW-3606] Fix Flake8 test & fix 
the Flake8 errors introduced since Flake8 test was broken
URL: https://github.com/apache/incubator-airflow/pull/4415
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


>  Fix Flake8 test & Fix the Flake8 errors introduced since Flake8 test was 
> broken
> 
>
> Key: AIRFLOW-3606
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3606
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Critical
>
> The flake8 test in the Travis CI was broken since 
> [https://github.com/apache/incubator-airflow/pull/4361] 
> ([https://github.com/apache/incubator-airflow/commit/7a6acbf5b343e4a6895d1cc8af75ecc02b4fd0e8]
>  )
>  
> And some Flake8 errors (code style/quality issues. found in 10 files) were 
> introduce since flake8 test was broken.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao closed pull request #4415: [AIRFLOW-3606] Fix Flake8 test & fix the Flake8 errors introduced since Flake8 test was broken

feng-tao closed pull request #4415: [AIRFLOW-3606] Fix Flake8 test & fix the 
Flake8 errors introduced since Flake8 test was broken
URL: https://github.com/apache/incubator-airflow/pull/4415
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/.flake8 b/.flake8
index 9c4bbcd661..2723df1f10 100644
--- a/.flake8
+++ b/.flake8
@@ -1,4 +1,3 @@
 [flake8]
 max-line-length = 110
 ignore = E731
-exclude = .git, env, venv, .venv, VENV, .tox, .eggs, .*
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index f4bee90c52..877bf34e20 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -69,7 +69,6 @@
 from airflow.www_rbac.app import create_app as create_app_rbac
 from airflow.www_rbac.app import cached_appbuilder
 
-from sqlalchemy import func
 from sqlalchemy.orm import exc
 
 api.load_auth()
diff --git a/airflow/migrations/versions/a56c9515abdc_remove_dag_stat_table.py 
b/airflow/migrations/versions/a56c9515abdc_remove_dag_stat_table.py
index 4cba5a0fbb..89dba33c3e 100644
--- a/airflow/migrations/versions/a56c9515abdc_remove_dag_stat_table.py
+++ b/airflow/migrations/versions/a56c9515abdc_remove_dag_stat_table.py
@@ -6,9 +6,9 @@
 # to you under the Apache License, Version 2.0 (the
 # "License"); you may not use this file except in compliance
 # with the License.  You may obtain a copy of the License at
-# 
+#
 #   http://www.apache.org/licenses/LICENSE-2.0
-# 
+#
 # Unless required by applicable law or agreed to in writing,
 # software distributed under the License is distributed on an
 # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -24,15 +24,15 @@
 
 """
 
+from alembic import op
+import sqlalchemy as sa
+
 # revision identifiers, used by Alembic.
 revision = 'a56c9515abdc'
 down_revision = 'c8ffec048a3b'
 branch_labels = None
 depends_on = None
 
-from alembic import op
-import sqlalchemy as sa
-
 
 def upgrade():
 op.drop_table("dag_stats")
diff --git a/airflow/migrations/versions/c8ffec048a3b_add_fields_to_dag.py 
b/airflow/migrations/versions/c8ffec048a3b_add_fields_to_dag.py
index a74e0b50a4..70282715ed 100644
--- a/airflow/migrations/versions/c8ffec048a3b_add_fields_to_dag.py
+++ b/airflow/migrations/versions/c8ffec048a3b_add_fields_to_dag.py
@@ -6,9 +6,9 @@
 # to you under the Apache License, Version 2.0 (the
 # "License"); you may not use this file except in compliance
 # with the License.  You may obtain a copy of the License at
-# 
+#
 #   http://www.apache.org/licenses/LICENSE-2.0
-# 
+#
 # Unless required by applicable law or agreed to in writing,
 # software distributed under the License is distributed on an
 # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -24,15 +24,15 @@
 
 """
 
+from alembic import op
+import sqlalchemy as sa
+
 # revision identifiers, used by Alembic.
 revision = 'c8ffec048a3b'
 down_revision = '41f5f12752f8'
 branch_labels = None
 depends_on = None
 
-from alembic import op
-import sqlalchemy as sa
-
 
 def upgrade():
 op.add_column('dag', sa.Column('description', sa.Text(), nullable=True))
diff --git a/airflow/models/__init__.py b/airflow/models/__init__.py
index d1236fa40b..b2561e2fdb 100755
--- a/airflow/models/__init__.py
+++ b/airflow/models/__init__.py
@@ -43,7 +43,6 @@
 import getpass
 import imp
 import importlib
-import itertools
 import zipfile
 import jinja2
 import json
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 7746fb6de5..96644cc29e 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -554,7 +554,9 @@ def dag_stats(self, session=None):
 dm = models.DagModel
 dag_ids = session.query(dm.dag_id)
 
-dag_state_stats = session.query(dr.dag_id, dr.state, 
sqla.func.count(dr.state)).group_by(dr.dag_id, dr.state)
+dag_state_stats = (
+session.query(dr.dag_id, dr.state, 
sqla.func.count(dr.state)).group_by(dr.dag_id, dr.state)
+)
 
 data = {}
 for (dag_id, ) in dag_ids:
diff --git a/airflow/www_rbac/views.py b/airflow/www_rbac/views.py
index 17efcdfe7c..1d423f65f6 100644
--- a/airflow/www_rbac/views.py
+++ b/airflow/www_rbac/views.py
@@ -281,7 +281,8 @@ def dag_stats(self, session=None):
 
 filter_dag_ids = appbuilder.sm.get_accessible_dag_ids()
 
-dag_state_stats = session.query(dr.dag_id, dr.state, 
sqla.func.count(dr.state)).group_by(dr.dag_id, dr.state)
+dag_state_stats = session.query(dr.dag_id, dr.state, 
sqla.func.count(dr.state))\
+.group_by(dr.dag_id, dr.state)
 
 payload = {}
 if filter_dag_ids:
diff --git a/tests/contrib/executors/test_kubernetes_executor.py 
b/tests/contrib/executors/test_kubernetes_executor.py
index 4836693002..4acb97086f 100644
--- a/tests/contrib/executors/test_kubernetes_executor.py
+++

[GitHub] feng-tao commented on issue #4415: [AIRFLOW-3606] Fix Flake8 test & fix the Flake8 errors introduced since Flake8 test was broken

feng-tao commented on issue #4415: [AIRFLOW-3606] Fix Flake8 test & fix the 
Flake8 errors introduced since Flake8 test was broken
URL: 
https://github.com/apache/incubator-airflow/pull/4415#issuecomment-450795767
 
 
   LGTM. thanks @XD-DENG 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on a change in pull request #4415: [AIRFLOW-3606] Fix Flake8 test & fix the Flake8 errors introduced since Flake8 test was broken

XD-DENG commented on a change in pull request #4415: [AIRFLOW-3606] Fix Flake8 
test & fix the Flake8 errors introduced since Flake8 test was broken
URL: https://github.com/apache/incubator-airflow/pull/4415#discussion_r244669014
 
 

 ##
 File path: airflow/bin/cli.py
 ##
 @@ -69,7 +69,6 @@
 from airflow.www_rbac.app import create_app as create_app_rbac
 from airflow.www_rbac.app import cached_appbuilder
 
-from sqlalchemy import func
 
 Review comment:
   Exactly. The func usage was removed in 
https://github.com/apache/incubator-airflow/commit/53e707428826d603f43aebb73e18fe077881c4b3#diff-1c2404a3a60f829127232842250ff406
 which was merged three days ago.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on a change in pull request #4415: [AIRFLOW-3606] Fix Flake8 test & fix the Flake8 errors introduced since Flake8 test was broken

feng-tao commented on a change in pull request #4415: [AIRFLOW-3606] Fix Flake8 
test & fix the Flake8 errors introduced since Flake8 test was broken
URL: https://github.com/apache/incubator-airflow/pull/4415#discussion_r244668921
 
 

 ##
 File path: airflow/bin/cli.py
 ##
 @@ -69,7 +69,6 @@
 from airflow.www_rbac.app import create_app as create_app_rbac
 from airflow.www_rbac.app import cached_appbuilder
 
-from sqlalchemy import func
 
 Review comment:
   is it because we remove the func usage in some commits recently?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on a change in pull request #4415: [AIRFLOW-3606] Fix Flake8 test & fix the Flake8 errors introduced since Flake8 test was broken

feng-tao commented on a change in pull request #4415: [AIRFLOW-3606] Fix Flake8 
test & fix the Flake8 errors introduced since Flake8 test was broken
URL: https://github.com/apache/incubator-airflow/pull/4415#discussion_r244668737
 
 

 ##
 File path: airflow/bin/cli.py
 ##
 @@ -69,7 +69,6 @@
 from airflow.www_rbac.app import create_app as create_app_rbac
 from airflow.www_rbac.app import cached_appbuilder
 
-from sqlalchemy import func
 
 Review comment:
   are we sure this line is not using?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] XD-DENG commented on issue #4395: [AIRFLOW-3587] Remove unnecessary condition checks in dag_stats & task_stats

XD-DENG commented on issue #4395: [AIRFLOW-3587] Remove unnecessary condition 
checks in dag_stats & task_stats
URL: 
https://github.com/apache/incubator-airflow/pull/4395#issuecomment-450794525
 
 
   Thanks @feng-tao .
   
   The logic you mentioned is in 
https://github.com/apache/incubator-airflow/pull/4395/files#diff-77df5adb32d964f37748c4557ffb3c4cR366
 , which is not touched.
   
   The two lines I tries to remove in this PR were introduced by @ffinfo 6 days 
ago.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-3613) Update Readme to add adobe as airflow user

2019-01-01 Thread raman (JIRA)

raman created AIRFLOW-3613:
--

 Summary: Update Readme to add adobe as airflow user
 Key: AIRFLOW-3613
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3613
 Project: Apache Airflow
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.10.1, 1.10.0
Reporter: raman






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao edited a comment on issue #4395: [AIRFLOW-3587] Remove unnecessary condition checks in dag_stats & task_stats

feng-tao edited a comment on issue #4395: [AIRFLOW-3587] Remove unnecessary 
condition checks in dag_stats & task_stats
URL: 
https://github.com/apache/incubator-airflow/pull/4395#issuecomment-450793077
 
 
   to give more context, if a user with a role of Admin, he will have all-dags 
privilege which could view/edit all the dags, otherwise, it will filter the dag 
only based on the role. So in this case your change is incorrect if we commit 
this change. A role(e.g which could only view example_python_operator) could 
view the task_stat or dag_stat for other dags as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao edited a comment on issue #4395: [AIRFLOW-3587] Remove unnecessary condition checks in dag_stats & task_stats

feng-tao edited a comment on issue #4395: [AIRFLOW-3587] Remove unnecessary 
condition checks in dag_stats & task_stats
URL: 
https://github.com/apache/incubator-airflow/pull/4395#issuecomment-450793077
 
 
   to give more context, if a user with a role of Admin, he will have all-dags 
privilege which could view/edit all the dags, otherwise, it will filter the dag 
only based on the role. So in this case your change is incorrect. A role(e.g 
which could only view example_python_operator) could view the task_stat or 
dag_stat for other dags as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on issue #4395: [AIRFLOW-3587] Remove unnecessary condition checks in dag_stats & task_stats

feng-tao commented on issue #4395: [AIRFLOW-3587] Remove unnecessary condition 
checks in dag_stats & task_stats
URL: 
https://github.com/apache/incubator-airflow/pull/4395#issuecomment-450793077
 
 
   to give more context, if a user with a role of Admin, he will have all-dags 
privilege which could view/edit all the dags, otherwise, it will filter the dag 
only based on the role.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dimberman commented on issue #4416: [AIRFLOW-3608] Allow devs to submit custom airflow images to k8s CI

dimberman commented on issue #4416: [AIRFLOW-3608]  Allow devs to submit custom 
airflow images to k8s CI
URL: 
https://github.com/apache/incubator-airflow/pull/4416#issuecomment-450791746
 
 
   Also @feng-tao if you're around :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dimberman commented on issue #4417: [AIRFLOW-3609] Fix bug in persistent volumes readWriteMany

dimberman commented on issue #4417: [AIRFLOW-3609] Fix bug in persistent 
volumes readWriteMany
URL: 
https://github.com/apache/incubator-airflow/pull/4417#issuecomment-450791593
 
 
   Thanks @feng-tao !


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3576) Remove unnecessray arg 'root' in url_for("airflow.delete", dag_id=dag.dag_id, root=root)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731777#comment-16731777
 ] 

ASF GitHub Bot commented on AIRFLOW-3576:
-

feng-tao commented on pull request #4380: [AIRFLOW-3576] Remove unnecessray arg 
'root' for /delete in dag.html
URL: https://github.com/apache/incubator-airflow/pull/4380
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Remove unnecessray arg 'root' in url_for("airflow.delete", dag_id=dag.dag_id, 
> root=root)
> 
>
> Key: AIRFLOW-3576
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3576
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Major
>
> In airflow/www/templates/airflow/dag.html or 
> airflow/www_rbac/templates/airflow/dag.html, in line * necessary, given it's not used anywhere in the '/delete' method in either 
> *airflow/www/views.py* or *airflow/www_rbac/views.py*.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao closed pull request #4380: [AIRFLOW-3576] Remove unnecessray arg 'root' for /delete in dag.html

feng-tao closed pull request #4380: [AIRFLOW-3576] Remove unnecessray arg 
'root' for /delete in dag.html
URL: https://github.com/apache/incubator-airflow/pull/4380
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/airflow/www/templates/airflow/dag.html 
b/airflow/www/templates/airflow/dag.html
index 3d171c8893..664a6f3120 100644
--- a/airflow/www/templates/airflow/dag.html
+++ b/airflow/www/templates/airflow/dag.html
@@ -101,7 +101,7 @@ 
 
   
   
-
 
 Delete
diff --git a/airflow/www_rbac/templates/airflow/dag.html 
b/airflow/www_rbac/templates/airflow/dag.html
index ed545c09cf..69b16a7666 100644
--- a/airflow/www_rbac/templates/airflow/dag.html
+++ b/airflow/www_rbac/templates/airflow/dag.html
@@ -101,7 +101,7 @@ 
 
   
   
-
 
 Delete


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on issue #4380: [AIRFLOW-3576] Remove unnecessray arg 'root' for /delete in dag.html

feng-tao commented on issue #4380: [AIRFLOW-3576] Remove unnecessray arg 'root' 
for /delete in dag.html
URL: 
https://github.com/apache/incubator-airflow/pull/4380#issuecomment-450791218
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (AIRFLOW-3609) PersistentVolumes in k8s CI have incorrect permissions



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731776#comment-16731776
 ] 

ASF GitHub Bot commented on AIRFLOW-3609:
-

feng-tao commented on pull request #4417: [AIRFLOW-3609] Fix bug in persistent 
volumes readWriteMany
URL: https://github.com/apache/incubator-airflow/pull/4417
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> PersistentVolumes in k8s CI have incorrect permissions
> --
>
> Key: AIRFLOW-3609
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3609
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: kubernetes
>Reporter: Daniel Imberman
>Assignee: Daniel Imberman
>Priority: Minor
>
> When running integration tests on a k8s cluster vs. Minikube
>  I discovered that we were actually using an invalid permission
>  structure for our persistent volume. This commit fixes that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao closed pull request #4417: [AIRFLOW-3609] Fix bug in persistent volumes readWriteMany

feng-tao closed pull request #4417: [AIRFLOW-3609] Fix bug in persistent 
volumes readWriteMany
URL: https://github.com/apache/incubator-airflow/pull/4417
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/scripts/ci/kubernetes/kube/volumes.yaml 
b/scripts/ci/kubernetes/kube/volumes.yaml
index b5488e7c7a..04124afdc6 100644
--- a/scripts/ci/kubernetes/kube/volumes.yaml
+++ b/scripts/ci/kubernetes/kube/volumes.yaml
@@ -23,7 +23,7 @@ metadata:
   name: airflow-dags
 spec:
   accessModes:
-- ReadWriteOnce
+- ReadOnlyMany
   capacity:
 storage: 2Gi
   hostPath:
@@ -35,7 +35,7 @@ metadata:
   name: airflow-dags
 spec:
   accessModes:
-- ReadWriteMany
+- ReadOnlyMany
   resources:
 requests:
   storage: 2Gi
@@ -46,7 +46,7 @@ metadata:
   name: airflow-logs
 spec:
   accessModes:
-- ReadWriteMany
+- ReadOnlyMany
   capacity:
 storage: 2Gi
   hostPath:
@@ -58,7 +58,7 @@ metadata:
   name: airflow-logs
 spec:
   accessModes:
-- ReadWriteMany
+- ReadOnlyMany
   resources:
 requests:
   storage: 2Gi
@@ -81,7 +81,7 @@ metadata:
   name: test-volume
 spec:
   accessModes:
-- ReadWriteMany
+- ReadWriteOnce
   resources:
 requests:
   storage: 2Gi


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] feng-tao commented on issue #4417: [AIRFLOW-3609] Fix bug in persistent volumes readWriteMany

feng-tao commented on issue #4417: [AIRFLOW-3609] Fix bug in persistent volumes 
readWriteMany
URL: 
https://github.com/apache/incubator-airflow/pull/4417#issuecomment-450791165
 
 
   lgtm


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-3612) Remove incubation/incubator mention

2019-01-01 Thread Tao Feng (JIRA)

Tao Feng created AIRFLOW-3612:
-

 Summary: Remove incubation/incubator mention
 Key: AIRFLOW-3612
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3612
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Tao Feng


https://github.com/apache/incubator-airflow/pull/4419



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] feng-tao opened a new pull request #4419: [AIRFLOW-XXX] Remove incubation/incubator mention

feng-tao opened a new pull request #4419: [AIRFLOW-XXX] Remove 
incubation/incubator mention
URL: https://github.com/apache/incubator-airflow/pull/4419
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-XXX
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [x] Here are some details about my PR, including screenshots of any UI 
changes:
   Airflow moved to TLP. Remove all the "incubation/incubator" mentioning.
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4418: [AIRFLOW-3610] Add region param for EMR jobflow creation

codecov-io commented on issue #4418: [AIRFLOW-3610] Add region param for EMR 
jobflow creation
URL: 
https://github.com/apache/incubator-airflow/pull/4418#issuecomment-450786610
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4418?src=pr=h1)
 Report
   > Merging 
[#4418](https://codecov.io/gh/apache/incubator-airflow/pull/4418?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8938d2c727b16762529ef9f1e257ba5fa39f4bea?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4418/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4418?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4418  +/-   ##
   ==
   + Coverage   78.38%   78.38%   +<.01% 
   ==
 Files 204  204  
 Lines   1644516445  
   ==
   + Hits1289012891   +1 
   + Misses   3555 3554   -1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4418?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/incubator-airflow/pull/4418/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `59.85% <0%> (+0.17%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4418?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4418?src=pr=footer).
 Last update 
[8938d2c...c451ced](https://codecov.io/gh/apache/incubator-airflow/pull/4418?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-3611) Simplified development workflow

2019-01-01 Thread Gerardo Curiel (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gerardo Curiel updated AIRFLOW-3611:

Description: This issue is meant to track work done to implement 
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplified+development+workflow
  (was: This issue is meant to track work done to implement 
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplify+development+workflow)

> Simplified development workflow
> ---
>
> Key: AIRFLOW-3611
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3611
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Gerardo Curiel
>Priority: Minor
>
> This issue is meant to track work done to implement 
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplified+development+workflow



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3611) Simplified development workflow

2019-01-01 Thread Gerardo Curiel (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gerardo Curiel updated AIRFLOW-3611:

Summary: Simplified development workflow  (was: Simplify development 
workflow)

> Simplified development workflow
> ---
>
> Key: AIRFLOW-3611
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3611
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Gerardo Curiel
>Priority: Minor
>
> This issue is meant to track work done to implement 
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplify+development+workflow



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3610) Set AWS Region when creating an EMR job flow via EmrCreateJobFlowOperator



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731758#comment-16731758
 ] 

ASF GitHub Bot commented on AIRFLOW-3610:
-

danabananarama commented on pull request #4418: [AIRFLOW-3610] Add region param 
for EMR jobflow creation
URL: https://github.com/apache/incubator-airflow/pull/4418
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [X] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title.
 - https://issues.apache.org/jira/browse/AIRFLOW-3610
   
   ### Description
   
   - [X] Here are some details about my PR, including screenshots of any UI 
changes:
   This PR is to allow setting the AWS region in the `EmrCreateJobFlowOperator` 
so that we can create an EMR job flow in any region.
   
   ### Tests
   
   - [X] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   Amending 
[test](https://github.com/danabananarama/incubator-airflow/blob/c451cedc3b760059c04293b25772c42db7917f10/tests/contrib/operators/test_emr_create_job_flow_operator.py#L81),
 did not add new test for functionality as it's essentially just persisting a 
new init param. Let me know if a new test needs to be added!
   
   ### Commits
   
   - [X] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [X] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
   
   ### Code Quality
   
   - [X] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Set AWS Region when creating an EMR job flow via EmrCreateJobFlowOperator
> -
>
> Key: AIRFLOW-3610
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3610
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: aws
>Affects Versions: 1.10.0
>Reporter: Dana Ma
>Assignee: Dana Ma
>Priority: Minor
>
> The EmrCreateJobFlowOperator currently does not accept AWS region name as a 
> parameter, so the only option is to create the EMR job flows in the default 
> region. We would like to be able to set the AWS region for which we create 
> the job flow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] danabananarama opened a new pull request #4418: [AIRFLOW-3610] Add region param for EMR jobflow creation

danabananarama opened a new pull request #4418: [AIRFLOW-3610] Add region param
for EMR jobflow creation
URL: https://github.com/apache/incubator-airflow/pull/4418

Make sure you have checked _all_ steps below.

### Jira

- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title.
- https://issues.apache.org/jira/browse/AIRFLOW-3610

### Description

- [X] Here are some details about my PR, including screenshots of any UI
changes:
This PR is to allow setting the AWS region in the `EmrCreateJobFlowOperator`
so that we can create an EMR job flow in any region.

### Tests

- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Amending
[test](https://github.com/danabananarama/incubator-airflow/blob/c451cedc3b760059c04293b25772c42db7917f10/tests/contrib/operators/test_emr_create_job_flow_operator.py#L81),
did not add new test for functionality as it's essentially just persisting a
new init param. Let me know if a new test needs to be added!

### Commits

- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"

### Documentation

- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does

### Code Quality

- [X] Passes `flake8`

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-3610) EMR CreateJobFlowOperator and Hook don't accept AWS region as a parameter

2019-01-01 Thread Dana Ma (JIRA)

Dana Ma created AIRFLOW-3610:


 Summary: EMR CreateJobFlowOperator and Hook don't accept AWS 
region as a parameter
 Key: AIRFLOW-3610
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3610
 Project: Apache Airflow
  Issue Type: Improvement
  Components: aws
Affects Versions: 1.10.0
Reporter: Dana Ma
Assignee: Dana Ma


The EmrCreateJobFlowOperator currently does not accept AWS region name as a 
parameter, so the only option is to create the EMR job flows in the default 
region. We would like to be able to set the AWS region for which we create the 
job flow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AIRFLOW-3610) Set AWS Region when creating an EMR job flow via EmrCreateJobFlowOperator

2019-01-01 Thread Dana Ma (JIRA)



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dana Ma updated AIRFLOW-3610:
-
Summary: Set AWS Region when creating an EMR job flow via 
EmrCreateJobFlowOperator  (was: EMR CreateJobFlowOperator and Hook don't accept 
AWS region as a parameter)

> Set AWS Region when creating an EMR job flow via EmrCreateJobFlowOperator
> -
>
> Key: AIRFLOW-3610
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3610
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: aws
>Affects Versions: 1.10.0
>Reporter: Dana Ma
>Assignee: Dana Ma
>Priority: Minor
>
> The EmrCreateJobFlowOperator currently does not accept AWS region name as a 
> parameter, so the only option is to create the EMR job flows in the default 
> region. We would like to be able to set the AWS region for which we create 
> the job flow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] codecov-io edited a comment on issue #4409: WIP [AIRFLOW-3601] add location support to BigQuery operators

codecov-io edited a comment on issue #4409: WIP [AIRFLOW-3601] add location 
support to BigQuery operators
URL: 
https://github.com/apache/incubator-airflow/pull/4409#issuecomment-450601800
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=h1)
 Report
   > Merging 
[#4409](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/b50ffa091dd4cdf1dabc8ed4ee701cac9d8318c6?src=pr=desc)
 will **decrease** coverage by `0.04%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4409/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=tree)
   
   ```diff
   @@Coverage Diff@@
   ##   master   #4409  +/-   ##
   =
   - Coverage   78.25%   78.2%   -0.05% 
   =
 Files 204 204  
 Lines   16433   16445  +12 
   =
   + Hits12859   12861   +2 
   - Misses   35743584  +10
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/api/experimental/endpoints.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBpL2V4cGVyaW1lbnRhbC9lbmRwb2ludHMucHk=)
 | `71.12% <0%> (-19.02%)` | :arrow_down: |
   | 
[airflow/api/common/experimental/delete\_dag.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9kZWxldGVfZGFnLnB5)
 | `84% <0%> (-4%)` | :arrow_down: |
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `74.88% <0%> (-2.56%)` | :arrow_down: |
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `92.55% <0%> (+0.02%)` | :arrow_up: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `69.69% <0%> (+0.22%)` | :arrow_up: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `73.71% <0%> (+0.93%)` | :arrow_up: |
   | 
[airflow/hooks/samba\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9zYW1iYV9ob29rLnB5)
 | `38.88% <0%> (+38.88%)` | :arrow_up: |
   | 
[airflow/operators/hive\_to\_samba\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvaGl2ZV90b19zYW1iYV9vcGVyYXRvci5weQ==)
 | `100% <0%> (+100%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=footer).
 Last update 
[b50ffa0...d6e6287](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4416: [AIRFLOW-3608] Allow devs to submit custom airflow images to k8s CI

codecov-io commented on issue #4416: [AIRFLOW-3608]  Allow devs to submit 
custom airflow images to k8s CI
URL: 
https://github.com/apache/incubator-airflow/pull/4416#issuecomment-450767774
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=h1)
 Report
   > Merging 
[#4416](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8938d2c727b16762529ef9f1e257ba5fa39f4bea?src=pr=desc)
 will **decrease** coverage by `0.17%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4416/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4416  +/-   ##
   ==
   - Coverage   78.38%   78.21%   -0.18% 
   ==
 Files 204  204  
 Lines   1644516445  
   ==
   - Hits1289012862  -28 
   - Misses   3555 3583  +28
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4416/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `74.88% <0%> (-2.56%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=footer).
 Last update 
[8938d2c...3011e19](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4416: [AIRFLOW-3608] Allow devs to submit custom airflow images to k8s CI

codecov-io edited a comment on issue #4416: [AIRFLOW-3608]  Allow devs to 
submit custom airflow images to k8s CI
URL: 
https://github.com/apache/incubator-airflow/pull/4416#issuecomment-450767774
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=h1)
 Report
   > Merging 
[#4416](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8938d2c727b16762529ef9f1e257ba5fa39f4bea?src=pr=desc)
 will **decrease** coverage by `0.17%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4416/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4416  +/-   ##
   ==
   - Coverage   78.38%   78.21%   -0.18% 
   ==
 Files 204  204  
 Lines   1644516445  
   ==
   - Hits1289012862  -28 
   - Misses   3555 3583  +28
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4416/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `74.88% <0%> (-2.56%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=footer).
 Last update 
[8938d2c...3011e19](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4416: [AIRFLOW-3608] Allow devs to submit custom airflow images to k8s CI

codecov-io edited a comment on issue #4416: [AIRFLOW-3608]  Allow devs to 
submit custom airflow images to k8s CI
URL: 
https://github.com/apache/incubator-airflow/pull/4416#issuecomment-450767774
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=h1)
 Report
   > Merging 
[#4416](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8938d2c727b16762529ef9f1e257ba5fa39f4bea?src=pr=desc)
 will **decrease** coverage by `0.17%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4416/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4416  +/-   ##
   ==
   - Coverage   78.38%   78.21%   -0.18% 
   ==
 Files 204  204  
 Lines   1644516445  
   ==
   - Hits1289012862  -28 
   - Misses   3555 3583  +28
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4416/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5)
 | `74.88% <0%> (-2.56%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=footer).
 Last update 
[8938d2c...3011e19](https://codecov.io/gh/apache/incubator-airflow/pull/4416?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dimberman edited a comment on issue #4247: [AIRFLOW-3402] Support global k8s affinity and toleration configs

dimberman edited a comment on issue #4247: [AIRFLOW-3402] Support global k8s 
affinity and toleration configs
URL: 
https://github.com/apache/incubator-airflow/pull/4247#issuecomment-450765778
 
 
   @kppullin looks like it's timing out on this test
   
   ```
   test_pod_node_selectors 
(tests.contrib.minikube.test_kubernetes_pod_operator.KubernetesPodOperatorTest) 
... 
   No output has been received in the last 10m0s, this potentially indicates a 
stalled build or something wrong with the build itself.
   Check the details on how to adjust your build configuration on: 
https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
   ```
   
   
   Seems like it could be related?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dimberman edited a comment on issue #4247: [AIRFLOW-3402] Support global k8s affinity and toleration configs

dimberman edited a comment on issue #4247: [AIRFLOW-3402] Support global k8s 
affinity and toleration configs
URL: 
https://github.com/apache/incubator-airflow/pull/4247#issuecomment-450765778
 
 
   @kppullin looks like it's timing out on this test
   
   ```test_pod_node_selectors 
(tests.contrib.minikube.test_kubernetes_pod_operator.KubernetesPodOperatorTest) 
... 
   No output has been received in the last 10m0s, this potentially indicates a 
stalled build or something wrong with the build itself.
   Check the details on how to adjust your build configuration on: 
https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
   ```
   
   
   Seems like it could be related?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dimberman commented on issue #4247: [AIRFLOW-3402] Support global k8s affinity and toleration configs

dimberman commented on issue #4247: [AIRFLOW-3402] Support global k8s affinity 
and toleration configs
URL: 
https://github.com/apache/incubator-airflow/pull/4247#issuecomment-450765778
 
 
   @kppullin looks like it's timing out on this test
   
   ```test_pod_node_selectors 
(tests.contrib.minikube.test_kubernetes_pod_operator.KubernetesPodOperatorTest) 
... 
   No output has been received in the last 10m0s, this potentially indicates a 
stalled build or something wrong with the build itself.
   Check the details on how to adjust your build configuration on: 
https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received```
   
   Seems like it could be related?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io commented on issue #4417: [AIRFLOW-3609] Fix bug in persistent volumes readWriteMany

codecov-io commented on issue #4417: [AIRFLOW-3609] Fix bug in persistent 
volumes readWriteMany
URL: 
https://github.com/apache/incubator-airflow/pull/4417#issuecomment-450749038
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4417?src=pr=h1)
 Report
   > Merging 
[#4417](https://codecov.io/gh/apache/incubator-airflow/pull/4417?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8938d2c727b16762529ef9f1e257ba5fa39f4bea?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4417/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4417?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#4417   +/-   ##
   ===
 Coverage   78.38%   78.38%   
   ===
 Files 204  204   
 Lines   1644516445   
   ===
 Hits1289012890   
 Misses   3555 3555
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4417?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4417?src=pr=footer).
 Last update 
[8938d2c...47a9562](https://codecov.io/gh/apache/incubator-airflow/pull/4417?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4417: [AIRFLOW-3609] Fix bug in persistent volumes readWriteMany

codecov-io edited a comment on issue #4417: [AIRFLOW-3609] Fix bug in 
persistent volumes readWriteMany
URL: 
https://github.com/apache/incubator-airflow/pull/4417#issuecomment-450749038
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4417?src=pr=h1)
 Report
   > Merging 
[#4417](https://codecov.io/gh/apache/incubator-airflow/pull/4417?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8938d2c727b16762529ef9f1e257ba5fa39f4bea?src=pr=desc)
 will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4417/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4417?src=pr=tree)
   
   ```diff
   @@   Coverage Diff   @@
   ##   master#4417   +/-   ##
   ===
 Coverage   78.38%   78.38%   
   ===
 Files 204  204   
 Lines   1644516445   
   ===
 Hits1289012890   
 Misses   3555 3555
   ```
   
   
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4417?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4417?src=pr=footer).
 Last update 
[8938d2c...47a9562](https://codecov.io/gh/apache/incubator-airflow/pull/4417?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-3609) PersistentVolumes in k8s CI have incorrect permissions

2019-01-01 Thread Daniel Imberman (JIRA)

Daniel Imberman created AIRFLOW-3609:


 Summary: PersistentVolumes in k8s CI have incorrect permissions
 Key: AIRFLOW-3609
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3609
 Project: Apache Airflow
  Issue Type: Bug
  Components: kubernetes
Reporter: Daniel Imberman
Assignee: Daniel Imberman


When running integration tests on a k8s cluster vs. Minikube
 I discovered that we were actually using an invalid permission
 structure for our persistent volume. This commit fixes that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AIRFLOW-3609) PersistentVolumes in k8s CI have incorrect permissions



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731649#comment-16731649
 ] 

ASF GitHub Bot commented on AIRFLOW-3609:
-

dimberman commented on pull request #4417: [AIRFLOW-3609] Fix bug in persistent 
volumes readWriteMany
URL: https://github.com/apache/incubator-airflow/pull/4417
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-3609
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   When running integration tests on a k8s cluster vs. Minikube
   I discovered that we were actually using an invalid permission
   structure for our persistent volume. This commit fixes that.
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> PersistentVolumes in k8s CI have incorrect permissions
> --
>
> Key: AIRFLOW-3609
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3609
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: kubernetes
>Reporter: Daniel Imberman
>Assignee: Daniel Imberman
>Priority: Minor
>
> When running integration tests on a k8s cluster vs. Minikube
>  I discovered that we were actually using an invalid permission
>  structure for our persistent volume. This commit fixes that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] dimberman opened a new pull request #4417: [AIRFLOW-3609] Fix bug in persistent volumes readWriteMany

dimberman opened a new pull request #4417: [AIRFLOW-3609] Fix bug in persistent 
volumes readWriteMany
URL: https://github.com/apache/incubator-airflow/pull/4417
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-3609
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   When running integration tests on a k8s cluster vs. Minikube
   I discovered that we were actually using an invalid permission
   structure for our persistent volume. This commit fixes that.
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] dimberman opened a new pull request #4416: [AIRFLOW-3608] Allow devs to submit custom airflow images to k8s CI

dimberman opened a new pull request #4416: [AIRFLOW-3608]  Allow devs to submit 
custom airflow images to k8s CI
URL: https://github.com/apache/incubator-airflow/pull/4416
 
 
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [ ] My PR addresses the following [Airflow 
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references 
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
 - https://issues.apache.org/jira/browse/AIRFLOW-3608
 - In case you are fixing a typo in the documentation you can prepend your 
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
   
   ### Description
   
   - [ ] Here are some details about my PR, including screenshots of any UI 
changes:
   
   To help move away from Minikube, we need to remove the dependency on
   a local docker registry and move towards a solution that can be used
   in any kubernetes cluster. Custom image names allow users to use
   systems like docker, artifactory and gcr
   
   ### Tests
   
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   
   ### Commits
   
   - [ ] My commits all reference Jira issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
 1. Subject is separated from body by a blank line
 1. Subject is limited to 50 characters (not including Jira issue reference)
 1. Subject does not end with a period
 1. Subject uses the imperative mood ("add", not "adding")
 1. Body wraps at 72 characters
 1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes 
how to use it.
 - When adding new operators/hooks/sensors, the autoclass documentation 
generation needs to be added.
 - All the public functions and the classes in the PR contain docstrings 
that explain what it does
   
   ### Code Quality
   
   - [ ] Passes `flake8`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (AIRFLOW-3608) Allow users to submit custom airflow images when running Airflow Kubernetes CI

2019-01-01 Thread Daniel Imberman (JIRA)

Daniel Imberman created AIRFLOW-3608:


 Summary: Allow users to submit custom airflow images when running 
Airflow Kubernetes CI
 Key: AIRFLOW-3608
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3608
 Project: Apache Airflow
  Issue Type: Improvement
  Components: kubernetes
Reporter: Daniel Imberman
Assignee: Daniel Imberman


To help move away from Minikube, we need to remove the dependency on
 a local docker registry and move towards a solution that can be used
 in any kubernetes cluster. Custom image names allow users to use
 systems like docker, artifactory and gcr



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] codecov-io edited a comment on issue #4390: [AIRFLOW-3584] Use ORM DAGs for index view.

codecov-io edited a comment on issue #4390: [AIRFLOW-3584] Use ORM DAGs for 
index view.
URL: 
https://github.com/apache/incubator-airflow/pull/4390#issuecomment-450291500
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4390?src=pr=h1)
 Report
   > Merging 
[#4390](https://codecov.io/gh/apache/incubator-airflow/pull/4390?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8938d2c727b16762529ef9f1e257ba5fa39f4bea?src=pr=desc)
 will **increase** coverage by `0.06%`.
   > The diff coverage is `90%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4390/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4390?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4390  +/-   ##
   ==
   + Coverage   78.38%   78.44%   +0.06% 
   ==
 Files 204  204  
 Lines   1644516440   -5 
   ==
   + Hits1289012896   +6 
   + Misses   3555 3544  -11
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4390?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/incubator-airflow/pull/4390/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `92.63% <100%> (+0.02%)` | :arrow_up: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4390/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `69.76% <73.33%> (+0.07%)` | :arrow_up: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4390/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `73.79% <90%> (+0.07%)` | :arrow_up: |
   | 
[airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/incubator-airflow/pull/4390/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5)
 | `81.81% <94.73%> (+3.24%)` | :arrow_up: |
   | 
[airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/incubator-airflow/pull/4390/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==)
 | `59.85% <0%> (+0.17%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4390?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4390?src=pr=footer).
 Last update 
[8938d2c...42b5dbd](https://codecov.io/gh/apache/incubator-airflow/pull/4390?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4412: [AIRFLOW-3605] Load plugins from entry_points

codecov-io edited a comment on issue #4412: [AIRFLOW-3605] Load plugins from 
entry_points
URL: 
https://github.com/apache/incubator-airflow/pull/4412#issuecomment-450662547
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=h1)
 Report
   > Merging 
[#4412](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8938d2c727b16762529ef9f1e257ba5fa39f4bea?src=pr=desc)
 will **decrease** coverage by `0.02%`.
   > The diff coverage is `66.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4412/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4412  +/-   ##
   ==
   - Coverage   78.38%   78.36%   -0.03% 
   ==
 Files 204  204  
 Lines   1644516462  +17 
   ==
   + Hits1289012900  +10 
   - Misses   3555 3562   +7
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/plugins\_manager.py](https://codecov.io/gh/apache/incubator-airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=)
 | `87.5% <66.66%> (-5.61%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=footer).
 Last update 
[8938d2c...5947470](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4412: [AIRFLOW-3605] Load plugins from entry_points

codecov-io edited a comment on issue #4412: [AIRFLOW-3605] Load plugins from 
entry_points
URL: 
https://github.com/apache/incubator-airflow/pull/4412#issuecomment-450662547
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=h1)
 Report
   > Merging 
[#4412](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8938d2c727b16762529ef9f1e257ba5fa39f4bea?src=pr=desc)
 will **decrease** coverage by `0.02%`.
   > The diff coverage is `66.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4412/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4412  +/-   ##
   ==
   - Coverage   78.38%   78.36%   -0.03% 
   ==
 Files 204  204  
 Lines   1644516462  +17 
   ==
   + Hits1289012900  +10 
   - Misses   3555 3562   +7
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/plugins\_manager.py](https://codecov.io/gh/apache/incubator-airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=)
 | `87.5% <66.66%> (-5.61%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=footer).
 Last update 
[8938d2c...5947470](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4409: WIP [AIRFLOW-3601] add location support to BigQuery operators

codecov-io edited a comment on issue #4409: WIP [AIRFLOW-3601] add location 
support to BigQuery operators
URL: 
https://github.com/apache/incubator-airflow/pull/4409#issuecomment-450601800
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=h1)
 Report
   > Merging 
[#4409](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/b50ffa091dd4cdf1dabc8ed4ee701cac9d8318c6?src=pr=desc)
 will **increase** coverage by `0.12%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4409/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4409  +/-   ##
   ==
   + Coverage   78.25%   78.37%   +0.12% 
   ==
 Files 204  204  
 Lines   1643316445  +12 
   ==
   + Hits1285912889  +30 
   + Misses   3574 3556  -18
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/api/experimental/endpoints.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBpL2V4cGVyaW1lbnRhbC9lbmRwb2ludHMucHk=)
 | `71.12% <0%> (-19.02%)` | :arrow_down: |
   | 
[airflow/api/common/experimental/delete\_dag.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9kZWxldGVfZGFnLnB5)
 | `84% <0%> (-4%)` | :arrow_down: |
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `92.55% <0%> (+0.02%)` | :arrow_up: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `69.69% <0%> (+0.22%)` | :arrow_up: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `73.71% <0%> (+0.93%)` | :arrow_up: |
   | 
[airflow/hooks/samba\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9zYW1iYV9ob29rLnB5)
 | `38.88% <0%> (+38.88%)` | :arrow_up: |
   | 
[airflow/operators/hive\_to\_samba\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvaGl2ZV90b19zYW1iYV9vcGVyYXRvci5weQ==)
 | `100% <0%> (+100%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=footer).
 Last update 
[b50ffa0...1ff0bc8](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-3607) Decreasing scheduler delay between tasks



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amichai Horvitz reassigned AIRFLOW-3607:


Assignee: Amichai Horvitz  (was: Dan Davydov)

> Decreasing scheduler delay between tasks
> 
>
> Key: AIRFLOW-3607
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3607
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Affects Versions: 1.10.0, 1.10.1
> Environment: ubuntu 14.04
>Reporter: Amichai Horvitz
>Assignee: Amichai Horvitz
>Priority: Major
> Fix For: 1.10.2
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> I came across the TODO in airflow/ti_deps/deps/trigger_rule_dep (line 52) 
> that says instead of checking the query for every task let the tasks report 
> to the dagrun. I have a dag with many tasks and the delay between tasks can 
> rise to 10 seconds or more, I already changed the configuration, added 
> processes and memory, checked the code and did research, profiling and other 
> experiments. I hope that this change will make a drastic change in the delay. 
> I would be happy to discuss this solution, the research and other solutions 
> for this issue.  
> Thanks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] codecov-io edited a comment on issue #4412: [AIRFLOW-3605] Load plugins from entry_points

codecov-io edited a comment on issue #4412: [AIRFLOW-3605] Load plugins from 
entry_points
URL: 
https://github.com/apache/incubator-airflow/pull/4412#issuecomment-450662547
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=h1)
 Report
   > Merging 
[#4412](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8938d2c727b16762529ef9f1e257ba5fa39f4bea?src=pr=desc)
 will **decrease** coverage by `0.02%`.
   > The diff coverage is `66.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4412/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4412  +/-   ##
   ==
   - Coverage   78.38%   78.36%   -0.03% 
   ==
 Files 204  204  
 Lines   1644516462  +17 
   ==
   + Hits1289012900  +10 
   - Misses   3555 3562   +7
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/plugins\_manager.py](https://codecov.io/gh/apache/incubator-airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=)
 | `87.5% <66.66%> (-5.61%)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=footer).
 Last update 
[8938d2c...6e587f4](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4412: [AIRFLOW-3605] Load plugins from entry_points

codecov-io edited a comment on issue #4412: [AIRFLOW-3605] Load plugins from 
entry_points
URL: 
https://github.com/apache/incubator-airflow/pull/4412#issuecomment-450662547
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=h1)
 Report
   > Merging 
[#4412](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/8938d2c727b16762529ef9f1e257ba5fa39f4bea?src=pr=desc)
 will **decrease** coverage by `62.12%`.
   > The diff coverage is `66.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4412/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=tree)
   
   ```diff
   @@ Coverage Diff @@
   ##   master#4412   +/-   ##
   ===
   - Coverage   78.38%   16.25%   -62.13% 
   ===
 Files 204  204   
 Lines   1644516462   +17 
   ===
   - Hits12890 2676-10214 
   - Misses   355513786+10231
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/plugins\_manager.py](https://codecov.io/gh/apache/incubator-airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=)
 | `86.53% <66.66%> (-6.57%)` | :arrow_down: |
   | 
[airflow/www\_rbac/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92YWxpZGF0b3JzLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/incubator-airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/www/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmFsaWRhdG9ycy5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/api/common/experimental/get\_dag\_runs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfZGFnX3J1bnMucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/www/forms.py](https://codecov.io/gh/apache/incubator-airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvZm9ybXMucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4412/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | ... and [166 
more](https://codecov.io/gh/apache/incubator-airflow/pull/4412/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=footer).
 Last update 
[8938d2c...6e587f4](https://codecov.io/gh/apache/incubator-airflow/pull/4412?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] codecov-io edited a comment on issue #4409: WIP [AIRFLOW-3601] add location support to BigQuery operators

codecov-io edited a comment on issue #4409: WIP [AIRFLOW-3601] add location 
support to BigQuery operators
URL: 
https://github.com/apache/incubator-airflow/pull/4409#issuecomment-450601800
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=h1)
 Report
   > Merging 
[#4409](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/b50ffa091dd4cdf1dabc8ed4ee701cac9d8318c6?src=pr=desc)
 will **decrease** coverage by `62.17%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4409/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=tree)
   
   ```diff
   @@ Coverage Diff @@
   ##   master#4409   +/-   ##
   ===
   - Coverage   78.25%   16.07%   -62.18% 
   ===
 Files 204  204   
 Lines   1643316445   +12 
   ===
   - Hits12859 2644-10215 
   - Misses   357413801+10227
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www\_rbac/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92YWxpZGF0b3JzLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/www/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmFsaWRhdG9ycy5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/api/common/experimental/get\_dag\_runs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfZGFnX3J1bnMucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/www/forms.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvZm9ybXMucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/operators/postgres\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvcG9zdGdyZXNfb3BlcmF0b3IucHk=)
 | `0% <0%> (-100%)` | :arrow_down: |
   | 
[airflow/api/common/experimental/get\_task.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfdGFzay5weQ==)
 | `0% <0%> (-100%)` | :arrow_down: |
   | ... and [164 
more](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=footer).
 Last update 
[b50ffa0...7eb3611](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (AIRFLOW-3607) Decreasing scheduler delay between tasks



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amichai Horvitz reassigned AIRFLOW-3607:


Assignee: Dan Davydov

> Decreasing scheduler delay between tasks
> 
>
> Key: AIRFLOW-3607
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3607
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Affects Versions: 1.10.0, 1.10.1
> Environment: ubuntu 14.04
>Reporter: Amichai Horvitz
>Assignee: Dan Davydov
>Priority: Major
> Fix For: 1.10.2
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> I came across the TODO in airflow/ti_deps/deps/trigger_rule_dep (line 52) 
> that says instead of checking the query for every task let the tasks report 
> to the dagrun. I have a dag with many tasks and the delay between tasks can 
> rise to 10 seconds or more, I already changed the configuration, added 
> processes and memory, checked the code and did research, profiling and other 
> experiments. I hope that this change will make a drastic change in the delay. 
> I would be happy to discuss this solution, the research and other solutions 
> for this issue.  
> Thanks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] codecov-io edited a comment on issue #4409: WIP [AIRFLOW-3601] add location support to BigQuery operators

codecov-io edited a comment on issue #4409: WIP [AIRFLOW-3601] add location 
support to BigQuery operators
URL: 
https://github.com/apache/incubator-airflow/pull/4409#issuecomment-450601800
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=h1)
 Report
   > Merging 
[#4409](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-airflow/commit/b50ffa091dd4cdf1dabc8ed4ee701cac9d8318c6?src=pr=desc)
 will **increase** coverage by `0.13%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-airflow/pull/4409/graphs/tree.svg?width=650=WdLKlKHOAU=150=pr)](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#4409  +/-   ##
   ==
   + Coverage   78.25%   78.38%   +0.13% 
   ==
 Files 204  204  
 Lines   1643316445  +12 
   ==
   + Hits1285912890  +31 
   + Misses   3574 3555  -19
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=tree) 
| Coverage Δ | |
   |---|---|---|
   | 
[airflow/www/api/experimental/endpoints.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBpL2V4cGVyaW1lbnRhbC9lbmRwb2ludHMucHk=)
 | `71.12% <0%> (-19.02%)` | :arrow_down: |
   | 
[airflow/api/common/experimental/delete\_dag.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9kZWxldGVfZGFnLnB5)
 | `84% <0%> (-4%)` | :arrow_down: |
   | 
[airflow/models/\_\_init\_\_.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvX19pbml0X18ucHk=)
 | `92.6% <0%> (+0.07%)` | :arrow_up: |
   | 
[airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=)
 | `69.69% <0%> (+0.22%)` | :arrow_up: |
   | 
[airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==)
 | `73.71% <0%> (+0.93%)` | :arrow_up: |
   | 
[airflow/hooks/samba\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9zYW1iYV9ob29rLnB5)
 | `38.88% <0%> (+38.88%)` | :arrow_up: |
   | 
[airflow/operators/hive\_to\_samba\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4409/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvaGl2ZV90b19zYW1iYV9vcGVyYXRvci5weQ==)
 | `100% <0%> (+100%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=footer).
 Last update 
[b50ffa0...6c5bbd1](https://codecov.io/gh/apache/incubator-airflow/pull/4409?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (AIRFLOW-3607) Decreasing scheduler delay between tasks



 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amichai Horvitz updated AIRFLOW-3607:
-
Description: 
I came across the TODO in airflow/ti_deps/deps/trigger_rule_dep (line 52) that 
says instead of checking the query for every task let the tasks report to the 
dagrun. I have a dag with many tasks and the delay between tasks can rise to 10 
seconds or more, I already changed the configuration, added processes and 
memory, checked the code and did research, profiling and other experiments. I 
hope that this change will make a drastic change in the delay. I would be happy 
to discuss this solution, the research and other solutions for this issue.  

Thanks

  was:
I came across the TODO in airflow/ti_deps/deps/trigger_rule_dep (line 52) that 
says instead of checking the query for every task let the task report to the 
dagrun. I have a dag with many task and delay between tasks can rise to 10 
seconds or more, I already changed the configuration, added processes and 
memory, checked the code and did research, profiling and other experiments. I 
hope that this will make a drastic change in the delay. I would be happy to 
discuss this solution, the research and other solutions for this issue.  

Thanks


> Decreasing scheduler delay between tasks
> 
>
> Key: AIRFLOW-3607
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3607
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: scheduler
>Affects Versions: 1.10.0, 1.10.1
> Environment: ubuntu 14.04
>Reporter: Amichai Horvitz
>Priority: Major
> Fix For: 1.10.2
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> I came across the TODO in airflow/ti_deps/deps/trigger_rule_dep (line 52) 
> that says instead of checking the query for every task let the tasks report 
> to the dagrun. I have a dag with many tasks and the delay between tasks can 
> rise to 10 seconds or more, I already changed the configuration, added 
> processes and memory, checked the code and did research, profiling and other 
> experiments. I hope that this change will make a drastic change in the delay. 
> I would be happy to discuss this solution, the research and other solutions 
> for this issue.  
> Thanks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-3607) Decreasing scheduler delay between tasks

Amichai Horvitz created AIRFLOW-3607:


 Summary: Decreasing scheduler delay between tasks
 Key: AIRFLOW-3607
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3607
 Project: Apache Airflow
  Issue Type: Improvement
  Components: scheduler
Affects Versions: 1.10.1, 1.10.0
 Environment: ubuntu 14.04
Reporter: Amichai Horvitz
 Fix For: 1.10.2


I came across the TODO in airflow/ti_deps/deps/trigger_rule_dep (line 52) that 
says instead of checking the query for every task let the task report to the 
dagrun. I have a dag with many task and delay between tasks can rise to 10 
seconds or more, I already changed the configuration, added processes and 
memory, checked the code and did research, profiling and other experiments. I 
hope that this will make a drastic change in the delay. I would be happy to 
discuss this solution, the research and other solutions for this issue.  

Thanks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] XD-DENG edited a comment on issue #4415: [AIRFLOW-3606] Fix Flake8 test & fix the Flake8 errors introduced since Flake8 test was broken