[GitHub] [airflow] potiuk commented on a change in pull request #4938: [AIRFLOW-4117] Multi-staging Image - Travis CI tests [Step 3/3]
potiuk commented on a change in pull request #4938: [AIRFLOW-4117]
Multi-staging Image - Travis CI tests [Step 3/3]
URL: https://github.com/apache/airflow/pull/4938#discussion_r300236996
##
File path: scripts/ci/kubernetes/minikube/start_minikube.sh
##
@@ -41,11 +39,10 @@ case "${_UNAME_OUT}" in
esac
echo "Local OS is ${_MY_OS}"
-export MINIKUBE_WANTUPDATENOTIFICATION=false
export MINIKUBE_WANTREPORTERRORPROMPT=false
export CHANGE_MINIKUBE_NONE_USER=true
-cd $_MY_DIR
+cd ${_MY_DIR}
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #4938: [AIRFLOW-4117] Multi-staging Image - Travis CI tests [Step 3/3]
potiuk commented on a change in pull request #4938: [AIRFLOW-4117]
Multi-staging Image - Travis CI tests [Step 3/3]
URL: https://github.com/apache/airflow/pull/4938#discussion_r300236965
##
File path: scripts/ci/kubernetes/minikube/_k8s.sh
##
@@ -26,15 +27,15 @@ function _wait_for_ready () {
local evidence="$1"
shift
local attempts=40
- echo "Waiting till ready (count: $count): $@"
- while [[ "$count" < $("$@" 2>&1 | tail -n +2 | awk '{print $2}' | grep -c
$evidence) ]];
+ echo "Waiting till ready (count: ${count}): $*"
+ while [[ "${count}" < $("$@" 2>&1 | tail -n +2 | awk '{print $2}' | grep -c
${evidence}) ]];
do
if [[ "$attempts" = "1" ]]; then
- echo "Last run: $@"
+ echo "Last run: $*"
Review comment:
OK. that was shellcheck-reported problem:
https://www.shellcheck.net/wiki/SC2145 - mixing arrays and strings might have
unexpected consequences. I kept the change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #4938: [AIRFLOW-4117] Multi-staging Image - Travis CI tests [Step 3/3]
potiuk commented on a change in pull request #4938: [AIRFLOW-4117]
Multi-staging Image - Travis CI tests [Step 3/3]
URL: https://github.com/apache/airflow/pull/4938#discussion_r300236357
##
File path: scripts/ci/kubernetes/kube/deploy.sh
##
@@ -123,72 +123,78 @@ ${SED_COMMAND} -i "s|{{AIRFLOW_TAG}}|$AIRFLOW_TAG|g"
${BUILD_DIRNAME}/airflow.ya
${SED_COMMAND} -i "s|{{CONFIGMAP_GIT_REPO}}|$CONFIGMAP_GIT_REPO|g"
${BUILD_DIRNAME}/airflow.yaml
${SED_COMMAND} -i "s|{{CONFIGMAP_BRANCH}}|$CONFIGMAP_BRANCH|g"
${BUILD_DIRNAME}/airflow.yaml
${SED_COMMAND} -i "s|{{INIT_DAGS_VOLUME_NAME}}|$INIT_DAGS_VOLUME_NAME|g"
${BUILD_DIRNAME}/airflow.yaml
-${SED_COMMAND} -i
"s|{{POD_AIRFLOW_DAGS_VOLUME_NAME}}|$POD_AIRFLOW_DAGS_VOLUME_NAME|g"
${BUILD_DIRNAME}/airflow.yaml
+${SED_COMMAND} -i
"s|{{POD_AIRFLOW_DAGS_VOLUME_NAME}}|$POD_AIRFLOW_DAGS_VOLUME_NAME|g" \
+${BUILD_DIRNAME}/airflow.yaml
${SED_COMMAND} "s|{{CONFIGMAP_DAGS_FOLDER}}|$CONFIGMAP_DAGS_FOLDER|g" \
${TEMPLATE_DIRNAME}/configmaps.template.yaml >
${BUILD_DIRNAME}/configmaps.yaml
${SED_COMMAND} -i "s|{{CONFIGMAP_GIT_REPO}}|$CONFIGMAP_GIT_REPO|g"
${BUILD_DIRNAME}/configmaps.yaml
${SED_COMMAND} -i "s|{{CONFIGMAP_BRANCH}}|$CONFIGMAP_BRANCH|g"
${BUILD_DIRNAME}/configmaps.yaml
-${SED_COMMAND} -i
"s|{{CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT}}|$CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT|g"
${BUILD_DIRNAME}/configmaps.yaml
-${SED_COMMAND} -i
"s|{{CONFIGMAP_DAGS_VOLUME_CLAIM}}|$CONFIGMAP_DAGS_VOLUME_CLAIM|g"
${BUILD_DIRNAME}/configmaps.yaml
+${SED_COMMAND} -i
"s|{{CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT}}|$CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT|g"
\
+${BUILD_DIRNAME}/configmaps.yaml
+${SED_COMMAND} -i
"s|{{CONFIGMAP_DAGS_VOLUME_CLAIM}}|$CONFIGMAP_DAGS_VOLUME_CLAIM|g" \
+${BUILD_DIRNAME}/configmaps.yaml
cat ${BUILD_DIRNAME}/airflow.yaml
cat ${BUILD_DIRNAME}/configmaps.yaml
# Fix file permissions
+# TODO: Check this - this should be TRAVIS-independent
if [[ "${TRAVIS}" == true ]]; then
sudo chown -R travis.travis $HOME/.kube $HOME/.minikube
fi
-kubectl delete -f $DIRNAME/postgres.yaml
-kubectl delete -f $BUILD_DIRNAME/airflow.yaml
-kubectl delete -f $DIRNAME/secrets.yaml
+kubectl delete -f ${DIRNAME}/postgres.yaml
+kubectl delete -f ${BUILD_DIRNAME}/airflow.yaml
+kubectl delete -f ${DIRNAME}/secrets.yaml
set -e
-kubectl apply -f $DIRNAME/secrets.yaml
-kubectl apply -f $BUILD_DIRNAME/configmaps.yaml
-kubectl apply -f $DIRNAME/postgres.yaml
-kubectl apply -f $DIRNAME/volumes.yaml
-kubectl apply -f $BUILD_DIRNAME/airflow.yaml
+kubectl apply -f ${DIRNAME}/secrets.yaml
+kubectl apply -f ${BUILD_DIRNAME}/configmaps.yaml
+kubectl apply -f ${DIRNAME}/postgres.yaml
+kubectl apply -f ${DIRNAME}/volumes.yaml
+kubectl apply -f ${BUILD_DIRNAME}/airflow.yaml
dump_logs() {
echo "--- pod description ---"
- kubectl describe pod $POD
+ kubectl describe pod ${POD}
echo "--- webserver init container logs - init ---"
- kubectl logs $POD -c init || true
- if [ "${GIT_SYNC}" = 1 ]; then
+ kubectl logs ${POD} -c init || true
+ if [[ "${GIT_SYNC}" = 1 ]]; then
echo "--- webserver init container logs - git-sync-clone ---"
- kubectl logs $POD -c git-sync-clone || true
+ kubectl logs ${POD} -c git-sync-clone || true
fi
echo "--- webserver logs ---"
- kubectl logs $POD -c webserver || true
+ kubectl logs ${POD} -c webserver || true
echo "--- scheduler logs ---"
- kubectl logs $POD -c scheduler || true
+ kubectl logs ${POD} -c scheduler || true
echo "--"
}
set +x
# wait for up to 10 minutes for everything to be deployed
PODS_ARE_READY=0
-for i in {1..150}
+for((i=1;i<=150;i+=1))
Review comment:
Whole deploy.sh is also corrected with shellcheck now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #4938: [AIRFLOW-4117] Multi-staging Image - Travis CI tests [Step 3/3]
potiuk commented on a change in pull request #4938: [AIRFLOW-4117]
Multi-staging Image - Travis CI tests [Step 3/3]
URL: https://github.com/apache/airflow/pull/4938#discussion_r300236418
##
File path: scripts/ci/kubernetes/kube/deploy.sh
##
@@ -123,72 +123,78 @@ ${SED_COMMAND} -i "s|{{AIRFLOW_TAG}}|$AIRFLOW_TAG|g"
${BUILD_DIRNAME}/airflow.ya
${SED_COMMAND} -i "s|{{CONFIGMAP_GIT_REPO}}|$CONFIGMAP_GIT_REPO|g"
${BUILD_DIRNAME}/airflow.yaml
${SED_COMMAND} -i "s|{{CONFIGMAP_BRANCH}}|$CONFIGMAP_BRANCH|g"
${BUILD_DIRNAME}/airflow.yaml
${SED_COMMAND} -i "s|{{INIT_DAGS_VOLUME_NAME}}|$INIT_DAGS_VOLUME_NAME|g"
${BUILD_DIRNAME}/airflow.yaml
-${SED_COMMAND} -i
"s|{{POD_AIRFLOW_DAGS_VOLUME_NAME}}|$POD_AIRFLOW_DAGS_VOLUME_NAME|g"
${BUILD_DIRNAME}/airflow.yaml
+${SED_COMMAND} -i
"s|{{POD_AIRFLOW_DAGS_VOLUME_NAME}}|$POD_AIRFLOW_DAGS_VOLUME_NAME|g" \
+${BUILD_DIRNAME}/airflow.yaml
${SED_COMMAND} "s|{{CONFIGMAP_DAGS_FOLDER}}|$CONFIGMAP_DAGS_FOLDER|g" \
${TEMPLATE_DIRNAME}/configmaps.template.yaml >
${BUILD_DIRNAME}/configmaps.yaml
${SED_COMMAND} -i "s|{{CONFIGMAP_GIT_REPO}}|$CONFIGMAP_GIT_REPO|g"
${BUILD_DIRNAME}/configmaps.yaml
${SED_COMMAND} -i "s|{{CONFIGMAP_BRANCH}}|$CONFIGMAP_BRANCH|g"
${BUILD_DIRNAME}/configmaps.yaml
-${SED_COMMAND} -i
"s|{{CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT}}|$CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT|g"
${BUILD_DIRNAME}/configmaps.yaml
-${SED_COMMAND} -i
"s|{{CONFIGMAP_DAGS_VOLUME_CLAIM}}|$CONFIGMAP_DAGS_VOLUME_CLAIM|g"
${BUILD_DIRNAME}/configmaps.yaml
+${SED_COMMAND} -i
"s|{{CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT}}|$CONFIGMAP_GIT_DAGS_FOLDER_MOUNT_POINT|g"
\
+${BUILD_DIRNAME}/configmaps.yaml
+${SED_COMMAND} -i
"s|{{CONFIGMAP_DAGS_VOLUME_CLAIM}}|$CONFIGMAP_DAGS_VOLUME_CLAIM|g" \
+${BUILD_DIRNAME}/configmaps.yaml
cat ${BUILD_DIRNAME}/airflow.yaml
cat ${BUILD_DIRNAME}/configmaps.yaml
# Fix file permissions
+# TODO: Check this - this should be TRAVIS-independent
if [[ "${TRAVIS}" == true ]]; then
sudo chown -R travis.travis $HOME/.kube $HOME/.minikube
fi
-kubectl delete -f $DIRNAME/postgres.yaml
-kubectl delete -f $BUILD_DIRNAME/airflow.yaml
-kubectl delete -f $DIRNAME/secrets.yaml
+kubectl delete -f ${DIRNAME}/postgres.yaml
+kubectl delete -f ${BUILD_DIRNAME}/airflow.yaml
+kubectl delete -f ${DIRNAME}/secrets.yaml
set -e
-kubectl apply -f $DIRNAME/secrets.yaml
-kubectl apply -f $BUILD_DIRNAME/configmaps.yaml
-kubectl apply -f $DIRNAME/postgres.yaml
-kubectl apply -f $DIRNAME/volumes.yaml
-kubectl apply -f $BUILD_DIRNAME/airflow.yaml
+kubectl apply -f ${DIRNAME}/secrets.yaml
+kubectl apply -f ${BUILD_DIRNAME}/configmaps.yaml
+kubectl apply -f ${DIRNAME}/postgres.yaml
+kubectl apply -f ${DIRNAME}/volumes.yaml
+kubectl apply -f ${BUILD_DIRNAME}/airflow.yaml
Review comment:
Everything shellchecked.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #4938: [AIRFLOW-4117] Multi-staging Image - Travis CI tests [Step 3/3]
potiuk commented on a change in pull request #4938: [AIRFLOW-4117]
Multi-staging Image - Travis CI tests [Step 3/3]
URL: https://github.com/apache/airflow/pull/4938#discussion_r300234244
##
File path: scripts/ci/in_container/entrypoint_ci.sh
##
@@ -0,0 +1,220 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Bash sanity settings (error on exit, complain for undefined vars, error when
pipe fails)
+set -euo pipefail
+MY_DIR=$(cd "$(dirname "$0")"; pwd)
+
+if [[ ${AIRFLOW_CI_VERBOSE:="false"} == "true" ]]; then
+set -x
+fi
+
+# shellcheck source=./_check_in_container.sh
+. ${MY_DIR}/_check_in_container.sh
+
+AIRFLOW_ROOT="${MY_DIR}/../.."
+
+PYTHON_VERSION=${PYTHON_VERSION:=3.6}
+ENV=${ENV:=docker}
+BACKEND=${BACKEND:=sqlite}
+KUBERNETES_VERSION=${KUBERNETES_VERSION:=""}
+
+export AIRFLOW_HOME=${AIRFLOW_HOME:=${HOME}}
+
+if [[ -z "${AIRFLOW_SOURCES:=}" ]]; then
+echo
+echo AIRFLOW_SOURCES not set
+echo
+exit 1
+fi
+
+echo
+echo "Airflow home: ${AIRFLOW_HOME}"
+echo "Airflow sources: ${AIRFLOW_SOURCES}"
+echo "Airflow core SQL connection: ${AIRFLOW__CORE__SQL_ALCHEMY_CONN:=}"
+echo
+
+ARGS=( "$@" )
+
+RUN_TESTS=${RUN_TESTS:="true"}
+
+if [[ ! -d "${AIRFLOW_SOURCES}/airflow/www/node_modules" && "${CLEAN_FILES}"
== "false" ]]; then
+echo
+echo "Installing NPM modules as they are not yet installed (Sources
mounted from Host)"
+echo
+pushd "${AIRFLOW_SOURCES}/airflow/www/"
+npm ci
+echo
+popd
+fi
+if [[ ! -d "${AIRFLOW_SOURCES}/airflow/www/static/dist" && ${CLEAN_FILES} ==
"false" ]]; then
+pushd "${AIRFLOW_SOURCES}/airflow/www/"
+echo
+echo "Building production version of javascript files (Sources mounted
from Host)"
+echo
+echo
+npm run prod
+echo
+echo
+popd
+fi
+
+if [[ ${AIRFLOW_CI_VERBOSE} == "true" ]]; then
+echo
+echo "Using ${HADOOP_DISTRO:=} distribution of Hadoop from
${HADOOP_HOME:=}"
+echo
+fi
+
+AIRFLOW_ROOT="$(cd ${MY_DIR}; cd ../../..; pwd)"
+
+export AIRFLOW__CORE__DAGS_FOLDER="${AIRFLOW_SOURCES}/tests/dags"
+
+# add test/test_utils to PYTHONPATH (TODO: Do we need it?)
+export PYTHONPATH=${PYTHONPATH:-${AIRFLOW_SOURCES}/tests/test_utils}
+
+export AIRFLOW__CORE__UNIT_TEST_MODE=True
+export HADOOP_DISTRO
+
+echo "###"
+echo
+echo " You can drop into ipdb debugger by adding this line to your code:"
+echo
+echo " import ipdb; ipdb.set_trace()"
+echo
+echo " Then run your tests with 'run-tests --nocapture' "
+echo
+echo "###"
+
+
+# Fix codecov build path
+# TODO: Check this - this should be made travis-independent
+if [[ ! -h /home/travis/build/apache/airflow ]]; then
+ sudo mkdir -p /home/travis/build/apache
+ sudo ln -s ${AIRFLOW_ROOT} /home/travis/build/apache/airflow
+fi
+
+# Fix file permissions
+if [[ -d $HOME/.minikube ]]; then
+sudo chown -R airflow.airflow $HOME/.kube $HOME/.minikube
+fi
+
+# Cleanup the logs when entering the environment
+sudo rm -rf ${AIRFLOW_HOME}/logs/*
+
+if [[ "${ENV}" == "docker" ]]; then
+# Start MiniCluster
+java -cp "/tmp/minicluster-1.1-SNAPSHOT/*" com.ing.minicluster.MiniCluster
>/dev/null 2>&1 &
+
+# Set up ssh keys
+echo 'yes' | ssh-keygen -t rsa -C your_em...@youremail.com -P '' -f
~/.ssh/id_rsa >/dev/null 2>&1
+cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
+ln -s -f ~/.ssh/authorized_keys ~/.ssh/authorized_keys2
+chmod 600 ~/.ssh/*
+
+# SSH Service
+sudo service ssh restart >/dev/null 2>&1
+
+# Setting up kerberos
+
+FQDN=`hostname`
+ADMIN="admin"
+PASS="airflow"
+KRB5_KTNAME=/etc/airflow.keytab
+
+if [[ ${AIRFLOW_CI_VERBOSE} == "true" ]]; then
+echo
+echo "Hosts:"
+echo
+cat /etc/hosts
+echo
+echo "Hostname: ${FQDN}"
+echo
+fi
+
+sudo cp ${MY_DIR}/krb5/krb5.conf /etc/krb5.conf
+
+echo -e "${PASS}\n${PASS}" | \
+sudo kadmin -p ${ADMIN}/admin -w ${PASS} -q "addprinc -randkey
airflow/${FQDN}" >/dev/null 2>&1
+sudo kadmin -p ${ADMIN}/admin -w ${PASS} -q
[GitHub] [airflow] jmcarp commented on issue #5259: [AIRFLOW-4478] Lazily instantiate default `Resources` object
jmcarp commented on issue #5259: [AIRFLOW-4478] Lazily instantiate default `Resources` object URL: https://github.com/apache/airflow/pull/5259#issuecomment-508323724 Ping @ashb when you have time for another look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #5531: [AIRFLOW-4894]Add hook and operator for GCP DLP
codecov-io edited a comment on issue #5531: [AIRFLOW-4894]Add hook and operator for GCP DLP URL: https://github.com/apache/airflow/pull/5531#issuecomment-508307246 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5531?src=pr=h1) Report > Merging [#5531](https://codecov.io/gh/apache/airflow/pull/5531?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/be6a0e76a9d87c74cac0d362ca05964c5ed175a8?src=pr=desc) will **increase** coverage by `0.12%`. > The diff coverage is `97.94%`. [](https://codecov.io/gh/apache/airflow/pull/5531?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5531 +/- ## == + Coverage 79.07% 79.19% +0.12% == Files 489 491 +2 Lines 3074430939 +195 == + Hits2431124503 +192 - Misses 6433 6436 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5531?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/operators/gcp\_run\_dlp\_job.py](https://codecov.io/gh/apache/airflow/pull/5531/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfcnVuX2RscF9qb2IucHk=) | `100% <100%> (ø)` | | | [...low/contrib/hooks/gcp\_data\_loss\_prevention\_hook.py](https://codecov.io/gh/apache/airflow/pull/5531/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9kYXRhX2xvc3NfcHJldmVudGlvbl9ob29rLnB5) | `97.74% <97.74%> (ø)` | | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5531/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.18% <0%> (+0.16%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5531?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5531?src=pr=footer). Last update [be6a0e7...e0a4328](https://codecov.io/gh/apache/airflow/pull/5531?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #5531: [AIRFLOW-4894]Add hook and operator for GCP DLP
codecov-io commented on issue #5531: [AIRFLOW-4894]Add hook and operator for GCP DLP URL: https://github.com/apache/airflow/pull/5531#issuecomment-508307246 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5531?src=pr=h1) Report > Merging [#5531](https://codecov.io/gh/apache/airflow/pull/5531?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/be6a0e76a9d87c74cac0d362ca05964c5ed175a8?src=pr=desc) will **increase** coverage by `0.12%`. > The diff coverage is `97.94%`. [](https://codecov.io/gh/apache/airflow/pull/5531?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5531 +/- ## == + Coverage 79.07% 79.19% +0.12% == Files 489 491 +2 Lines 3074430939 +195 == + Hits2431124503 +192 - Misses 6433 6436 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5531?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/operators/gcp\_run\_dlp\_job.py](https://codecov.io/gh/apache/airflow/pull/5531/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9nY3BfcnVuX2RscF9qb2IucHk=) | `100% <100%> (ø)` | | | [...low/contrib/hooks/gcp\_data\_loss\_prevention\_hook.py](https://codecov.io/gh/apache/airflow/pull/5531/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2djcF9kYXRhX2xvc3NfcHJldmVudGlvbl9ob29rLnB5) | `97.74% <97.74%> (ø)` | | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5531/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.18% <0%> (+0.16%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5531?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5531?src=pr=footer). Last update [be6a0e7...e0a4328](https://codecov.io/gh/apache/airflow/pull/5531?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #5530: [AIRFLOW-1740] Fix xcom creation and update via UI
codecov-io commented on issue #5530: [AIRFLOW-1740] Fix xcom creation and update via UI URL: https://github.com/apache/airflow/pull/5530#issuecomment-50830 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5530?src=pr=h1) Report > Merging [#5530](https://codecov.io/gh/apache/airflow/pull/5530?src=pr=desc) into [v1-10-test](https://codecov.io/gh/apache/airflow/commit/19bb7f7e83d9a8a01e1766b3b0ad99eda8ef2287?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/5530?src=pr=tree) ```diff @@ Coverage Diff @@ ## v1-10-test#5530 +/- ## == + Coverage 79.37% 79.37% +<.01% == Files 250 250 Lines 1831618318 +2 == + Hits1453814540 +2 Misses 3778 3778 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5530?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/5530/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `68.4% <50%> (-0.03%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5530/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.14% <0%> (+0.16%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5530?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5530?src=pr=footer). Last update [19bb7f7...83b8b09](https://codecov.io/gh/apache/airflow/pull/5530?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #5530: [AIRFLOW-1740] Fix xcom creation and update via UI
codecov-io edited a comment on issue #5530: [AIRFLOW-1740] Fix xcom creation and update via UI URL: https://github.com/apache/airflow/pull/5530#issuecomment-50830 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5530?src=pr=h1) Report > Merging [#5530](https://codecov.io/gh/apache/airflow/pull/5530?src=pr=desc) into [v1-10-test](https://codecov.io/gh/apache/airflow/commit/19bb7f7e83d9a8a01e1766b3b0ad99eda8ef2287?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/5530?src=pr=tree) ```diff @@ Coverage Diff @@ ## v1-10-test#5530 +/- ## == + Coverage 79.37% 79.37% +<.01% == Files 250 250 Lines 1831618318 +2 == + Hits1453814540 +2 Misses 3778 3778 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5530?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/5530/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `68.4% <50%> (-0.03%)` | :arrow_down: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5530/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.14% <0%> (+0.16%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5530?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5530?src=pr=footer). Last update [19bb7f7...83b8b09](https://codecov.io/gh/apache/airflow/pull/5530?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] XD-DENG commented on a change in pull request #4291: [AIRFLOW-1488] Add the TriggeredDagRunSensor operator
XD-DENG commented on a change in pull request #4291: [AIRFLOW-1488] Add the
TriggeredDagRunSensor operator
URL: https://github.com/apache/airflow/pull/4291#discussion_r300197351
##
File path: airflow/contrib/sensors/triggered_dagrun_sensor.py
##
@@ -0,0 +1,72 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from airflow.exceptions import AirflowException
+from airflow.utils import db
+from airflow.utils.state import State
+from airflow.utils.decorators import apply_defaults
+from airflow.utils.trigger_rule import TriggerRule
+from airflow.models import DagRun
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+
+
+class TriggeredDagRunSensor(BaseSensorOperator):
+"""
+Waits for triggered DAG run(s) to complete and checks status.
+
+:param trigger_task_id: The id of the task that triggered the dags
+and returns a list of dagrun ids to monitor.
+:type trigger_task_id: str
+:param sensor_rule: criteria for success after dagruns complete.
+Default is ``TriggerRule.ONE_SUCCESS``
+:type sensor_rule: str
+"""
+@apply_defaults
+def __init__(
+self,
+trigger_task_id,
+sensor_rule=None,
+*args, **kwargs):
+super(TriggeredDagRunSensor, self).__init__(*args, **kwargs)
+self.sensor_rule = sensor_rule or TriggerRule.ONE_SUCCESS
+self.trigger_task_id = trigger_task_id
+
+def poke(self, context):
+with db.create_session() as session:
+runcount = 0
+ti = context['ti']
+dagrun_ids = ti.xcom_pull(task_ids=self.trigger_task_id)
+if dagrun_ids:
+ids = dagrun_ids[:2]
+ids = ids + ['...'] if len(dagrun_ids) > 2 else ids
+self.log.info('Poking for %s', ','.join(ids))
+runcount = session.query(DagRun).filter(
+DagRun.run_id.in_(dagrun_ids),
+DagRun.state == State.RUNNING,
+).count()
+else:
+raise AirflowException("No dagrun ids returned by '{}'".format(
+self.trigger_task_id))
+self.log.info('runcount=%s', runcount)
+if runcount == 0:
+successcount = session.query(DagRun).filter(
+DagRun.run_id.in_(dagrun_ids),
+DagRun.state == State.SUCCESS,
+).count()
+if self.sensor_rule == TriggerRule.ONE_SUCCESS:
+if successcount == 0:
+raise AirflowException('No dagruns completed
successfully.')
+else:
+raise AirflowException("sensor rule '{}' is not
supported".format(
+self.sensor_rule))
Review comment:
Thanks. I would like to invite other committers to shed some light on this,
and leave the decision to them, in case I missed/misunderstood any of your
points.
@ashb @mik-laj @kaxil @feng-tao May you guys have a look at this and advise?
Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #5529: Update README
codecov-io commented on issue #5529: Update README URL: https://github.com/apache/airflow/pull/5529#issuecomment-508295917 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5529?src=pr=h1) Report > Merging [#5529](https://codecov.io/gh/apache/airflow/pull/5529?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/be6a0e76a9d87c74cac0d362ca05964c5ed175a8?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/5529?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#5529 +/- ## === Coverage 79.07% 79.07% === Files 489 489 Lines 3074430744 === Hits2431124311 Misses 6433 6433 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5529?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5529?src=pr=footer). Last update [be6a0e7...543dd5f](https://codecov.io/gh/apache/airflow/pull/5529?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4894) Add hook and operator for GCP Data Loss Prevention API
[
https://issues.apache.org/jira/browse/AIRFLOW-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878242#comment-16878242
]
ASF GitHub Bot commented on AIRFLOW-4894:
-
zzlbuaa commented on pull request #5531: [AIRFLOW-4894]Add hook and operator
for GCP DLP
URL: https://github.com/apache/airflow/pull/5531
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-4894) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Add a hook and operator for GCP Data Loss Prevention(DLP).
DLP API allows users to inspect or redact sensitive data in text contents or
GCP storage locations.
The hook includes the following APIs that addresses most use cases in
sensitive data inspection and redaction:
- inspect/deidentify/reidentify for content items
- create/delete/get/list/patch for inspectTemplates and storedInfoTypes
- create/list/get/delete/cancel for dlpJobs
Since the dlp job are long-running and created asynchronously, the operator
creates a dlp job(for storage inspection or risk analysis), keeps polling its
status and waits for it to be done or canceled/deleted.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
See tests/contrib/hooks/test_gcp_data_loss_prevention_hook.py
and tests/contrib/operators/test_gcp_run_dlp_job.py
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add hook and operator for GCP Data Loss Prevention API
> --
>
> Key: AIRFLOW-4894
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4894
> Project: Apache Airflow
> Issue Type: New Feature
> Components: api, gcp, hooks, operators, tests
>Affects Versions: 1.10.3
>Reporter: Zhengliang Zhu
>Assignee: Zhengliang Zhu
>Priority: Minor
>
> Add a hook and operator to manipulate and use Google Cloud Data Loss
> Prevention(DLP) API. DLP API allow users to inspect or redact sensitive data
> in text contents or GCP storage locations.
> The hook includes the following APIs, implemented with Google service
> discovery API:
> * inspect/deidentify/reidentify for content:
> [https://cloud.google.com/dlp/docs/reference/rest/v2/projects.content]
> * create/delete/get/list/patch for inspectTemplates:
> [https://cloud.google.com/dlp/docs/reference/rest/v2/organizations.inspectTemplates],
>
> [https://cloud.google.com/dlp/docs/reference/rest/v2/projects.inspectTemplates]
> * create/delete/get/list/patch for storedInfoTypes:
>
[GitHub] [airflow] zzlbuaa opened a new pull request #5531: [AIRFLOW-4894]Add hook and operator for GCP DLP
zzlbuaa opened a new pull request #5531: [AIRFLOW-4894]Add hook and operator
for GCP DLP
URL: https://github.com/apache/airflow/pull/5531
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-4894) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Add a hook and operator for GCP Data Loss Prevention(DLP).
DLP API allows users to inspect or redact sensitive data in text contents or
GCP storage locations.
The hook includes the following APIs that addresses most use cases in
sensitive data inspection and redaction:
- inspect/deidentify/reidentify for content items
- create/delete/get/list/patch for inspectTemplates and storedInfoTypes
- create/list/get/delete/cancel for dlpJobs
Since the dlp job are long-running and created asynchronously, the operator
creates a dlp job(for storage inspection or risk analysis), keeps polling its
status and waits for it to be done or canceled/deleted.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
See tests/contrib/hooks/test_gcp_data_loss_prevention_hook.py
and tests/contrib/operators/test_gcp_run_dlp_job.py
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-1740) Cannot create/update XCOM via UI in PY3
[
https://issues.apache.org/jira/browse/AIRFLOW-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878237#comment-16878237
]
ASF GitHub Bot commented on AIRFLOW-1740:
-
feluelle commented on pull request #5530: [AIRFLOW-1740] Fix xcom creation and
update via UI
URL: https://github.com/apache/airflow/pull/5530
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-1740
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Cannot create/update XCOM via UI in PY3
> ---
>
> Key: AIRFLOW-1740
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1740
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Affects Versions: 1.9.0
> Environment: PY3
>Reporter: Joy Gao
>Assignee: Felix Uellendall
>Priority: Minor
>
> I cannot create/update XCOM via UI in PY3.
> When attempting to update an existing dag's xcom, the following error is
> received:
> {code:java}
> Failed to update record. (builtins.TypeError) string argument without an
> encoding [SQL: 'UPDATE xcom SET value=%s WHERE xcom.id = %s'] [parameters:
> [{'xcom_id': 165, 'value': "b'bar'"}]]
> {code}
> And for creating a new xcom:
> {code:java}
> Failed to create record. (builtins.TypeError) string argument without an
> encoding [SQL: 'INSERT INTO xcom (`key`, value, timestamp, execution_date,
> task_id, dag_id) VALUES (%s, %s, now(), %s, %s, %s)'] [parameters:
> [{'execution_date': datetime.datetime(2017, 10, 7, 1, 1), 'value': 'bar',
> 'task_id': 'test_task', 'key': 'foo', 'dag_id': 'test_dag'}]]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] feluelle opened a new pull request #5530: [AIRFLOW-1740] Fix xcom creation and update via UI
feluelle opened a new pull request #5530: [AIRFLOW-1740] Fix xcom creation and
update via UI
URL: https://github.com/apache/airflow/pull/5530
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-1740
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-4894) Add hook and operator for GCP Data Loss Prevention API
Zhengliang Zhu created AIRFLOW-4894: --- Summary: Add hook and operator for GCP Data Loss Prevention API Key: AIRFLOW-4894 URL: https://issues.apache.org/jira/browse/AIRFLOW-4894 Project: Apache Airflow Issue Type: New Feature Components: api, gcp, hooks, operators, tests Affects Versions: 1.10.3 Reporter: Zhengliang Zhu Assignee: Zhengliang Zhu Add a hook and operator to manipulate and use Google Cloud Data Loss Prevention(DLP) API. DLP API allow users to inspect or redact sensitive data in text contents or GCP storage locations. The hook includes the following APIs, implemented with Google service discovery API: * inspect/deidentify/reidentify for content: [https://cloud.google.com/dlp/docs/reference/rest/v2/projects.content] * create/delete/get/list/patch for inspectTemplates: [https://cloud.google.com/dlp/docs/reference/rest/v2/organizations.inspectTemplates], [https://cloud.google.com/dlp/docs/reference/rest/v2/projects.inspectTemplates] * create/delete/get/list/patch for storedInfoTypes: [https://cloud.google.com/dlp/docs/reference/rest/v2/organizations.storedInfoTypes], [https://cloud.google.com/dlp/docs/reference/rest/v2/projects.storedInfoTypes] * create/list/get/delete/cancel for dlpJobs: [https://cloud.google.com/dlp/docs/reference/rest/v2/projects.dlpJobs] The operator creates a long-running dlp job (for storage inspection or risk analysis), keeps polling its status and waits for it to be done or canceled/deleted. Apart from unit tests, also tested locally in DAG level(not included in PR). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] smogili1 opened a new pull request #5529: Update README
smogili1 opened a new pull request #5529: Update README
URL: https://github.com/apache/airflow/pull/5529
adding a new company (Cryptalizer.com) that uses airflow for ETL workflows
Make sure you have checked _all_ steps below.
### Jira
- [X ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [X] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Assigned] (AIRFLOW-1740) Cannot create/update XCOM via UI in PY3
[
https://issues.apache.org/jira/browse/AIRFLOW-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall reassigned AIRFLOW-1740:
-
Assignee: Felix Uellendall
> Cannot create/update XCOM via UI in PY3
> ---
>
> Key: AIRFLOW-1740
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1740
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Affects Versions: 1.9.0
> Environment: PY3
>Reporter: Joy Gao
>Assignee: Felix Uellendall
>Priority: Minor
>
> I cannot create/update XCOM via UI in PY3.
> When attempting to update an existing dag's xcom, the following error is
> received:
> {code:java}
> Failed to update record. (builtins.TypeError) string argument without an
> encoding [SQL: 'UPDATE xcom SET value=%s WHERE xcom.id = %s'] [parameters:
> [{'xcom_id': 165, 'value': "b'bar'"}]]
> {code}
> And for creating a new xcom:
> {code:java}
> Failed to create record. (builtins.TypeError) string argument without an
> encoding [SQL: 'INSERT INTO xcom (`key`, value, timestamp, execution_date,
> task_id, dag_id) VALUES (%s, %s, now(), %s, %s, %s)'] [parameters:
> [{'execution_date': datetime.datetime(2017, 10, 7, 1, 1), 'value': 'bar',
> 'task_id': 'test_task', 'key': 'foo', 'dag_id': 'test_dag'}]]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] codecov-io commented on issue #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
codecov-io commented on issue #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC URL: https://github.com/apache/airflow/pull/5525#issuecomment-508283543 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5525?src=pr=h1) Report > Merging [#5525](https://codecov.io/gh/apache/airflow/pull/5525?src=pr=desc) into [v1-10-test](https://codecov.io/gh/apache/airflow/commit/19bb7f7e83d9a8a01e1766b3b0ad99eda8ef2287?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/5525?src=pr=tree) ```diff @@ Coverage Diff @@ ## v1-10-test#5525 +/- ## == + Coverage 79.37% 79.38% +0.01% == Files 250 250 Lines 1831618321 +5 == + Hits1453814544 +6 + Misses 3778 3777 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5525?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/5525/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `68.44% <100%> (+0.01%)` | :arrow_up: | | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5525/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.18% <100%> (+0.21%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5525?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5525?src=pr=footer). Last update [19bb7f7...4d2a3ef](https://codecov.io/gh/apache/airflow/pull/5525?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-4893) Location parameter error in Google Cloud SQL Connection even when set
Andy Shinn created AIRFLOW-4893:
---
Summary: Location parameter error in Google Cloud SQL Connection
even when set
Key: AIRFLOW-4893
URL: https://issues.apache.org/jira/browse/AIRFLOW-4893
Project: Apache Airflow
Issue Type: Bug
Components: gcp
Affects Versions: 1.10.3
Reporter: Andy Shinn
I think there is a bug in Google Cloud SQL Connection at
[https://github.com/apache/airflow/blob/12a2a414662c39a2484721ee67d9b1e61d384edd/airflow/contrib/hooks/gcp_sql_hook.py#L881-L882].
When creating a connection with the Extra field containing JSON with location
set and running a task using it still fails with the error:
{code:java}
The location parameter needs to be set in connection{code}
This particular connection is using Cloud SQL Proxy and doesn't need a
public_ip. The connection does work if i put a fake IP in the Host field to
satisfy the check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4494) Unwanted kwarg shell=True in DaskExecutor
[
https://issues.apache.org/jira/browse/AIRFLOW-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878211#comment-16878211
]
Andy Shinn commented on AIRFLOW-4494:
-
Is there something preventing this from getting merged? I'm hoping to get this
into 1.10.4. The DaskExecutor is broken without this.
> Unwanted kwarg shell=True in DaskExecutor
> -
>
> Key: AIRFLOW-4494
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4494
> Project: Apache Airflow
> Issue Type: Bug
> Components: executors
>Affects Versions: 1.10.3
>Reporter: GuoJiaqi
>Priority: Major
>
> This line:
> [https://github.com/apache/airflow/blob/master/airflow/executors/dask_executor.py#L68]
> If {{Popen}} is with {{shell=True}} and args is a list, the first item of
> list specifies the command string, the other items are the arguments of
> /bin/sh on POSIX.
> So the command {{airflow run ...}} will raise an error from dask-worker:
> {code:java}
> usage: airflow [-h]
> {backfill,list_dag_runs,list_tasks,clear,pause,unpause,trigger_dag,delete_dag,pool,variables,kerberos,render,run,initdb,list_dags,dag_state,task_failed_deps,task_state,serve_logs,test,webserver,resetdb,upgradedb,scheduler,worker,flower,version,connections,create_user,delete_user,list_users,sync_perm,next_execution,rotate_fernet_key}
> ...
> airflow: error: the following arguments are required: subcommand
> distributed.worker - WARNING - Compute Failed
> Function: airflow_run
> args: ()
> kwargs: {}
> Exception: CalledProcessError(2, ['airflow', 'run', 'tutorial', 'print_date',
> '2019-05-08T00:00:00+00:00', '--local', '-sd',
> '/Users//projects/dask_airflow/.venv/lib/python3.6/site-packages/airflow/example_dags/tutorial.py']){code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] ybendana commented on a change in pull request #4291: [AIRFLOW-1488] Add the TriggeredDagRunSensor operator
ybendana commented on a change in pull request #4291: [AIRFLOW-1488] Add the
TriggeredDagRunSensor operator
URL: https://github.com/apache/airflow/pull/4291#discussion_r300171893
##
File path: airflow/contrib/sensors/triggered_dagrun_sensor.py
##
@@ -0,0 +1,72 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from airflow.exceptions import AirflowException
+from airflow.utils import db
+from airflow.utils.state import State
+from airflow.utils.decorators import apply_defaults
+from airflow.utils.trigger_rule import TriggerRule
+from airflow.models import DagRun
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+
+
+class TriggeredDagRunSensor(BaseSensorOperator):
+"""
+Waits for triggered DAG run(s) to complete and checks status.
+
+:param trigger_task_id: The id of the task that triggered the dags
+and returns a list of dagrun ids to monitor.
+:type trigger_task_id: str
+:param sensor_rule: criteria for success after dagruns complete.
+Default is ``TriggerRule.ONE_SUCCESS``
+:type sensor_rule: str
+"""
+@apply_defaults
+def __init__(
+self,
+trigger_task_id,
+sensor_rule=None,
+*args, **kwargs):
+super(TriggeredDagRunSensor, self).__init__(*args, **kwargs)
+self.sensor_rule = sensor_rule or TriggerRule.ONE_SUCCESS
+self.trigger_task_id = trigger_task_id
+
+def poke(self, context):
+with db.create_session() as session:
+runcount = 0
+ti = context['ti']
+dagrun_ids = ti.xcom_pull(task_ids=self.trigger_task_id)
+if dagrun_ids:
+ids = dagrun_ids[:2]
+ids = ids + ['...'] if len(dagrun_ids) > 2 else ids
+self.log.info('Poking for %s', ','.join(ids))
+runcount = session.query(DagRun).filter(
+DagRun.run_id.in_(dagrun_ids),
+DagRun.state == State.RUNNING,
+).count()
+else:
+raise AirflowException("No dagrun ids returned by '{}'".format(
+self.trigger_task_id))
+self.log.info('runcount=%s', runcount)
+if runcount == 0:
+successcount = session.query(DagRun).filter(
+DagRun.run_id.in_(dagrun_ids),
+DagRun.state == State.SUCCESS,
+).count()
+if self.sensor_rule == TriggerRule.ONE_SUCCESS:
+if successcount == 0:
+raise AirflowException('No dagruns completed
successfully.')
+else:
+raise AirflowException("sensor rule '{}' is not
supported".format(
+self.sensor_rule))
Review comment:
@XD-DENG I don't know what else I can do to convince of the utility of this
sensor. All I can say is that it is useful for us and may be for others. If
you would like me to change anything else, let me know.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ybendana commented on a change in pull request #4291: [AIRFLOW-1488] Add the TriggeredDagRunSensor operator
ybendana commented on a change in pull request #4291: [AIRFLOW-1488] Add the
TriggeredDagRunSensor operator
URL: https://github.com/apache/airflow/pull/4291#discussion_r300171301
##
File path: airflow/contrib/sensors/triggered_dagrun_sensor.py
##
@@ -0,0 +1,72 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from airflow.exceptions import AirflowException
+from airflow.utils import db
+from airflow.utils.state import State
+from airflow.utils.decorators import apply_defaults
+from airflow.utils.trigger_rule import TriggerRule
+from airflow.models import DagRun
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+
+
+class TriggeredDagRunSensor(BaseSensorOperator):
+"""
+Waits for triggered DAG run(s) to complete and checks status.
+
+:param trigger_task_id: The id of the task that triggered the dags
+and returns a list of dagrun ids to monitor.
+:type trigger_task_id: str
+:param sensor_rule: criteria for success after dagruns complete.
+Default is ``TriggerRule.ONE_SUCCESS``
+:type sensor_rule: str
+"""
+@apply_defaults
+def __init__(
+self,
+trigger_task_id,
+sensor_rule=None,
+*args, **kwargs):
+super(TriggeredDagRunSensor, self).__init__(*args, **kwargs)
+self.sensor_rule = sensor_rule or TriggerRule.ONE_SUCCESS
+self.trigger_task_id = trigger_task_id
+
+def poke(self, context):
+with db.create_session() as session:
+runcount = 0
+ti = context['ti']
+dagrun_ids = ti.xcom_pull(task_ids=self.trigger_task_id)
+if dagrun_ids:
+ids = dagrun_ids[:2]
+ids = ids + ['...'] if len(dagrun_ids) > 2 else ids
+self.log.info('Poking for %s', ','.join(ids))
+runcount = session.query(DagRun).filter(
+DagRun.run_id.in_(dagrun_ids),
+DagRun.state == State.RUNNING,
+).count()
+else:
+raise AirflowException("No dagrun ids returned by '{}'".format(
+self.trigger_task_id))
+self.log.info('runcount=%s', runcount)
+if runcount == 0:
+successcount = session.query(DagRun).filter(
+DagRun.run_id.in_(dagrun_ids),
+DagRun.state == State.SUCCESS,
+).count()
+if self.sensor_rule == TriggerRule.ONE_SUCCESS:
+if successcount == 0:
+raise AirflowException('No dagruns completed
successfully.')
+else:
+raise AirflowException("sensor rule '{}' is not
supported".format(
+self.sensor_rule))
Review comment:
@XD-DENG @yustoris: I updated the code to handle the ALL_SUCCESS rule as
well. It was my intention to have the sensor handle more than the ONE_SUCCESS
rule (which is what we use) in the future, but I had forgotten to update it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] Doncode commented on a change in pull request #5503: [AIRFLOW-3360] Make the DAGs search respect other querystring parameters
Doncode commented on a change in pull request #5503: [AIRFLOW-3360] Make the
DAGs search respect other querystring parameters
URL: https://github.com/apache/airflow/pull/5503#discussion_r300149383
##
File path: airflow/www/templates/airflow/dags.html
##
@@ -218,8 +218,9 @@ DAGs
$('#dag_query').on('keypress', function (e) {
// check for key press on ENTER (key code 13) to trigger the search
if (e.which === ENTER_KEY_CODE) {
- search_query = $('#dag_query').val();
- window.location = DAGS_INDEX + "?search="+ encodeURI(search_query);
+ var query = new URLSearchParams(window.location.search);
Review comment:
@ashb please review
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] galak75 commented on issue #4691: [AIRFLOW-1814] : templatize PythonOperator op_args and op_kwargs fields
galak75 commented on issue #4691: [AIRFLOW-1814] : templatize PythonOperator op_args and op_kwargs fields URL: https://github.com/apache/airflow/pull/4691#issuecomment-508245449 > There is a report that this PR is breaking change: > https://issues.apache.org/jira/browse/AIRFLOW-4451 > @galak75 can you address this? I will try to This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4451) [AIRFLOW-1814] converts namedtuples args in PythonOperators to lists
[ https://issues.apache.org/jira/browse/AIRFLOW-4451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878149#comment-16878149 ] Galak commented on AIRFLOW-4451: [~rossmechanic] : can you please provide a short source code example? > [AIRFLOW-1814] converts namedtuples args in PythonOperators to lists > > > Key: AIRFLOW-4451 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4451 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.10.3 >Reporter: Ross Mechanic >Priority: Major > > Upgrading to Airflow 1.10.3 from Airflow 1.10.2 removed support for passing > in `namedtuple`s as `op_kwargs` or `op_args` to `PythonOperator`. The > specific PR that made the breaking change is > [https://github.com/apache/airflow/pull/4691] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] ashb commented on issue #5527: [AIRFLOW-4892] Fix connection creation via UIs
ashb commented on issue #5527: [AIRFLOW-4892] Fix connection creation via UIs URL: https://github.com/apache/airflow/pull/5527#issuecomment-508244973 Tests are falling on py2 cos of `super()` - I'll fix that This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Quimstr opened a new pull request #5528: Add connection protocol mapping for https:// connections
Quimstr opened a new pull request #5528: Add connection protocol mapping for
https:// connections
URL: https://github.com/apache/airflow/pull/5528
Make sure you have checked _all_ steps below.
### Jira
- [ ] There is no Jira story addressing issue I've encountered.
### Description
- [ ] When 'creating' connections by environmental variables, right now it
is impossible to add connection starting with https:// , which is common case
for connections to chat services (google, slack). My fix address this issue.
### Tests
- [ ] There are no tests, it's only expanding functionality by adding
additional mapping.
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] I guess no documentation is needed for this change.
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] andyshinn edited a comment on issue #5273: [AIRFLOW-4494] Remove `shell=True` in DaskExecutor
andyshinn edited a comment on issue #5273: [AIRFLOW-4494] Remove `shell=True` in DaskExecutor URL: https://github.com/apache/airflow/pull/5273#issuecomment-508227498 Is it possible to get this merged for 1.10.4? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] andyshinn commented on issue #5273: [AIRFLOW-4494] Remove `shell=True` in DaskExecutor
andyshinn commented on issue #5273: [AIRFLOW-4494] Remove `shell=True` in DaskExecutor URL: https://github.com/apache/airflow/pull/5273#issuecomment-508227498 Is it possible to get this merged for 10.0.4? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
feluelle commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC URL: https://github.com/apache/airflow/pull/5525#discussion_r300097217 ## File path: airflow/models/taskinstance.py ## @@ -33,6 +33,7 @@ import pendulum import dill +from flask import url_for Review comment: But now I did `optimize imports` :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
feluelle commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC URL: https://github.com/apache/airflow/pull/5525#discussion_r300097049 ## File path: airflow/models/taskinstance.py ## @@ -33,6 +33,7 @@ import pendulum import dill +from flask import url_for Review comment: God damn it. Normally I would have `option + command + o`'d but since this would've changed so many lines I didn't do that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #5526: [AIRFLOW-XXX] Add City of Toronto to official users list
ashb commented on issue #5526: [AIRFLOW-XXX] Add City of Toronto to official users list URL: https://github.com/apache/airflow/pull/5526#issuecomment-508198109 Welcome This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb merged pull request #5526: [AIRFLOW-XXX] Add City of Toronto to official users list
ashb merged pull request #5526: [AIRFLOW-XXX] Add City of Toronto to official users list URL: https://github.com/apache/airflow/pull/5526 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4862) Allow directly using IP address as hostname in airflow.utils.net.get_hostname()
[
https://issues.apache.org/jira/browse/AIRFLOW-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878039#comment-16878039
]
ASF subversion and git services commented on AIRFLOW-4862:
--
Commit 98f2019ac209fe22f8fdb5c829f360d1877be9cf in airflow's branch
refs/heads/v1-10-stable from Xiaodong
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=98f2019 ]
[AIRFLOW-4862] Fix bug for earlier change to allow using IP as hostname (#5513)
(cherry picked from commit 0fe8cbe31484f7ca390725b684e620355ac222df)
> Allow directly using IP address as hostname in
> airflow.utils.net.get_hostname()
> ---
>
> Key: AIRFLOW-4862
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4862
> Project: Apache Airflow
> Issue Type: Improvement
> Components: utils
>Affects Versions: 1.10.3
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
> Fix For: 1.10.4
>
>
> In airflow.utils.net.get_hostname(), the default function used to get
> hostname for nodes (like worker) is *socket.getfqdn()*, which will return
> fully qualified domain name.
> In some cases, we do need to ensure that hostnames can resolved so that nodes
> can talk to each other. One example is: if I use S3 for remote logging, then
> the log will only be pushed to S3 after the job is finished (either success
> or failure); when the job is still running, webserver will first check if the
> log is available in its own volume, if not, webserver will fetch log from
> worker.
> If workers' hostnames are something like "airflow-worker-53-4bp8v" (e.g.,
> when running on OpenShift or K8S), it's possible that the hostname can't be
> resolved, then we will observe errors like below
> {code:java}
> *** Log file does not exist:
> /opt/app-root/airflow/logs/example_python_operator/sleep_for_3/2019-06-18T08:14:15.313472+00:00/2.log
> *** Fetching from:
> http://airflow-worker-57-n69vb:8793/log/example_python_operator/sleep_for_3/2019-06-18T08:14:15.313472+00:00/2.log
> *** Failed to fetch log file from worker.
> HTTPConnectionPool(host='airflow-worker-57-n69vb', port=8793): Max retries
> exceeded with url:
> /log/example_python_operator/sleep_for_3/2019-06-18T08:14:15.313472+00:00/2.log
> (Caused by NewConnectionError(' 0x7fec3266a6d8>: Failed to establish a new connection: [Errno -2] Name or
> service not known',))
> {code}
>
> This may be addressed by properly setting service discovery, but the users
> may not always have the privilege to do that (like myself in my organization).
> Another solution is to change "hostname_callable" in Airflow's configuration
> ([core] section). But it will only work if you have a function which can
> return a resolvable hostname and it can't take any argument (due to the
> existing implementation
> [https://github.com/apache/airflow/blob/dd08ae3469a50a145f9ae7f819ed1840fe2a5bd6/airflow/utils/net.py#L41-L45).]
>
> The change I would like to propose is: allow users to use IP address directly
> as hostname.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4892) Creation of Connections via UI fails in 1.10.4rc2
[
https://issues.apache.org/jira/browse/AIRFLOW-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878026#comment-16878026
]
ASF GitHub Bot commented on AIRFLOW-4892:
-
ashb commented on pull request #5527: [AIRFLOW-4892] Fix connection creation
via UIs
URL: https://github.com/apache/airflow/pull/5527
## Jira
- [x] https://issues.apache.org/jira/browse/AIRFLOW-4892
### Description
- [x] The form validation for classic UI was failing with an error on the
(most often hidden) Num Retries field.
RBAC was failing because the form wasn't rendering due to a typo in a
cherry-pick/backport to www_rbac.
Both of these apply to the release branch only as they were the result of
me back-porting PRs from master that had in-sufficient test. I have a separate
PR to add more tests to the connection form on master.
### Tests
- [x] Added some tests.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Creation of Connections via UI fails in 1.10.4rc2
> -
>
> Key: AIRFLOW-4892
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4892
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Affects Versions: 1.10.4
>Reporter: Felix Uellendall
>Assignee: Ash Berlin-Taylor
>Priority: Major
> Fix For: 1.10.4
>
>
> Via classic UI: When trying to create/save a new connection. The form only
> reloads without saving the connection.
> With RBAC: Error when rendering the connections form:
> {noformat}
> jinja2.exceptions.UndefinedError: 'airflow.www_rbac.forms.ConnectionForm
> object' has no attribute 'extra__grpc__credentials_pem_file'
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] ashb opened a new pull request #5527: [AIRFLOW-4892] Fix connection creation via UIs
ashb opened a new pull request #5527: [AIRFLOW-4892] Fix connection creation
via UIs
URL: https://github.com/apache/airflow/pull/5527
## Jira
- [x] https://issues.apache.org/jira/browse/AIRFLOW-4892
### Description
- [x] The form validation for classic UI was failing with an error on the
(most often hidden) Num Retries field.
RBAC was failing because the form wasn't rendering due to a typo in a
cherry-pick/backport to www_rbac.
Both of these apply to the release branch only as they were the result of
me back-porting PRs from master that had in-sufficient test. I have a separate
PR to add more tests to the connection form on master.
### Tests
- [x] Added some tests.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #5522: [AIRFLOW-4882] Make GCP tests and examples Pylint compatible
mik-laj commented on a change in pull request #5522: [AIRFLOW-4882] Make GCP tests and examples Pylint compatible URL: https://github.com/apache/airflow/pull/5522#discussion_r300083518 ## File path: airflow/contrib/example_dags/example_gcp_sql.py ## @@ -186,6 +186,7 @@ ) as dag: def next_dep(task, prev): Review comment: This function is used in this file? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
ashb commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC URL: https://github.com/apache/airflow/pull/5525#discussion_r300081413 ## File path: airflow/models/taskinstance.py ## @@ -33,6 +33,7 @@ import pendulum import dill +from flask import url_for Review comment: ```suggestion ``` Don't need this anymore :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
ashb commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link
in TaskInstance's View for Non-RBAC
URL: https://github.com/apache/airflow/pull/5525#discussion_r300081249
##
File path: airflow/models/taskinstance.py
##
@@ -376,12 +377,13 @@ def log_filepath(self):
def log_url(self):
iso = quote(self.execution_date.isoformat())
base_url = configuration.conf.get('webserver', 'BASE_URL')
-return base_url + (
-"/log?"
-"execution_date={iso}"
-"_id={task_id}"
-"_id={dag_id}"
-).format(iso=iso, task_id=self.task_id, dag_id=self.dag_id)
+
+relative_url =
"/log?execution_date={iso}_id={task_id}_id={dag_id}".format(
+iso=iso, task_id=self.task_id, dag_id=self.dag_id)
Review comment:
Can you url-quote these three please?
```suggestion
iso=quote_plus(iso), task_id=quote_plus(self.task_id),
dag_id=quote_plus(self.dag_id))
```
And
```
from six.moves.urllib.parse import quote_plus
```
at the top
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-4892) Creation of Connections via UI fails in 1.10.4rc2
[
https://issues.apache.org/jira/browse/AIRFLOW-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-4892:
---
Description:
Via classic UI: When trying to create/save a new connection. The form only
reloads without saving the connection.
With RBAC: Error when rendering the connections form:
{noformat}
jinja2.exceptions.UndefinedError: 'airflow.www_rbac.forms.ConnectionForm
object' has no attribute 'extra__grpc__credentials_pem_file'
{noformat}
was:When trying to create/save a new connection. The form only reloads
without saving the connection.
Summary: Creation of Connections via UI fails in 1.10.4rc2 (was:
Creation of Connections via non-RBAC UI fails in 1.10.4rc2)
> Creation of Connections via UI fails in 1.10.4rc2
> -
>
> Key: AIRFLOW-4892
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4892
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Affects Versions: 1.10.4
>Reporter: Felix Uellendall
>Assignee: Ash Berlin-Taylor
>Priority: Major
> Fix For: 1.10.4
>
>
> Via classic UI: When trying to create/save a new connection. The form only
> reloads without saving the connection.
> With RBAC: Error when rendering the connections form:
> {noformat}
> jinja2.exceptions.UndefinedError: 'airflow.www_rbac.forms.ConnectionForm
> object' has no attribute 'extra__grpc__credentials_pem_file'
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4888) Add migration system for adding RBAC permissions to existing roles
[
https://issues.apache.org/jira/browse/AIRFLOW-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878006#comment-16878006
]
Ash Berlin-Taylor commented on AIRFLOW-4888:
[~XD-DENG] Admin users are able top edit the permissions, but needing anyone
who wants to use Op/User roles adding some permissions seems wrong - it's a
"missing" permission in the default install so I feel we should fix it for
everyone.
(Plus we don't allow _any_ users to be admins in our cluster. We're strange
like that)
> Add migration system for adding RBAC permissions to existing roles
> --
>
> Key: AIRFLOW-4888
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4888
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: 2.0.0
>Reporter: Ash Berlin-Taylor
>Priority: Major
> Labels: permissions
>
> In our clusters we don't allow any users to be Admin, so we use the Op, User
> and Viewer roles. It turns out that these roles are missing the
> {{can_dagrun_success}} and {{can_dagrun_failure}} permissions.
> Fixing this for new installs is easy, but due to AIRFLOW-3271
> (https://github.com/apache/airflow/pull/4118) we won't alter the roles if
> they already exist, so having some mechanism for adding permissions to roles
> via migrations might be useful.
> As a palyground I started working on
> https://gist.github.com/ashb/f43741740fb0eae59948d52634cda575 - I'm not sure
> if this is too complex or not. (It's also not a complete solution yet)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4890) Fix Log link in TaskInstance's View for Non-RBAC
[ https://issues.apache.org/jira/browse/AIRFLOW-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878002#comment-16878002 ] Felix Uellendall commented on AIRFLOW-4890: --- [~ash] actually I think this ticket does not resolve the issues I linked, does it? > Fix Log link in TaskInstance's View for Non-RBAC > > > Key: AIRFLOW-4890 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4890 > Project: Apache Airflow > Issue Type: Bug > Components: logging, ui >Affects Versions: 1.8.0, 1.10.3 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4890) Fix Log link in TaskInstance's View for Non-RBAC
[ https://issues.apache.org/jira/browse/AIRFLOW-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-4890: -- Component/s: ui > Fix Log link in TaskInstance's View for Non-RBAC > > > Key: AIRFLOW-4890 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4890 > Project: Apache Airflow > Issue Type: Bug > Components: logging, ui >Affects Versions: 1.8.0, 1.10.3 >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] feluelle commented on issue #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
feluelle commented on issue #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC URL: https://github.com/apache/airflow/pull/5525#issuecomment-508178735 Updated. PTAL @ashb This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4888) Add migration system for adding RBAC permissions to existing roles
[
https://issues.apache.org/jira/browse/AIRFLOW-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877986#comment-16877986
]
Ash Berlin-Taylor commented on AIRFLOW-4888:
Yeah, I'm thinking about the upgrade path from ~1.10.0 to 1.10.4 and 2.0.0
onwards.
I haven't worked out the exact details of what the migration would do (I think
maybe just invoke FAB SecurityManager code directly. Not sure, but I think that
is less fragile than creating migrations for the FAB security models as they
aren't "ours"
> Add migration system for adding RBAC permissions to existing roles
> --
>
> Key: AIRFLOW-4888
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4888
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: 2.0.0
>Reporter: Ash Berlin-Taylor
>Priority: Major
> Labels: permissions
>
> In our clusters we don't allow any users to be Admin, so we use the Op, User
> and Viewer roles. It turns out that these roles are missing the
> {{can_dagrun_success}} and {{can_dagrun_failure}} permissions.
> Fixing this for new installs is easy, but due to AIRFLOW-3271
> (https://github.com/apache/airflow/pull/4118) we won't alter the roles if
> they already exist, so having some mechanism for adding permissions to roles
> via migrations might be useful.
> As a palyground I started working on
> https://gist.github.com/ashb/f43741740fb0eae59948d52634cda575 - I'm not sure
> if this is too complex or not. (It's also not a complete solution yet)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4888) Add migration system for adding RBAC permissions to existing roles
[
https://issues.apache.org/jira/browse/AIRFLOW-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877984#comment-16877984
]
Tao Feng commented on AIRFLOW-4888:
---
[~ash] I vaguely remembered the issue was due to the fact that I changed the
permission for the default roles?I assume if we are going to make a migration
script, it is mostly for migration from old 1.10 setup to 1.10.2/1.10.3?
The only concern is that RBAC model has many different tables. I am not sure
how easily to cover it in a migration script. And should we move the model
files inside airflow as well? If you feel it is not complicated, I am +1 for
the migration script.
> Add migration system for adding RBAC permissions to existing roles
> --
>
> Key: AIRFLOW-4888
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4888
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: 2.0.0
>Reporter: Ash Berlin-Taylor
>Priority: Major
> Labels: permissions
>
> In our clusters we don't allow any users to be Admin, so we use the Op, User
> and Viewer roles. It turns out that these roles are missing the
> {{can_dagrun_success}} and {{can_dagrun_failure}} permissions.
> Fixing this for new installs is easy, but due to AIRFLOW-3271
> (https://github.com/apache/airflow/pull/4118) we won't alter the roles if
> they already exist, so having some mechanism for adding permissions to roles
> via migrations might be useful.
> As a palyground I started working on
> https://gist.github.com/ashb/f43741740fb0eae59948d52634cda575 - I'm not sure
> if this is too complex or not. (It's also not a complete solution yet)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (AIRFLOW-4892) Creation of Connections via non-RBAC UI fails in 1.10.4rc2
[ https://issues.apache.org/jira/browse/AIRFLOW-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-4892 started by Ash Berlin-Taylor. -- > Creation of Connections via non-RBAC UI fails in 1.10.4rc2 > -- > > Key: AIRFLOW-4892 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4892 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.4 >Reporter: Felix Uellendall >Assignee: Ash Berlin-Taylor >Priority: Major > Fix For: 1.10.4 > > > When trying to create/save a new connection. The form only reloads without > saving the connection. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4862) Allow directly using IP address as hostname in airflow.utils.net.get_hostname()
[
https://issues.apache.org/jira/browse/AIRFLOW-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-4862:
---
Fix Version/s: (was: 1.10.5)
1.10.4
> Allow directly using IP address as hostname in
> airflow.utils.net.get_hostname()
> ---
>
> Key: AIRFLOW-4862
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4862
> Project: Apache Airflow
> Issue Type: Improvement
> Components: utils
>Affects Versions: 1.10.3
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
> Fix For: 1.10.4
>
>
> In airflow.utils.net.get_hostname(), the default function used to get
> hostname for nodes (like worker) is *socket.getfqdn()*, which will return
> fully qualified domain name.
> In some cases, we do need to ensure that hostnames can resolved so that nodes
> can talk to each other. One example is: if I use S3 for remote logging, then
> the log will only be pushed to S3 after the job is finished (either success
> or failure); when the job is still running, webserver will first check if the
> log is available in its own volume, if not, webserver will fetch log from
> worker.
> If workers' hostnames are something like "airflow-worker-53-4bp8v" (e.g.,
> when running on OpenShift or K8S), it's possible that the hostname can't be
> resolved, then we will observe errors like below
> {code:java}
> *** Log file does not exist:
> /opt/app-root/airflow/logs/example_python_operator/sleep_for_3/2019-06-18T08:14:15.313472+00:00/2.log
> *** Fetching from:
> http://airflow-worker-57-n69vb:8793/log/example_python_operator/sleep_for_3/2019-06-18T08:14:15.313472+00:00/2.log
> *** Failed to fetch log file from worker.
> HTTPConnectionPool(host='airflow-worker-57-n69vb', port=8793): Max retries
> exceeded with url:
> /log/example_python_operator/sleep_for_3/2019-06-18T08:14:15.313472+00:00/2.log
> (Caused by NewConnectionError(' 0x7fec3266a6d8>: Failed to establish a new connection: [Errno -2] Name or
> service not known',))
> {code}
>
> This may be addressed by properly setting service discovery, but the users
> may not always have the privilege to do that (like myself in my organization).
> Another solution is to change "hostname_callable" in Airflow's configuration
> ([core] section). But it will only work if you have a function which can
> return a resolvable hostname and it can't take any argument (due to the
> existing implementation
> [https://github.com/apache/airflow/blob/dd08ae3469a50a145f9ae7f819ed1840fe2a5bd6/airflow/utils/net.py#L41-L45).]
>
> The change I would like to propose is: allow users to use IP address directly
> as hostname.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] ashb commented on issue #5501: [AIRFLOW-4862] Allow directly using IP address as hostname
ashb commented on issue #5501: [AIRFLOW-4862] Allow directly using IP address as hostname URL: https://github.com/apache/airflow/pull/5501#issuecomment-508169988 Fixed PR pulled in for RC3. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4833) Jinja templating removes newlines
[
https://issues.apache.org/jira/browse/AIRFLOW-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877967#comment-16877967
]
Galak commented on AIRFLOW-4833:
[~thenumenorean]: I agree with you, {{keep_trailing_newline}} should default to
{{True}}; but it defaults to {{False}} now, and can't be changed for backward
compatibility. See [https://github.com/pallets/jinja/issues/848] and
[https://github.com/pallets/jinja/issues/949]
I'll try to submit a PR for this in the coming weeks
> Jinja templating removes newlines
> -
>
> Key: AIRFLOW-4833
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4833
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
>Affects Versions: 1.10.2
>Reporter: Francesco Macagno
>Priority: Minor
>
> When using an operator that has Jinja templating enabled for a field, if the
> field value ends with a newline then the newline is removed, regardless of
> whether there was a template in the string.
>
> This came up when attempting to send data to Prometheus pushgateway using the
> SimpleHttpOperator. Pushgateway requires a newline at the end of every entry,
> so the removal of the newline at the end of the data parameter causes the
> request to fail in a way that is difficult to debug.
>
> This can be gotten around by including a space after the newline character,
> though this is not a great solution. The space is ignored by pushgateway.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4892) Creation of Connections via non-RBAC UI fails in 1.10.4rc2
[ https://issues.apache.org/jira/browse/AIRFLOW-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-4892: -- Description: When trying to create/save a new connection. The form only reloads without saving the connection. > Creation of Connections via non-RBAC UI fails in 1.10.4rc2 > -- > > Key: AIRFLOW-4892 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4892 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.4 >Reporter: Felix Uellendall >Assignee: Ash Berlin-Taylor >Priority: Major > Fix For: 1.10.4 > > > When trying to create/save a new connection. The form only reloads without > saving the connection. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4888) Add migration system for adding RBAC permissions to existing roles
[
https://issues.apache.org/jira/browse/AIRFLOW-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877956#comment-16877956
]
Xiaodong DENG commented on AIRFLOW-4888:
[~ash] not sure if I get your point completely. But
https://issues.apache.org/jira/browse/AIRFLOW-3271 is fixed by
[https://github.com/apache/airflow/pull/4118] and released in 1.10.2 (and RBAC
is only generally available from 1.10.0 if I'm not wrong), meaning from 1.10.2,
users are able to edit permissions in UI and the changes would persist. Why
should we provide separate migration tool? Or is there more specific scenario?
> Add migration system for adding RBAC permissions to existing roles
> --
>
> Key: AIRFLOW-4888
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4888
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: 2.0.0
>Reporter: Ash Berlin-Taylor
>Priority: Major
> Labels: permissions
>
> In our clusters we don't allow any users to be Admin, so we use the Op, User
> and Viewer roles. It turns out that these roles are missing the
> {{can_dagrun_success}} and {{can_dagrun_failure}} permissions.
> Fixing this for new installs is easy, but due to AIRFLOW-3271
> (https://github.com/apache/airflow/pull/4118) we won't alter the roles if
> they already exist, so having some mechanism for adding permissions to roles
> via migrations might be useful.
> As a palyground I started working on
> https://gist.github.com/ashb/f43741740fb0eae59948d52634cda575 - I'm not sure
> if this is too complex or not. (It's also not a complete solution yet)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4892) Creation of Connections via non-RBAC UI fails in 1.10.4rc2
[ https://issues.apache.org/jira/browse/AIRFLOW-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-4892: -- Summary: Creation of Connections via non-RBAC UI fails in 1.10.4rc2 (was: Creation of Connections via the non-RBAC UI fails in 1.10.4rc2) > Creation of Connections via non-RBAC UI fails in 1.10.4rc2 > -- > > Key: AIRFLOW-4892 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4892 > Project: Apache Airflow > Issue Type: Bug > Components: ui >Affects Versions: 1.10.4 >Reporter: Felix Uellendall >Assignee: Ash Berlin-Taylor >Priority: Major > Fix For: 1.10.4 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4892) Creation of Connections via the non-RBAC UI fails in 1.10.4rc2
Felix Uellendall created AIRFLOW-4892: - Summary: Creation of Connections via the non-RBAC UI fails in 1.10.4rc2 Key: AIRFLOW-4892 URL: https://issues.apache.org/jira/browse/AIRFLOW-4892 Project: Apache Airflow Issue Type: Bug Components: ui Affects Versions: 1.10.4 Reporter: Felix Uellendall Assignee: Ash Berlin-Taylor Fix For: 1.10.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] exploy commented on issue #5508: [AIRFLOW-4876] Making tests in CoreTest rerunnable
exploy commented on issue #5508: [AIRFLOW-4876] Making tests in CoreTest rerunnable URL: https://github.com/apache/airflow/pull/5508#issuecomment-508161171 > Only comment is that the change from `models.DagBag` to `DagBag` etc make the diff messier than needed. Don't mind it though. Wanted to organize the imports so made this change. It may be tricky but I could extract it to separate commit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] exploy commented on issue #5508: [AIRFLOW-4876] Making tests in CoreTest rerunnable
exploy commented on issue #5508: [AIRFLOW-4876] Making tests in CoreTest rerunnable URL: https://github.com/apache/airflow/pull/5508#issuecomment-508160316 > Looks good to me @ashb maybe you have other comments? > > One NIT: I think the linter improvements make TestCore pylint-able - so maybe you can remove it from pylint_todo.txt already ? The change improves Pylint a bit but does not solve all errors This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
feluelle commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log
link in TaskInstance's View for Non-RBAC
URL: https://github.com/apache/airflow/pull/5525#discussion_r300043017
##
File path: airflow/models/taskinstance.py
##
@@ -374,14 +375,12 @@ def log_filepath(self):
@property
def log_url(self):
-iso = quote(self.execution_date.isoformat())
-base_url = configuration.conf.get('webserver', 'BASE_URL')
-return base_url + (
-"/log?"
-"execution_date={iso}"
-"_id={task_id}"
-"_id={dag_id}"
-).format(iso=iso, task_id=self.task_id, dag_id=self.dag_id)
+url = url_for(
Review comment:
So like that:
```python
@property
def log_url(self):
iso = quote(self.execution_date.isoformat())
base_url = configuration.conf.get('webserver', 'BASE_URL')
relative_url =
"/log?execution_date={iso}_id={task_id}_id={dag_id}".format(
iso=iso, task_id=self.task_id, dag_id=self.dag_id)
if configuration.conf.get('webserver', 'rbac'):
return base_url + relative_url
return base_url + '/admin/airflow' + relative_url
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
feluelle commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log
link in TaskInstance's View for Non-RBAC
URL: https://github.com/apache/airflow/pull/5525#discussion_r300037639
##
File path: airflow/models/taskinstance.py
##
@@ -374,14 +375,12 @@ def log_filepath(self):
@property
def log_url(self):
-iso = quote(self.execution_date.isoformat())
-base_url = configuration.conf.get('webserver', 'BASE_URL')
-return base_url + (
-"/log?"
-"execution_date={iso}"
-"_id={task_id}"
-"_id={dag_id}"
-).format(iso=iso, task_id=self.task_id, dag_id=self.dag_id)
+url = url_for(
Review comment:
> Also actually making the args to work for request_context() may be hard
(not to mention messy in a model.)
Agree. This would be messy. I don't like that.
So we can only let it be like it is and differentiate depending on the value
in the airflow config `rbac=True/False` to build the url either with
`airflow/admin` or not?!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
ashb commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link
in TaskInstance's View for Non-RBAC
URL: https://github.com/apache/airflow/pull/5525#discussion_r300035785
##
File path: airflow/models/taskinstance.py
##
@@ -374,14 +375,12 @@ def log_filepath(self):
@property
def log_url(self):
-iso = quote(self.execution_date.isoformat())
-base_url = configuration.conf.get('webserver', 'BASE_URL')
-return base_url + (
-"/log?"
-"execution_date={iso}"
-"_id={task_id}"
-"_id={dag_id}"
-).format(iso=iso, task_id=self.task_id, dag_id=self.dag_id)
+url = url_for(
Review comment:
Yeah - that would fix the column in task instances view at least
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
feluelle commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log
link in TaskInstance's View for Non-RBAC
URL: https://github.com/apache/airflow/pull/5525#discussion_r300034573
##
File path: airflow/models/taskinstance.py
##
@@ -374,14 +375,12 @@ def log_filepath(self):
@property
def log_url(self):
-iso = quote(self.execution_date.isoformat())
-base_url = configuration.conf.get('webserver', 'BASE_URL')
-return base_url + (
-"/log?"
-"execution_date={iso}"
-"_id={task_id}"
-"_id={dag_id}"
-).format(iso=iso, task_id=self.task_id, dag_id=self.dag_id)
+url = url_for(
Review comment:
> Oh also - this url is going to be wrong for RBAC too :( (Airflow.log is
what we need for RBAC)
Yes, I noticed that, too.
So we probably should not try to access the request context in the task
instance. But for the `log_url_formatter` it should work and since we have a
rbac view and a non-rbac view we can use `Airflow.log` and `airflow.log`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
feluelle commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log
link in TaskInstance's View for Non-RBAC
URL: https://github.com/apache/airflow/pull/5525#discussion_r300030199
##
File path: airflow/models/taskinstance.py
##
@@ -374,14 +375,12 @@ def log_filepath(self):
@property
def log_url(self):
-iso = quote(self.execution_date.isoformat())
-base_url = configuration.conf.get('webserver', 'BASE_URL')
-return base_url + (
-"/log?"
-"execution_date={iso}"
-"_id={task_id}"
-"_id={dag_id}"
-).format(iso=iso, task_id=self.task_id, dag_id=self.dag_id)
+url = url_for(
Review comment:
`RuntimeError: Attempted to generate a URL without the application context
being pushed. This has to be executed when application context is available.`
Thank you. :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
ashb commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link
in TaskInstance's View for Non-RBAC
URL: https://github.com/apache/airflow/pull/5525#discussion_r300026819
##
File path: airflow/models/taskinstance.py
##
@@ -374,14 +375,12 @@ def log_filepath(self):
@property
def log_url(self):
-iso = quote(self.execution_date.isoformat())
-base_url = configuration.conf.get('webserver', 'BASE_URL')
-return base_url + (
-"/log?"
-"execution_date={iso}"
-"_id={task_id}"
-"_id={dag_id}"
-).format(iso=iso, task_id=self.task_id, dag_id=self.dag_id)
+url = url_for(
Review comment:
I'm not sure I like it, but:
```python
if not flask._request_ctx_stack:
with request_context(... MESSY_ARGS ... ):
return url_for(...)
else:
return url_for(...)
```
Also actually making the args to work for `request_context()` may be hard
(not to mention messy in a model.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
ashb commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link
in TaskInstance's View for Non-RBAC
URL: https://github.com/apache/airflow/pull/5525#discussion_r300024073
##
File path: airflow/models/taskinstance.py
##
@@ -374,14 +375,12 @@ def log_filepath(self):
@property
def log_url(self):
-iso = quote(self.execution_date.isoformat())
-base_url = configuration.conf.get('webserver', 'BASE_URL')
-return base_url + (
-"/log?"
-"execution_date={iso}"
-"_id={task_id}"
-"_id={dag_id}"
-).format(iso=iso, task_id=self.task_id, dag_id=self.dag_id)
+url = url_for(
Review comment:
Oh also - this url is going to be wrong for RBAC too :( (`Airflow.log` is
what we need for RBAC)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
feluelle commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC URL: https://github.com/apache/airflow/pull/5525#discussion_r300022706 ## File path: airflow/www/views.py ## @@ -122,10 +122,15 @@ def dag_link(v, c, m, p): def log_url_formatter(v, c, m, p): +url = url_for( Review comment: True. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
ashb commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC URL: https://github.com/apache/airflow/pull/5525#discussion_r300022113 ## File path: airflow/www/views.py ## @@ -122,10 +122,15 @@ def dag_link(v, c, m, p): def log_url_formatter(v, c, m, p): +url = url_for( Review comment: Could we use `m.log_url` here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
feluelle commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log
link in TaskInstance's View for Non-RBAC
URL: https://github.com/apache/airflow/pull/5525#discussion_r300022133
##
File path: airflow/models/taskinstance.py
##
@@ -374,14 +375,12 @@ def log_filepath(self):
@property
def log_url(self):
-iso = quote(self.execution_date.isoformat())
-base_url = configuration.conf.get('webserver', 'BASE_URL')
-return base_url + (
-"/log?"
-"execution_date={iso}"
-"_id={task_id}"
-"_id={dag_id}"
-).format(iso=iso, task_id=self.task_id, dag_id=self.dag_id)
+url = url_for(
Review comment:
Ah I understand. Let me check if it also works in the email alert.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
ashb commented on a change in pull request #5525: [AIRFLOW-4890] Fix Log link
in TaskInstance's View for Non-RBAC
URL: https://github.com/apache/airflow/pull/5525#discussion_r300021162
##
File path: airflow/models/taskinstance.py
##
@@ -374,14 +375,12 @@ def log_filepath(self):
@property
def log_url(self):
-iso = quote(self.execution_date.isoformat())
-base_url = configuration.conf.get('webserver', 'BASE_URL')
-return base_url + (
-"/log?"
-"execution_date={iso}"
-"_id={task_id}"
-"_id={dag_id}"
-).format(iso=iso, task_id=self.task_id, dag_id=self.dag_id)
+url = url_for(
Review comment:
I think using url_for is going to break some uses of this - namely in
failure emails, where there is no active flask request, so url_for won't work.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (AIRFLOW-4882) Make GCP tests and examples Pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-4882. --- Resolution: Fixed Fix Version/s: 2.0.0 > Make GCP tests and examples Pylint compatible > - > > Key: AIRFLOW-4882 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4882 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] candu opened a new pull request #5526: [AIRFLOW-XXX] Add City of Toronto to official users list
candu opened a new pull request #5526: [AIRFLOW-XXX] Add City of Toronto to
official users list
URL: https://github.com/apache/airflow/pull/5526
The [City of Toronto Big Data Innovation
Team](https://www.toronto.ca/services-payments/streets-parking-transportation/road-safety/big-data-innovation-team/)
has started using Airflow to manage some of its ETL jobs and data analysis
pipelines.
Make sure you have checked _all_ steps below.
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
(as per: "In case you are fixing a typo in the documentation you can prepend
your commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.")
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
It adds City of Toronto to the list of official users in `README.md`. There
are no code, UI, etc. changes within Airflow itself.
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
There are no unit tests, as it's a one-line change to `README.md`. I did
double-check the commit diff to verify that it only adds the one line as
expected, and also checked the Preview output to verify that it renders as
expected.
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
I believe the commit message meets all subject line guidelines, except the
one about Jira issues - there is no associated Jira issue. There is no body,
as the change is straightforward.
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
There is no new functionality; using the `README.md` is largely
self-explanatory, I hope.
### Code Quality
- [X] Passes `flake8`
This is not applicable, as there are no changes to Python code with this
pull request.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] feluelle commented on issue #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
feluelle commented on issue #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC URL: https://github.com/apache/airflow/pull/5525#issuecomment-508119235 @ashb may you take a look at this? I targeted the change against `v1-10-test` since I thought this could be worth adding to 1.10.4 before releasing? There are already two related tickets (see my jira ticket) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4890) Fix Log link in TaskInstance's View for Non-RBAC
[
https://issues.apache.org/jira/browse/AIRFLOW-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877886#comment-16877886
]
ASF GitHub Bot commented on AIRFLOW-4890:
-
feluelle commented on pull request #5525: [AIRFLOW-4890] Fix Log link in
TaskInstance's View for Non-RBAC
URL: https://github.com/apache/airflow/pull/5525
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-4890
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Fixes Log link in TaskInstance's View for Non-RBAC by changing the static
creation of log link to dynamic generation via flask url_for
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
NOTE: I removed all related tests since there isn't much of testing this.
The only case that could be tested is the redirect to the page. A test for
this already exists "test_get_file_task_log". The creation of ti.log_url is the
"url_for" statement only.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fix Log link in TaskInstance's View for Non-RBAC
>
>
> Key: AIRFLOW-4890
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4890
> Project: Apache Airflow
> Issue Type: Bug
> Components: logging
>Affects Versions: 1.8.0, 1.10.3
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] feluelle opened a new pull request #5525: [AIRFLOW-4890] Fix Log link in TaskInstance's View for Non-RBAC
feluelle opened a new pull request #5525: [AIRFLOW-4890] Fix Log link in
TaskInstance's View for Non-RBAC
URL: https://github.com/apache/airflow/pull/5525
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-4890
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Fixes Log link in TaskInstance's View for Non-RBAC by changing the static
creation of log link to dynamic generation via flask url_for
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
NOTE: I removed all related tests since there isn't much of testing this.
The only case that could be tested is the redirect to the page. A test for
this already exists "test_get_file_task_log". The creation of ti.log_url is the
"url_for" statement only.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Closed] (AIRFLOW-874) Backfill dates spanning multiple years fail to parse
[
https://issues.apache.org/jira/browse/AIRFLOW-874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor closed AIRFLOW-874.
-
Resolution: Not A Problem
If you copy and pasted the command the issue is {{–}} vs {{-}} in your 2017
date - you have an en-dash, not a hyphen there
> Backfill dates spanning multiple years fail to parse
>
>
> Key: AIRFLOW-874
> URL: https://issues.apache.org/jira/browse/AIRFLOW-874
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Affects Versions: 1.7.1.3
> Environment: Ubuntu
>Reporter: Russell Jurney
>Priority: Critical
> Labels: easyfix
>
> In following output, note that 2017-01-01 is a valid iso date, but airflow
> says otherwise. My guess is that this has to do with the range spanning
> years? I could be wrong.
> -
> airflow backfill internet_honey_spark_workflow -t spark_postgres_to_pyspark
> -s 2016-10-01 -e 2017–01-02
> [2017-02-14 04:36:46,927] {__init__.py:36} INFO - Using executor
> SequentialExecutor
> usage: airflow backfill [-h] [-t TASK_REGEX] [-s START_DATE] [-e END_DATE]
> [-m] [-l] [-x] [-a] [-i] [-I] [-sd SUBDIR]
> [--pool POOL] [-dr]
> dag_id
> airflow backfill: error: argument -e/--end_date: invalid parse value:
> '2017–01-02'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [airflow] nuclearpinguin commented on issue #5524: [AIRFLOW-4891] Extend list of pylint good-names
nuclearpinguin commented on issue #5524: [AIRFLOW-4891] Extend list of pylint
good-names
URL: https://github.com/apache/airflow/pull/5524#issuecomment-508115990
From macro documentation:
> {{ ts }} | same as execution_date.isoformat(). Example:
2018-01-01T00:00:00+00:00
You are right that `t` is often used in tests for task / sensor / operator.
But there are also other usages like `t = MySQLdb.constants.FIELD_TYPE` so I
would consider `t` as to ambiguous.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (AIRFLOW-2317) Support for multiple resource pools
[ https://issues.apache.org/jira/browse/AIRFLOW-2317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-2317. Resolution: Duplicate > Support for multiple resource pools > --- > > Key: AIRFLOW-2317 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2317 > Project: Apache Airflow > Issue Type: Wish > Components: scheduler >Reporter: James Davidheiser >Priority: Minor > Labels: pool > > We are migrating to Airflow from Luigi, where we have the capability to > require multiple pools, and multiple "units" of a pool, for a given task. > This is very useful for a variety of use cases, but the two core examples are: > * If a task accessing a data store is extremely resource-intensive, we might > define a pool size of 10 resources, but use 5 of those in a single task to > denote the extra load. When smaller tasks are using the same data store, > it's completely fine to allow more of them to run at once. > If a task is connecting to two different data stores, to transfer data > between them, we might want to require a resource for each data store, so we > can limit concurrency on both simultaneously. > > > I know there are a lot of other issues related to capacity scheduling, which > are tracked more broadly in https://issues.apache.org/jira/browse/AIRFLOW-72, > and using more than one pool slot was suggested in > https://issues.apache.org/jira/browse/AIRFLOW-1467, so this seems like it > could be a useful thing to consider as part of larger efforts around managing > capacity. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4082) Pool capacity is not surfaced in the UI
[ https://issues.apache.org/jira/browse/AIRFLOW-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877881#comment-16877881 ] Ash Berlin-Taylor commented on AIRFLOW-4082: Also related to https://issues.apache.org/jira/browse/AIRFLOW-3790 > Pool capacity is not surfaced in the UI > --- > > Key: AIRFLOW-4082 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4082 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Affects Versions: 1.10.1 >Reporter: James Meickle >Priority: Minor > > We had a task get blocked due to insufficient pool capacity. This wasn't > available anywhere in the UI as far as I can tell. I had to go look at the > scheduler logs to see that, slowing down debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-4082) Pool capacity is not surfaced in the UI
[ https://issues.apache.org/jira/browse/AIRFLOW-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-4082. Resolution: Not A Problem > Pool capacity is not surfaced in the UI > --- > > Key: AIRFLOW-4082 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4082 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Affects Versions: 1.10.1 >Reporter: James Meickle >Priority: Minor > > We had a task get blocked due to insufficient pool capacity. This wasn't > available anywhere in the UI as far as I can tell. I had to go look at the > scheduler logs to see that, slowing down debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] ashb edited a comment on issue #5524: [AIRFLOW-4891] Extend list of pylint good-names
ashb edited a comment on issue #5524: [AIRFLOW-4891] Extend list of pylint good-names URL: https://github.com/apache/airflow/pull/5524#issuecomment-508111445 Yes to `df`. `ts` = timestamp? What are the uses of `t`? That's probably `task` isn't it? It may actually be clearer to use `task` instead of `t`. Not sure why I feel different about that var name... :D This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] shuwen-zhang commented on a change in pull request #5516: [AIRFLOW-4884] Roll up import_errors in UI
shuwen-zhang commented on a change in pull request #5516: [AIRFLOW-4884] Roll
up import_errors in UI
URL: https://github.com/apache/airflow/pull/5516#discussion_r299979821
##
File path: airflow/www/templates/appbuilder/flash.html
##
@@ -0,0 +1,68 @@
+
+
+
+
+
+{% with messages = get_flashed_messages(with_categories=true) %}
+{% if messages %}
+
+{% set broken_dag_messages = [] %}
+{% set other_messages = [] %}
+
+{% for category, m in messages %}
+{% if m.startswith('Broken DAG') %}
Review comment:
Hi @ashb thanks for reviewing. Instead of matching string-prefix we could
create another `category` like `quiet_error` or `collapsible_error`, and flash
the Broken DAG error in airflow/www/views.py with that category instead.
(Others could also flash messages with that category elsewhere.) Then in the
flash.html file, we could filter for that new category and apply the
alert-error css so those messages have the red background in the collapsible.
What do you think of this approach?
We’ll also add in the missing Apache license headers that led to the build
fail.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #5524: [AIRFLOW-4891] Extend list of pylint good-names
ashb commented on issue #5524: [AIRFLOW-4891] Extend list of pylint good-names URL: https://github.com/apache/airflow/pull/5524#issuecomment-508111445 Yes to `df`. `ts` = timestamp? What are the uses of `t`? That's probably `task` isn't it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #5524: [AIRFLOW-4891] Extend list of pylint good-names
nuclearpinguin commented on issue #5524: [AIRFLOW-4891] Extend list of pylint good-names URL: https://github.com/apache/airflow/pull/5524#issuecomment-508110908 Here is a list of top 10 invalid names with number of occurrences: ``` 12 Variable name "t2" doesn't conform to snake_case naming style (invalid-name) 13 Variable name "ts" doesn't conform to snake_case naming style (invalid-name) 15 Variable name "df" doesn't conform to snake_case naming style (invalid-name) 17 Variable name "v" doesn't conform to snake_case naming style (invalid-name) 19 Variable name "p" doesn't conform to snake_case naming style (invalid-name) 22 Variable name "b" doesn't conform to snake_case naming style (invalid-name) 23 Variable name "m" doesn't conform to snake_case naming style (invalid-name) 24 Variable name "c" doesn't conform to snake_case naming style (invalid-name) 26 Variable name "s" doesn't conform to snake_case naming style (invalid-name) 39 Variable name "d" doesn't conform to snake_case naming style (invalid-name) 107 Variable name "t" doesn't conform to snake_case naming style (invalid-name) ``` So I would suggest also `df` as it's popular name for `pd.DataFrame`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Issue Comment Deleted] (AIRFLOW-3013) LDAPInvalidPortError: port must be an integer
[
https://issues.apache.org/jira/browse/AIRFLOW-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jack updated AIRFLOW-3013:
--
Comment: was deleted
(was: Can you check if this happens to you in the new UI?)
> LDAPInvalidPortError: port must be an integer
> -
>
> Key: AIRFLOW-3013
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3013
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication
>Affects Versions: 1.10.0
> Environment: debian stretch, kubernetes, docker
>Reporter: sebastianwth
>Priority: Minor
>
> I am receiving the following error when attempting to login to airflow
> through it's webui after configuring ldap.
> {code}
> Traceback (most recent call last): File
> "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1982, in wsgi_app
> response = self.full_dispatch_request() File
> "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1614, in
> full_dispatch_request rv = self.handle_user_exception(e) File
> "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1517, in
> handle_user_exception reraise(exc_type, exc_value, tb) File
> "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in
> full_dispatch_request rv = self.dispatch_request() File
> "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in
> dispatch_request return self.view_functions[rule.endpoint](**req.view_args)
> File "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 69,
> in inner return self._run_view(f, *args, **kwargs) File
> "/usr/local/lib/python2.7/dist-packages/flask_admin/base.py", line 368, in
> _run_view return fn(self, *args, **kwargs) File
> "/usr/local/lib/python2.7/dist-packages/airflow/www/views.py", line 735, in
> login return airflow.login.login(self, request) File
> "/usr/local/lib/python2.7/dist-packages/airflow/utils/db.py", line 74, in
> wrapper return func(*args, **kwargs) File
> "/usr/local/lib/python2.7/dist-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 296, in login LdapUser.try_login(username, password) File
> "/usr/local/lib/python2.7/dist-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 187, in try_login configuration.conf.get("ldap", "bind_password")) File
> "/usr/local/lib/python2.7/dist-packages/airflow/contrib/auth/backends/ldap_auth.py",
> line 68, in get_ldap_connection server =
> Server(configuration.conf.get("ldap", "uri"), use_ssl, tls_configuration)
> File "/usr/local/lib/python2.7/dist-packages/ldap3/core/server.py", line 118,
> in __init__ raise LDAPInvalidPortError('port must be an integer')
> LDAPInvalidPortError: port must be an integer{code}
> My airflow.cfg looks as such:
> {code:java}
> [ webserver ]
> authenticate = True
> auth_backend = airflow.contrib.auth.backends.ldap_auth
> [ldap]
> uri = ldaps://ldap.jumpcloud.com:636
> user_filter = objectClass=*
> user_name_attr = uid
> group_member_attr = memberOf
> bind_user = uid=airflowldapservice,ou=Users,o=ORGANIZATION,dc=jumpcloud,dc=com
> bind_password = PASSWORD
> basedn = ou=Users,o=ORGANIZATION,dc=jumpcloud,dc=com
> cacert = /etc/ca/ldap_ca.crt
> search_scope = LEVEL{code}
> My pip packages
> {code:java}
> adal==1.0.2
> alabaster==0.7.11
> alembic==0.8.10
> amqp==2.3.2
> analytics-python==1.2.9
> apache-airflow==1.10.0
> asn1crypto==0.24.0
> atlasclient==0.1.2
> awscli==1.16.1
> azure-common==1.1.14
> azure-datalake-store==0.0.19
> azure-mgmt-datalake-nspkg==2.0.0
> azure-mgmt-datalake-store==0.4.0
> azure-mgmt-nspkg==2.0.0
> azure-mgmt-resource==1.2.2
> azure-nspkg==2.0.0
> azure-storage==0.36.0
> Babel==2.6.0
> backports-abc==0.5
> backports.ssl-match-hostname==3.5.0.1
> bcrypt==3.1.4
> billiard==3.5.0.4
> bleach==2.1.2
> boto==2.49.0
> boto3==1.8.5
> botocore==1.11.1
> cachetools==2.1.0
> cassandra-driver==3.15.0
> celery==4.2.1
> certifi==2018.8.24
> cffi==1.11.5
> chardet==3.0.4
> click==6.7
> cloudant==0.5.10
> colorama==0.3.9
> configparser==3.5.0
> cookies==2.2.1
> croniter==0.3.25
> cryptography==2.3.1
> cx-Oracle==6.4.1
> datadog==0.22.0
> decorator==4.3.0
> defusedxml==0.5.0
> dicttoxml==1.7.4
> dill==0.2.8.2
> docker==3.5.0
> docker-pycreds==0.3.0
> docopt==0.6.2
> docutils==0.14
> elasticsearch==5.5.3
> elasticsearch-dsl==5.4.0
> entrypoints==0.2.3
> enum34==1.1.6
> fastavro==0.9.6
> Flask==0.12.4
> Flask-Admin==1.4.1
> Flask-AppBuilder==1.11.1
> Flask-Babel==0.11.1
> Flask-Bcrypt==0.7.1
> Flask-Caching==1.3.3
> Flask-Login==0.2.11
> Flask-OpenID==1.2.5
> Flask-SQLAlchemy==2.1
> flask-swagger==0.2.13
> Flask-WTF==0.14.2
> flower==0.9.2
> freezegun==0.3.10
> funcsigs==1.0.0
> future==0.16.0
> future-fstrings==0.4.4
> futures==3.2.0
> gitdb2==2.0.4
> GitPython==2.1.11
> google-api-core==1.3.0
> google-api-python-client==1.7.4
> google-auth==1.5.1
> google-auth-httplib2==0.0.3
>
[GitHub] [airflow] ashb commented on issue #5524: [AIRFLOW-4891] Extend list of pylint good-names
ashb commented on issue #5524: [AIRFLOW-4891] Extend list of pylint good-names URL: https://github.com/apache/airflow/pull/5524#issuecomment-508102084 Are there any others do you think? Should we wait/keep this Jira open for a while? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on issue #5524: [AIRFLOW-4891] Extend list of pylint good-names
nuclearpinguin commented on issue #5524: [AIRFLOW-4891] Extend list of pylint good-names URL: https://github.com/apache/airflow/pull/5524#issuecomment-508101188 @mik-laj @potiuk @ashb what are your thoughts? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-4891) Extend list of pylint good-names
[ https://issues.apache.org/jira/browse/AIRFLOW-4891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877855#comment-16877855 ] ASF GitHub Bot commented on AIRFLOW-4891: - nuclearpinguin commented on pull request #5524: [AIRFLOW-4891] Extend list of pylint good-names URL: https://github.com/apache/airflow/pull/5524 I would like to propose further extension of pylint good-names: - `v` used when iterating :`for k, v in dictionary.items():` - 'ds` and `ts` used in templates (https://airflow.apache.org/macros.html) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Extend list of pylint good-names > - > > Key: AIRFLOW-4891 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4891 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: Tomasz Urbaszek >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [airflow] nuclearpinguin opened a new pull request #5524: [AIRFLOW-4891] Extend list of pylint good-names
nuclearpinguin opened a new pull request #5524: [AIRFLOW-4891] Extend list of pylint good-names URL: https://github.com/apache/airflow/pull/5524 I would like to propose further extension of pylint good-names: - `v` used when iterating :`for k, v in dictionary.items():` - 'ds` and `ts` used in templates (https://airflow.apache.org/macros.html) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] exploy commented on issue #5508: [AIRFLOW-4876] Making tests in CoreTest rerunnable
exploy commented on issue #5508: [AIRFLOW-4876] Making tests in CoreTest rerunnable URL: https://github.com/apache/airflow/pull/5508#issuecomment-508100912 Extracted the commit to the separate PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #5508: [AIRFLOW-4876] Making tests in CoreTest rerunnable
codecov-io edited a comment on issue #5508: [AIRFLOW-4876] Making tests in CoreTest rerunnable URL: https://github.com/apache/airflow/pull/5508#issuecomment-507625595 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5508?src=pr=h1) Report > Merging [#5508](https://codecov.io/gh/apache/airflow/pull/5508?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/7448d3c0cd85770021c230bbea33f758718e300b?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/5508?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5508 +/- ## == + Coverage 79.07% 79.07% +<.01% == Files 489 489 Lines 3074430744 == + Hits2431124312 +1 + Misses 6433 6432 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5508?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/5508/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.18% <0%> (+0.16%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5508?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5508?src=pr=footer). Last update [7448d3c...639f679](https://codecov.io/gh/apache/airflow/pull/5508?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] exploy commented on issue #5523: [AIRFLOW-XXX] Add Growbots as the user of Airflow
exploy commented on issue #5523: [AIRFLOW-XXX] Add Growbots as the user of Airflow URL: https://github.com/apache/airflow/pull/5523#issuecomment-508100595 Thx This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb merged pull request #5523: [AIRFLOW-XXX] Add Growbots as the user of Airflow
ashb merged pull request #5523: [AIRFLOW-XXX] Add Growbots as the user of Airflow URL: https://github.com/apache/airflow/pull/5523 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #5523: [AIRFLOW-XXX] Add Growbots as the user of Airflow
ashb commented on issue #5523: [AIRFLOW-XXX] Add Growbots as the user of Airflow URL: https://github.com/apache/airflow/pull/5523#issuecomment-508100144 Welcome! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] exploy opened a new pull request #5523: [AIRFLOW-XXX] Add Growbots as the user of Airflow
exploy opened a new pull request #5523: [AIRFLOW-XXX] Add Growbots as the user
of Airflow
URL: https://github.com/apache/airflow/pull/5523
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
Simply add Growbots as Airflow users
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3244) Introduce offset on the execution date for data settlement
[ https://issues.apache.org/jira/browse/AIRFLOW-3244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877847#comment-16877847 ] jack commented on AIRFLOW-3244: --- I believe this is resolved by https://issues.apache.org/jira/browse/AIRFLOW-3560 which introduced DayOfWeekSensor? > Introduce offset on the execution date for data settlement > -- > > Key: AIRFLOW-3244 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3244 > Project: Apache Airflow > Issue Type: Improvement > Components: DAG >Affects Versions: 1.10.0 >Reporter: Alberto Anceschi >Priority: Minor > Labels: features, request > > Hi everyone, > > I'm trying to port my current cronjobs into Airflow. Let's consider a real > case scenario: I've to send every week a report and through the pipeline data > from Google Analytics needs to be collected, so I need 2 days before running > the DAG (data assessment). Week starts on Monday and ends on Sunday, so I > need the DAG to run on Wednesday at Midnight UTC. > In order to see on the Airflow dashboard start_date/exection_date that make > sense to me, for now I've used a TimeDeltaSensor that adds that 2 day offset > I need, but this is not its purpose. I also use Celery executor, so its > workers keep polling during those 2 days, making them unavailable for other > DAGs. > I think that the assumption that at the end of the period scheduled data are > ready is not correct and at the same time it's much more intuitive seeing on > the dashboard Monday execution dates instead of Tuesday ones. > > What do you think about this request? Any suggestion? Thank you, > > Alberto > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4761) Airflow Task Clear function throws error
[
https://issues.apache.org/jira/browse/AIRFLOW-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877845#comment-16877845
]
jack commented on AIRFLOW-4761:
---
Can you provide more info on what sort of tasks this happens?
> Airflow Task Clear function throws error
>
>
> Key: AIRFLOW-4761
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4761
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, DagRun
>Affects Versions: 1.10.3
> Environment: CentOS 7, Python 2.7.10
>Reporter: Ben Storrie
>Priority: Major
>
> When using the airflow webserver to clear a task inside a dagrun, an error is
> thrown on certain types of tasks:
>
> {code:java}
> Traceback (most recent call last):
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/site-packages/flask/app.py",
> line 2311, in wsgi_app
> response = self.full_dispatch_request()
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/site-packages/flask/app.py",
> line 1834, in full_dispatch_request
> rv = self.handle_user_exception(e)
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/site-packages/flask/app.py",
> line 1737, in handle_user_exception
> reraise(exc_type, exc_value, tb)
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/site-packages/flask/app.py",
> line 1832, in full_dispatch_request
> rv = self.dispatch_request()
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/site-packages/flask/app.py",
> line 1818, in dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/site-packages/flask_admin/base.py",
> line 69, in inner
> return self._run_view(f, *args, **kwargs)
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/site-packages/flask_admin/base.py",
> line 368, in _run_view
> return fn(self, *args, **kwargs)
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/site-packages/flask_login/utils.py",
> line 261, in decorated_view
> return func(*args, **kwargs)
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/site-packages/airflow/www/utils.py",
> line 275, in wrapper
> return f(*args, **kwargs)
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/site-packages/airflow/www/utils.py",
> line 322, in wrapper
> return f(*args, **kwargs)
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/site-packages/airflow/www/views.py",
> line 1202, in clear
> include_upstream=upstream)
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/site-packages/airflow/models/__init__.py",
> line 3830, in sub_dag
> dag = copy.deepcopy(self)
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/copy.py",
> line 174, in deepcopy
> y = copier(memo)
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/site-packages/airflow/models/__init__.py",
> line 3815, in __deepcopy__
> setattr(result, k, copy.deepcopy(v, memo))
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/copy.py",
> line 163, in deepcopy
> y = copier(x, memo)
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/copy.py",
> line 257, in _deepcopy_dict
> y[deepcopy(key, memo)] = deepcopy(value, memo)
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/copy.py",
> line 174, in deepcopy
> y = copier(memo)
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/site-packages/airflow/models/__init__.py",
> line 2492, in __deepcopy__
> setattr(result, k, copy.copy(v))
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/copy.py",
> line 96, in copy
> return _reconstruct(x, rv, 0)
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/copy.py",
> line 329, in _reconstruct
> y = callable(*args)
> File
> "/opt/my-miniconda/miniconda/envs/my-hadoop-airflow/lib/python2.7/copy_reg.py",
> line 93, in __newobj__
> return cls.__new__(cls, *args)
> TypeError: instancemethod expected at least 2 arguments, got 0{code}
>
> I had expected AIRFLOW-2060 being resolved to resolve this on upgrade to
> 1.10.3:
> {code:java}
> (my-hadoop-airflow) [user@hostname ~]$ pip freeze | grep pendulum
> pendulum==1.4.4{code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2917) Set AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed for k8s executor
[ https://issues.apache.org/jira/browse/AIRFLOW-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877846#comment-16877846 ] John Cheng commented on AIRFLOW-2917: - I agree all secret should be handle with k8s sectet, including fernet key and SMTP password, but not coded in the config map. > Set AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed for k8s executor > - > > Key: AIRFLOW-2917 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2917 > Project: Apache Airflow > Issue Type: Improvement > Components: executors >Affects Versions: 1.10.0 >Reporter: John Cheng >Assignee: John Cheng >Priority: Minor > Fix For: 1.10.2 > > > In Kubernetes executor, `AIRFLOW__CORE__SQL_ALCHEMY_CONN` is set as an > environment variable even when it is specified in configmap or secrets. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2917) Set AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed for k8s executor
[ https://issues.apache.org/jira/browse/AIRFLOW-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877840#comment-16877840 ] John Cheng commented on AIRFLOW-2917: - Passing connection string to env is more dangerous. I just want to avoid that. > Set AIRFLOW__CORE__SQL_ALCHEMY_CONN only when needed for k8s executor > - > > Key: AIRFLOW-2917 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2917 > Project: Apache Airflow > Issue Type: Improvement > Components: executors >Affects Versions: 1.10.0 >Reporter: John Cheng >Assignee: John Cheng >Priority: Minor > Fix For: 1.10.2 > > > In Kubernetes executor, `AIRFLOW__CORE__SQL_ALCHEMY_CONN` is set as an > environment variable even when it is specified in configmap or secrets. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1498) Add Google Analytics to Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877839#comment-16877839 ] jack commented on AIRFLOW-1498: --- [~msumit] What do you mean by add Google Analytics to Airflow *UI*? > Add Google Analytics to Airflow > --- > > Key: AIRFLOW-1498 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1498 > Project: Apache Airflow > Issue Type: New Feature > Components: ui >Reporter: Sumit Maheshwari >Priority: Major > > It would be very useful if we can configure google analytics into Airflow UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4890) Fix Log link in TaskInstance's View for Non-RBAC
Felix Uellendall created AIRFLOW-4890: - Summary: Fix Log link in TaskInstance's View for Non-RBAC Key: AIRFLOW-4890 URL: https://issues.apache.org/jira/browse/AIRFLOW-4890 Project: Apache Airflow Issue Type: Bug Components: logging Affects Versions: 1.10.3, 1.8.0 Reporter: Felix Uellendall Assignee: Felix Uellendall -- This message was sent by Atlassian JIRA (v7.6.3#76005)
