[jira] [Commented] (AIRFLOW-490) Scope should be auto-added to the connection per GCP hook
[ https://issues.apache.org/jira/browse/AIRFLOW-490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719850#comment-16719850 ] jack commented on AIRFLOW-490: -- After checking... this was already implemented: https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/gcp_api_base_hook.py#L32 > Scope should be auto-added to the connection per GCP hook > - > > Key: AIRFLOW-490 > URL: https://issues.apache.org/jira/browse/AIRFLOW-490 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Major > > The GCP hooks should auto add the scope they need to work to the GCP > connection so a user don't need specify the obvious scopes, like: > https://www.googleapis.com/auth/cloud-platform > We should keep the scope field for extra scopes requested for special cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ron819 commented on issue #4312: AIRFLOW-3508: add slugify env to build.sh script
ron819 commented on issue #4312: AIRFLOW-3508: add slugify env to build.sh script URL: https://github.com/apache/incubator-airflow/pull/4312#issuecomment-446866479 Related also to : https://issues.apache.org/jira/browse/AIRFLOW-3400 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3508) docker build script fails because of the existing areski/python-nvd3 dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719838#comment-16719838 ] ASF GitHub Bot commented on AIRFLOW-3508: - eran-levy opened a new pull request #4312: AIRFLOW-3508: add slugify env to build.sh script URL: https://github.com/apache/incubator-airflow/pull/4312 See AIRFLOW-3508 Jira ticket docker build script fails because of the existing areski/python-nvd3 dependency. incubator-airflow/scripts/ci/kubernetes/docker/build.sh fails to start setup because of the existing areski/python-nvd3 dependency that requires python-slugify which installs unidecode (GPL) for its decoding needs. The python setup.py sdist -q fails with the following error: "By default one of Airflow's dependencies installs a GPL dependency (unidecode). To avoid this..." Many experience the same issue, so its better to avoid it by adding SLUGIFY_USES_TEXT_UNIDECODE=yes env to install and use text-unidecode This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > docker build script fails because of the existing areski/python-nvd3 > dependency > --- > > Key: AIRFLOW-3508 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3508 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 1.10.1 >Reporter: Eran Levy >Assignee: Eran Levy >Priority: Major > > incubator-airflow/scripts/ci/kubernetes/docker/build.sh fails to start setup > because of the existing areski/python-nvd3 dependency that requires > python-slugify which installs *unidecode* (GPL) for its decoding needs. > The > python setup.py sdist -q fails with the following error: > "By default one of Airflow's dependencies installs a GPL dependency > (unidecode). To avoid this..." > Many experience the same issue, so its better to avoid it by adding > SLUGIFY_USES_TEXT_UNIDECODE=yes env to install and use text-unidecode > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] eran-levy opened a new pull request #4312: AIRFLOW-3508: add slugify env to build.sh script
eran-levy opened a new pull request #4312: AIRFLOW-3508: add slugify env to build.sh script URL: https://github.com/apache/incubator-airflow/pull/4312 See AIRFLOW-3508 Jira ticket docker build script fails because of the existing areski/python-nvd3 dependency. incubator-airflow/scripts/ci/kubernetes/docker/build.sh fails to start setup because of the existing areski/python-nvd3 dependency that requires python-slugify which installs unidecode (GPL) for its decoding needs. The python setup.py sdist -q fails with the following error: "By default one of Airflow's dependencies installs a GPL dependency (unidecode). To avoid this..." Many experience the same issue, so its better to avoid it by adding SLUGIFY_USES_TEXT_UNIDECODE=yes env to install and use text-unidecode This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work started] (AIRFLOW-3508) docker build script fails because of the existing areski/python-nvd3 dependency
[ https://issues.apache.org/jira/browse/AIRFLOW-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-3508 started by Eran Levy. -- > docker build script fails because of the existing areski/python-nvd3 > dependency > --- > > Key: AIRFLOW-3508 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3508 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 1.10.1 >Reporter: Eran Levy >Assignee: Eran Levy >Priority: Major > > incubator-airflow/scripts/ci/kubernetes/docker/build.sh fails to start setup > because of the existing areski/python-nvd3 dependency that requires > python-slugify which installs *unidecode* (GPL) for its decoding needs. > The > python setup.py sdist -q fails with the following error: > "By default one of Airflow's dependencies installs a GPL dependency > (unidecode). To avoid this..." > Many experience the same issue, so its better to avoid it by adding > SLUGIFY_USES_TEXT_UNIDECODE=yes env to install and use text-unidecode > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3508) docker build script fails because of the existing areski/python-nvd3 dependency
Eran Levy created AIRFLOW-3508: -- Summary: docker build script fails because of the existing areski/python-nvd3 dependency Key: AIRFLOW-3508 URL: https://issues.apache.org/jira/browse/AIRFLOW-3508 Project: Apache Airflow Issue Type: Bug Components: ci Affects Versions: 1.10.1 Reporter: Eran Levy Assignee: Eran Levy incubator-airflow/scripts/ci/kubernetes/docker/build.sh fails to start setup because of the existing areski/python-nvd3 dependency that requires python-slugify which installs *unidecode* (GPL) for its decoding needs. The python setup.py sdist -q fails with the following error: "By default one of Airflow's dependencies installs a GPL dependency (unidecode). To avoid this..." Many experience the same issue, so its better to avoid it by adding SLUGIFY_USES_TEXT_UNIDECODE=yes env to install and use text-unidecode -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i…
feng-tao commented on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i… URL: https://github.com/apache/incubator-airflow/pull/3683#issuecomment-446859158 thanks @dimberman , how about you do some investigations over the weekend? If you couldn't figure out, we could then revert the pr next week. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dimberman commented on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i…
dimberman commented on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i… URL: https://github.com/apache/incubator-airflow/pull/3683#issuecomment-446858775 It's super weird. I noticed that too but the tests were passing when I checked this branch. Especially noticeable that the jobs are stuck in queued (which tell me the executor is not launching anything). I'm at Kubecon until tomorrow but should be around this weekend. Should we revert and I'll investigate further? I apologize for the inconvenience. One of my short/medium term plans is to improve the test coverage for the k8s executor to catch these problems earlier. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3507) Fix Airflow k8s CI
[ https://issues.apache.org/jira/browse/AIRFLOW-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng updated AIRFLOW-3507: -- Summary: Fix Airflow k8s CI (was: Fix k8s ci) > Fix Airflow k8s CI > -- > > Key: AIRFLOW-3507 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3507 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Assignee: Tao Feng >Priority: Major > > The k8s ci is failed with two > tests([https://travis-ci.org/apache/incubator-airflow/jobs/467343912).] Fix > the tests to unblock airflow k8s ci. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3507) Fix Airflow k8s CI
[ https://issues.apache.org/jira/browse/AIRFLOW-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng reassigned AIRFLOW-3507: - Assignee: (was: Tao Feng) > Fix Airflow k8s CI > -- > > Key: AIRFLOW-3507 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3507 > Project: Apache Airflow > Issue Type: Bug >Reporter: Tao Feng >Priority: Major > > The k8s ci is failed with two > tests([https://travis-ci.org/apache/incubator-airflow/jobs/467343912).] Fix > the tests to unblock airflow k8s ci. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3505) Change name of 'dags_in_docker' field to 'dags_in_image'
[ https://issues.apache.org/jira/browse/AIRFLOW-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719755#comment-16719755 ] ASF GitHub Bot commented on AIRFLOW-3505: - feng-tao closed pull request #4311: [AIRFLOW-3505] replace 'dags_in_docker' with 'dags_in_image' URL: https://github.com/apache/incubator-airflow/pull/4311 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/config_templates/default_airflow.cfg b/airflow/config_templates/default_airflow.cfg index 7b631b31bc..9c21f5d47e 100644 --- a/airflow/config_templates/default_airflow.cfg +++ b/airflow/config_templates/default_airflow.cfg @@ -607,7 +607,7 @@ airflow_configmap = # For docker image already contains DAGs, this is set to `True`, and the worker will search for dags in dags_folder, # otherwise use git sync or dags volumn chaim to mount DAGs -dags_in_docker = FALSE +dags_in_image = FALSE # For either git sync or volume mounted DAGs, the worker will look in this subpath for DAGs dags_volume_subpath = diff --git a/airflow/contrib/executors/kubernetes_executor.py b/airflow/contrib/executors/kubernetes_executor.py index 784ffcafc3..ca0cc1d128 100644 --- a/airflow/contrib/executors/kubernetes_executor.py +++ b/airflow/contrib/executors/kubernetes_executor.py @@ -139,7 +139,7 @@ def __init__(self): # NOTE: user can build the dags into the docker image directly, # this will set to True if so -self.dags_in_docker = conf.get(self.kubernetes_section, 'dags_in_docker') +self.dags_in_image = conf.get(self.kubernetes_section, 'dags_in_image') # NOTE: `git_repo` and `git_branch` must be specified together as a pair # The http URL of the git repository to clone from @@ -208,12 +208,12 @@ def __init__(self): self._validate() def _validate(self): -if not self.dags_volume_claim and not self.dags_in_docker \ +if not self.dags_volume_claim and not self.dags_in_image \ and (not self.git_repo or not self.git_branch): raise AirflowConfigException( 'In kubernetes mode the following must be set in the `kubernetes` ' 'config section: `dags_volume_claim` or `git_repo and git_branch` ' -'or `dags_in_docker`') +'or `dags_in_image`') class KubernetesJobWatcher(multiprocessing.Process, LoggingMixin, object): diff --git a/airflow/contrib/kubernetes/worker_configuration.py b/airflow/contrib/kubernetes/worker_configuration.py index aa23c4eef7..58cf9cbd20 100644 --- a/airflow/contrib/kubernetes/worker_configuration.py +++ b/airflow/contrib/kubernetes/worker_configuration.py @@ -38,7 +38,7 @@ def __init__(self, kube_config): def _get_init_containers(self, volume_mounts): """When using git to retrieve the DAGs, use the GitSync Init Container""" # If we're using volume claims to mount the dags, no init container is needed -if self.kube_config.dags_volume_claim or self.kube_config.dags_in_docker: +if self.kube_config.dags_volume_claim or self.kube_config.dags_in_image: return [] # Otherwise, define a git-sync init container @@ -134,7 +134,7 @@ def _construct_volume(name, claim): ) ] -if not self.kube_config.dags_in_docker: +if not self.kube_config.dags_in_image: volumes.append( _construct_volume( dags_volume_name, @@ -153,7 +153,7 @@ def _construct_volume(name, claim): logs_volume_mount ] -if not self.kube_config.dags_in_docker: +if not self.kube_config.dags_in_image: dag_volume_mount_path = "" if self.kube_config.dags_volume_claim: diff --git a/scripts/ci/kubernetes/kube/configmaps.yaml b/scripts/ci/kubernetes/kube/configmaps.yaml index 4c7ac388f4..759c7d637f 100644 --- a/scripts/ci/kubernetes/kube/configmaps.yaml +++ b/scripts/ci/kubernetes/kube/configmaps.yaml @@ -178,7 +178,7 @@ data: worker_container_image_pull_policy = IfNotPresent worker_dags_folder = /tmp/dags delete_worker_pods = True -dags_in_docker = False +dags_in_image = False git_repo = https://github.com/apache/incubator-airflow.git git_branch = master git_subpath = airflow/example_dags/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Change name of
[GitHub] BrechtDeVlieger commented on issue #4307: [AIRFLOW-3501] k8s executor - Support loading dags from image.
BrechtDeVlieger commented on issue #4307: [AIRFLOW-3501] k8s executor - Support loading dags from image. URL: https://github.com/apache/incubator-airflow/pull/4307#issuecomment-446851324 I believe this is already covered by [[AIRFLOW-2770] kubernetes: add support for dag folder in the docker image](https://github.com/apache/incubator-airflow/commit/e9a09d408e4cd1bda1d6e8b7670f08beab37de8a)? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3507) Fix k8s ci
Tao Feng created AIRFLOW-3507: - Summary: Fix k8s ci Key: AIRFLOW-3507 URL: https://issues.apache.org/jira/browse/AIRFLOW-3507 Project: Apache Airflow Issue Type: Bug Reporter: Tao Feng Assignee: Tao Feng The k8s ci is failed with two tests([https://travis-ci.org/apache/incubator-airflow/jobs/467343912).] Fix the tests to unblock airflow k8s ci. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] jmcarp commented on issue #4225: [AIRFLOW-3383] Rotate fernet keys.
jmcarp commented on issue #4225: [AIRFLOW-3383] Rotate fernet keys. URL: https://github.com/apache/incubator-airflow/pull/4225#issuecomment-446846988 Thanks for reviewing @ashb. I agree that `rotate_credentials` is too vague--I renamed the command to `rotate_fernet_key`. I also added documentation to `secure-connections.rst`. The build is failing, but looks like it's not related to these changes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Issue Comment Deleted] (AIRFLOW-3223) RBAC with GitHub Authentication
[ https://issues.apache.org/jira/browse/AIRFLOW-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng updated AIRFLOW-3223: -- Comment: was deleted (was: Would it be suitable to fix / add the feature in flask-appbuilder?) > RBAC with GitHub Authentication > > > Key: AIRFLOW-3223 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3223 > Project: Apache Airflow > Issue Type: Wish > Components: authentication >Affects Versions: 1.10.0 >Reporter: Vikram Fugro >Assignee: Sai Phanindhra >Priority: Major > > With airflow 1.10 released having RBAC support, I was wondering how I do > configure GitHub Auth with airflow's RBAC. In which case, I believe we don't > have to create any users using airflow. Are there any notes on this? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3223) RBAC with GitHub Authentication
[ https://issues.apache.org/jira/browse/AIRFLOW-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719757#comment-16719757 ] Tao Feng commented on AIRFLOW-3223: --- Would it be suitable to fix / add the feature in flask-appbuilder? > RBAC with GitHub Authentication > > > Key: AIRFLOW-3223 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3223 > Project: Apache Airflow > Issue Type: Wish > Components: authentication >Affects Versions: 1.10.0 >Reporter: Vikram Fugro >Assignee: Sai Phanindhra >Priority: Major > > With airflow 1.10 released having RBAC support, I was wondering how I do > configure GitHub Auth with airflow's RBAC. In which case, I believe we don't > have to create any users using airflow. Are there any notes on this? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i…
feng-tao commented on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i… URL: https://github.com/apache/incubator-airflow/pull/3683#issuecomment-446844127 @yeluolei @dimberman , it seems the integration for airflow k8s starts to fail with this change (CI log: https://travis-ci.org/apache/incubator-airflow/jobs/467296257). Do you know if it is related or it is a transient error? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on issue #4311: [AIRFLOW-3505] replace 'dags_in_docker' with 'dags_in_image'
feng-tao commented on issue #4311: [AIRFLOW-3505] replace 'dags_in_docker' with 'dags_in_image' URL: https://github.com/apache/incubator-airflow/pull/4311#issuecomment-446843682 lgtm This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao closed pull request #4311: [AIRFLOW-3505] replace 'dags_in_docker' with 'dags_in_image'
feng-tao closed pull request #4311: [AIRFLOW-3505] replace 'dags_in_docker' with 'dags_in_image' URL: https://github.com/apache/incubator-airflow/pull/4311 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/airflow/config_templates/default_airflow.cfg b/airflow/config_templates/default_airflow.cfg index 7b631b31bc..9c21f5d47e 100644 --- a/airflow/config_templates/default_airflow.cfg +++ b/airflow/config_templates/default_airflow.cfg @@ -607,7 +607,7 @@ airflow_configmap = # For docker image already contains DAGs, this is set to `True`, and the worker will search for dags in dags_folder, # otherwise use git sync or dags volumn chaim to mount DAGs -dags_in_docker = FALSE +dags_in_image = FALSE # For either git sync or volume mounted DAGs, the worker will look in this subpath for DAGs dags_volume_subpath = diff --git a/airflow/contrib/executors/kubernetes_executor.py b/airflow/contrib/executors/kubernetes_executor.py index 784ffcafc3..ca0cc1d128 100644 --- a/airflow/contrib/executors/kubernetes_executor.py +++ b/airflow/contrib/executors/kubernetes_executor.py @@ -139,7 +139,7 @@ def __init__(self): # NOTE: user can build the dags into the docker image directly, # this will set to True if so -self.dags_in_docker = conf.get(self.kubernetes_section, 'dags_in_docker') +self.dags_in_image = conf.get(self.kubernetes_section, 'dags_in_image') # NOTE: `git_repo` and `git_branch` must be specified together as a pair # The http URL of the git repository to clone from @@ -208,12 +208,12 @@ def __init__(self): self._validate() def _validate(self): -if not self.dags_volume_claim and not self.dags_in_docker \ +if not self.dags_volume_claim and not self.dags_in_image \ and (not self.git_repo or not self.git_branch): raise AirflowConfigException( 'In kubernetes mode the following must be set in the `kubernetes` ' 'config section: `dags_volume_claim` or `git_repo and git_branch` ' -'or `dags_in_docker`') +'or `dags_in_image`') class KubernetesJobWatcher(multiprocessing.Process, LoggingMixin, object): diff --git a/airflow/contrib/kubernetes/worker_configuration.py b/airflow/contrib/kubernetes/worker_configuration.py index aa23c4eef7..58cf9cbd20 100644 --- a/airflow/contrib/kubernetes/worker_configuration.py +++ b/airflow/contrib/kubernetes/worker_configuration.py @@ -38,7 +38,7 @@ def __init__(self, kube_config): def _get_init_containers(self, volume_mounts): """When using git to retrieve the DAGs, use the GitSync Init Container""" # If we're using volume claims to mount the dags, no init container is needed -if self.kube_config.dags_volume_claim or self.kube_config.dags_in_docker: +if self.kube_config.dags_volume_claim or self.kube_config.dags_in_image: return [] # Otherwise, define a git-sync init container @@ -134,7 +134,7 @@ def _construct_volume(name, claim): ) ] -if not self.kube_config.dags_in_docker: +if not self.kube_config.dags_in_image: volumes.append( _construct_volume( dags_volume_name, @@ -153,7 +153,7 @@ def _construct_volume(name, claim): logs_volume_mount ] -if not self.kube_config.dags_in_docker: +if not self.kube_config.dags_in_image: dag_volume_mount_path = "" if self.kube_config.dags_volume_claim: diff --git a/scripts/ci/kubernetes/kube/configmaps.yaml b/scripts/ci/kubernetes/kube/configmaps.yaml index 4c7ac388f4..759c7d637f 100644 --- a/scripts/ci/kubernetes/kube/configmaps.yaml +++ b/scripts/ci/kubernetes/kube/configmaps.yaml @@ -178,7 +178,7 @@ data: worker_container_image_pull_policy = IfNotPresent worker_dags_folder = /tmp/dags delete_worker_pods = True -dags_in_docker = False +dags_in_image = False git_repo = https://github.com/apache/incubator-airflow.git git_branch = master git_subpath = airflow/example_dags/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-3411) create openfaas hook
[ https://issues.apache.org/jira/browse/AIRFLOW-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3411. - Resolution: Fixed Fix Version/s: 1.10.2 > create openfaas hook > > > Key: AIRFLOW-3411 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3411 > Project: Apache Airflow > Issue Type: Improvement >Reporter: tal >Assignee: tal >Priority: Minor > Fix For: 1.10.2 > > > add openfaas hook to trigger functions > (https://github.com/openfaas/faas) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] jmcarp commented on a change in pull request #4279: [AIRFLOW-3444] Explicitly set transfer operator description.
jmcarp commented on a change in pull request #4279: [AIRFLOW-3444] Explicitly
set transfer operator description.
URL: https://github.com/apache/incubator-airflow/pull/4279#discussion_r241245947
##
File path: airflow/contrib/hooks/gcp_transfer_hook.py
##
@@ -56,34 +56,19 @@ def get_conn(self):
http=http_authorized, cache_discovery=False)
return self._conn
-def create_transfer_job(self, project_id, transfer_spec, **kwargs):
-conn = self.get_conn()
-now = datetime.datetime.utcnow()
+def create_transfer_job(self, project_id, description, schedule,
transfer_spec):
transfer_job = {
'status': 'ENABLED',
'projectId': project_id,
+'description': description,
'transferSpec': transfer_spec,
-'schedule': {
-'scheduleStartDate': {
-'day': now.day,
-'month': now.month,
-'year': now.year,
-},
-'scheduleEndDate': {
-'day': now.day,
-'month': now.month,
-'year': now.year,
-}
-}
+'schedule': schedule or self._schedule_once_now(),
}
-transfer_job.update(kwargs)
-result = conn.transferJobs().create(body=transfer_job).execute()
-self.wait_for_transfer_job(result, conn=conn)
+return
self.get_conn().transferJobs().create(body=transfer_job).execute()
-def wait_for_transfer_job(self, job, conn=None):
-conn = conn or self.get_conn()
+def wait_for_transfer_job(self, job):
Review comment:
It's used in `S3ToGoogleCloudStorageTransferOperator.execute`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-3411) create openfaas hook
[
https://issues.apache.org/jira/browse/AIRFLOW-3411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719625#comment-16719625
]
ASF GitHub Bot commented on AIRFLOW-3411:
-
kaxil closed pull request #4267: [AIRFLOW-3411] create openfaas hook
URL: https://github.com/apache/incubator-airflow/pull/4267
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/hooks/openfaas_hook.py
b/airflow/contrib/hooks/openfaas_hook.py
new file mode 100644
index 00..8f5062cbc7
--- /dev/null
+++ b/airflow/contrib/hooks/openfaas_hook.py
@@ -0,0 +1,99 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.hooks.base_hook import BaseHook
+import requests
+from airflow import AirflowException
+
+OK_STATUS_CODE = 202
+
+
+class OpenFaasHook(BaseHook):
+"""
+Interact with Openfaas to query, deploy, invoke and update function
+
+:param function_name: Name of the function, Defaults to None
+:type query: str
+:param conn_id: openfass connection to use, Defaults to open_faas_default
+for example host : http://openfaas.faas.com, Conn Type : Http
+:type conn_id: str
+"""
+
+GET_FUNCTION = "/system/function/"
+INVOKE_ASYNC_FUNCTION = "/async-function/"
+DEPLOY_FUNCTION = "/system/functions"
+UPDATE_FUNCTION = "/system/functions"
+
+def __init__(self,
+ function_name=None,
+ conn_id='open_faas_default',
+ *args, **kwargs):
+self.function_name = function_name
+self.conn_id = conn_id
+super(BaseHook, self).__init__(*args, **kwargs)
+
+def get_conn(self):
+conn = self.get_connection(self.conn_id)
+return conn
+
+def deploy_function(self, overwrite_function_if_exist, body):
+if overwrite_function_if_exist:
+self.log.info("Function already exist " + self.function_name + "
going to update")
+self.update_function(body)
+else:
+url = self.get_conn().host + self.DEPLOY_FUNCTION
+self.log.info("Deploying function " + url)
+response = requests.post(url, body)
+if (response.status_code != OK_STATUS_CODE):
+self.log.error("Response status " + str(response.status_code))
+self.log.error("Failed to deploy")
+raise AirflowException('failed to deploy')
+else:
+self.log.info("Function deployed " + self.function_name)
+
+def invoke_async_function(self, body):
+url = self.get_conn().host + self.INVOKE_ASYNC_FUNCTION +
self.function_name
+self.log.info("Invoking function " + url)
+response = requests.post(url, body)
+if (response.ok):
+self.log.info("Invoked " + self.function_name)
+else:
+self.log.error("Response status " + str(response.status_code))
+raise AirflowException('failed to invoke function')
+
+def update_function(self, body):
+url = self.get_conn().host + self.UPDATE_FUNCTION
+self.log.info("Updating function " + url)
+response = requests.put(url, body)
+if (response.status_code != OK_STATUS_CODE):
+self.log.error("Response status " + str(response.status_code))
+self.log.error("Failed to update response " +
response.content.decode("utf-8"))
+raise AirflowException('failed to update ' + self.function_name)
+else:
+self.log.info("Function was updated")
+
+def does_function_exist(self):
+url = self.get_conn().host + self.GET_FUNCTION + self.function_name
+
+response = requests.get(url)
+if (response.ok):
+return True
+else:
+self.log.error("Failed to find function " + self.function_name)
+return False
diff --git a/docs/code.rst b/docs/code.rst
index 968cdd600e..1fd7862b21 10064
[GitHub] kaxil closed pull request #4267: [AIRFLOW-3411] create openfaas hook
kaxil closed pull request #4267: [AIRFLOW-3411] create openfaas hook
URL: https://github.com/apache/incubator-airflow/pull/4267
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/hooks/openfaas_hook.py
b/airflow/contrib/hooks/openfaas_hook.py
new file mode 100644
index 00..8f5062cbc7
--- /dev/null
+++ b/airflow/contrib/hooks/openfaas_hook.py
@@ -0,0 +1,99 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.hooks.base_hook import BaseHook
+import requests
+from airflow import AirflowException
+
+OK_STATUS_CODE = 202
+
+
+class OpenFaasHook(BaseHook):
+"""
+Interact with Openfaas to query, deploy, invoke and update function
+
+:param function_name: Name of the function, Defaults to None
+:type query: str
+:param conn_id: openfass connection to use, Defaults to open_faas_default
+for example host : http://openfaas.faas.com, Conn Type : Http
+:type conn_id: str
+"""
+
+GET_FUNCTION = "/system/function/"
+INVOKE_ASYNC_FUNCTION = "/async-function/"
+DEPLOY_FUNCTION = "/system/functions"
+UPDATE_FUNCTION = "/system/functions"
+
+def __init__(self,
+ function_name=None,
+ conn_id='open_faas_default',
+ *args, **kwargs):
+self.function_name = function_name
+self.conn_id = conn_id
+super(BaseHook, self).__init__(*args, **kwargs)
+
+def get_conn(self):
+conn = self.get_connection(self.conn_id)
+return conn
+
+def deploy_function(self, overwrite_function_if_exist, body):
+if overwrite_function_if_exist:
+self.log.info("Function already exist " + self.function_name + "
going to update")
+self.update_function(body)
+else:
+url = self.get_conn().host + self.DEPLOY_FUNCTION
+self.log.info("Deploying function " + url)
+response = requests.post(url, body)
+if (response.status_code != OK_STATUS_CODE):
+self.log.error("Response status " + str(response.status_code))
+self.log.error("Failed to deploy")
+raise AirflowException('failed to deploy')
+else:
+self.log.info("Function deployed " + self.function_name)
+
+def invoke_async_function(self, body):
+url = self.get_conn().host + self.INVOKE_ASYNC_FUNCTION +
self.function_name
+self.log.info("Invoking function " + url)
+response = requests.post(url, body)
+if (response.ok):
+self.log.info("Invoked " + self.function_name)
+else:
+self.log.error("Response status " + str(response.status_code))
+raise AirflowException('failed to invoke function')
+
+def update_function(self, body):
+url = self.get_conn().host + self.UPDATE_FUNCTION
+self.log.info("Updating function " + url)
+response = requests.put(url, body)
+if (response.status_code != OK_STATUS_CODE):
+self.log.error("Response status " + str(response.status_code))
+self.log.error("Failed to update response " +
response.content.decode("utf-8"))
+raise AirflowException('failed to update ' + self.function_name)
+else:
+self.log.info("Function was updated")
+
+def does_function_exist(self):
+url = self.get_conn().host + self.GET_FUNCTION + self.function_name
+
+response = requests.get(url)
+if (response.ok):
+return True
+else:
+self.log.error("Failed to find function " + self.function_name)
+return False
diff --git a/docs/code.rst b/docs/code.rst
index 968cdd600e..1fd7862b21 100644
--- a/docs/code.rst
+++ b/docs/code.rst
@@ -442,6 +442,7 @@ Community contributed hooks
.. autoclass:: airflow.contrib.hooks.jenkins_hook.JenkinsHook
.. autoclass:: airflow.contrib.hooks.jira_hook.JiraHook
.. autoclass:: airflow.contrib.hooks.mongo_hook.Mong
[GitHub] XD-DENG commented on issue #4309: [AIRFLOW-3504] Extend/refine the functionality of "/health" endpoint
XD-DENG commented on issue #4309: [AIRFLOW-3504] Extend/refine the functionality of "/health" endpoint URL: https://github.com/apache/incubator-airflow/pull/4309#issuecomment-446807199 Sure. Thanks @kaxil . I believe some other checks can be added here to provide a more consolidated health glance. Kindly let me know if you have any good idea to further extend this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kaxil commented on a change in pull request #4279: [AIRFLOW-3444] Explicitly set transfer operator description.
kaxil commented on a change in pull request #4279: [AIRFLOW-3444] Explicitly
set transfer operator description.
URL: https://github.com/apache/incubator-airflow/pull/4279#discussion_r241244269
##
File path: airflow/contrib/hooks/gcp_transfer_hook.py
##
@@ -56,34 +56,19 @@ def get_conn(self):
http=http_authorized, cache_discovery=False)
return self._conn
-def create_transfer_job(self, project_id, transfer_spec, **kwargs):
-conn = self.get_conn()
-now = datetime.datetime.utcnow()
+def create_transfer_job(self, project_id, description, schedule,
transfer_spec):
transfer_job = {
'status': 'ENABLED',
'projectId': project_id,
+'description': description,
'transferSpec': transfer_spec,
-'schedule': {
-'scheduleStartDate': {
-'day': now.day,
-'month': now.month,
-'year': now.year,
-},
-'scheduleEndDate': {
-'day': now.day,
-'month': now.month,
-'year': now.year,
-}
-}
+'schedule': schedule or self._schedule_once_now(),
}
-transfer_job.update(kwargs)
-result = conn.transferJobs().create(body=transfer_job).execute()
-self.wait_for_transfer_job(result, conn=conn)
+return
self.get_conn().transferJobs().create(body=transfer_job).execute()
-def wait_for_transfer_job(self, job, conn=None):
-conn = conn or self.get_conn()
+def wait_for_transfer_job(self, job):
Review comment:
where do we use this method?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kaxil commented on issue #4309: [AIRFLOW-3504] Extend/refine the functionality of "/health" endpoint
kaxil commented on issue #4309: [AIRFLOW-3504] Extend/refine the functionality of "/health" endpoint URL: https://github.com/apache/incubator-airflow/pull/4309#issuecomment-446806491 This is a nice addition. Haven't reviewed it yet but will do when I have some good time. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-3310) Google Cloud Spanner deploy / delete operators
[ https://issues.apache.org/jira/browse/AIRFLOW-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-3310. - Resolution: Fixed Fix Version/s: 1.10.2 > Google Cloud Spanner deploy / delete operators > -- > > Key: AIRFLOW-3310 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3310 > Project: Apache Airflow > Issue Type: New Feature > Components: gcp >Reporter: Szymon Przedwojski >Assignee: Szymon Przedwojski >Priority: Minor > Fix For: 1.10.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3310) Google Cloud Spanner deploy / delete operators
[
https://issues.apache.org/jira/browse/AIRFLOW-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719615#comment-16719615
]
ASF GitHub Bot commented on AIRFLOW-3310:
-
kaxil closed pull request #4286: [AIRFLOW-3310] Google Cloud Spanner deploy /
delete operators
URL: https://github.com/apache/incubator-airflow/pull/4286
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/example_dags/example_gcp_spanner.py
b/airflow/contrib/example_dags/example_gcp_spanner.py
new file mode 100644
index 00..dd8b8c52b9
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcp_spanner.py
@@ -0,0 +1,92 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG that creates, updates and deletes a Cloud Spanner instance.
+
+This DAG relies on the following environment variables
+* PROJECT_ID - Google Cloud Platform project for the Cloud Spanner instance.
+* INSTANCE_ID - Cloud Spanner instance ID.
+* CONFIG_NAME - The name of the instance's configuration. Values are of the
form
+projects//instanceConfigs/.
+See also:
+
https://cloud.google.com/spanner/docs/reference/rest/v1/projects.instanceConfigs#InstanceConfig
+
https://cloud.google.com/spanner/docs/reference/rest/v1/projects.instanceConfigs/list#google.spanner.admin.instance.v1.InstanceAdmin.ListInstanceConfigs
+* NODE_COUNT - Number of nodes allocated to the instance.
+* DISPLAY_NAME - The descriptive name for this instance as it appears in UIs.
+Must be unique per project and between 4 and 30 characters in length.
+"""
+
+import os
+
+import airflow
+from airflow import models
+from airflow.contrib.operators.gcp_spanner_operator import \
+CloudSpannerInstanceDeployOperator, CloudSpannerInstanceDeleteOperator
+
+# [START howto_operator_spanner_arguments]
+PROJECT_ID = os.environ.get('PROJECT_ID', 'example-project')
+INSTANCE_ID = os.environ.get('INSTANCE_ID', 'testinstance')
+CONFIG_NAME = os.environ.get('CONFIG_NAME',
+ 'projects/example-project/instanceConfigs/eur3')
+NODE_COUNT = os.environ.get('NODE_COUNT', '1')
+DISPLAY_NAME = os.environ.get('DISPLAY_NAME', 'Test Instance')
+# [END howto_operator_spanner_arguments]
+
+default_args = {
+'start_date': airflow.utils.dates.days_ago(1)
+}
+
+with models.DAG(
+'example_gcp_spanner',
+default_args=default_args,
+schedule_interval=None # Override to match your needs
+) as dag:
+# Create
+# [START howto_operator_spanner_deploy]
+spanner_instance_create_task = CloudSpannerInstanceDeployOperator(
+project_id=PROJECT_ID,
+instance_id=INSTANCE_ID,
+configuration_name=CONFIG_NAME,
+node_count=int(NODE_COUNT),
+display_name=DISPLAY_NAME,
+task_id='spanner_instance_create_task'
+)
+# [END howto_operator_spanner_deploy]
+
+# Update
+spanner_instance_update_task = CloudSpannerInstanceDeployOperator(
+project_id=PROJECT_ID,
+instance_id=INSTANCE_ID,
+configuration_name=CONFIG_NAME,
+node_count=int(NODE_COUNT) + 1,
+display_name=DISPLAY_NAME + '_updated',
+task_id='spanner_instance_update_task'
+)
+
+# [START howto_operator_spanner_delete]
+spanner_instance_delete_task = CloudSpannerInstanceDeleteOperator(
+project_id=PROJECT_ID,
+instance_id=INSTANCE_ID,
+task_id='spanner_instance_delete_task'
+)
+# [END howto_operator_spanner_delete]
+
+spanner_instance_create_task >> spanner_instance_update_task \
+>> spanner_instance_delete_task
diff --git a/airflow/contrib/hooks/gcp_spanner_hook.py
b/airflow/contrib/hooks/gcp_spanner_hook.py
new file mode 100644
index 00..fc73562e8b
--- /dev/null
+++ b/airflow/contrib/hooks/gcp_spanner_hook.py
@@ -0,0 +1,183 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under on
[GitHub] kaxil closed pull request #4286: [AIRFLOW-3310] Google Cloud Spanner deploy / delete operators
kaxil closed pull request #4286: [AIRFLOW-3310] Google Cloud Spanner deploy /
delete operators
URL: https://github.com/apache/incubator-airflow/pull/4286
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/example_dags/example_gcp_spanner.py
b/airflow/contrib/example_dags/example_gcp_spanner.py
new file mode 100644
index 00..dd8b8c52b9
--- /dev/null
+++ b/airflow/contrib/example_dags/example_gcp_spanner.py
@@ -0,0 +1,92 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG that creates, updates and deletes a Cloud Spanner instance.
+
+This DAG relies on the following environment variables
+* PROJECT_ID - Google Cloud Platform project for the Cloud Spanner instance.
+* INSTANCE_ID - Cloud Spanner instance ID.
+* CONFIG_NAME - The name of the instance's configuration. Values are of the
form
+projects//instanceConfigs/.
+See also:
+
https://cloud.google.com/spanner/docs/reference/rest/v1/projects.instanceConfigs#InstanceConfig
+
https://cloud.google.com/spanner/docs/reference/rest/v1/projects.instanceConfigs/list#google.spanner.admin.instance.v1.InstanceAdmin.ListInstanceConfigs
+* NODE_COUNT - Number of nodes allocated to the instance.
+* DISPLAY_NAME - The descriptive name for this instance as it appears in UIs.
+Must be unique per project and between 4 and 30 characters in length.
+"""
+
+import os
+
+import airflow
+from airflow import models
+from airflow.contrib.operators.gcp_spanner_operator import \
+CloudSpannerInstanceDeployOperator, CloudSpannerInstanceDeleteOperator
+
+# [START howto_operator_spanner_arguments]
+PROJECT_ID = os.environ.get('PROJECT_ID', 'example-project')
+INSTANCE_ID = os.environ.get('INSTANCE_ID', 'testinstance')
+CONFIG_NAME = os.environ.get('CONFIG_NAME',
+ 'projects/example-project/instanceConfigs/eur3')
+NODE_COUNT = os.environ.get('NODE_COUNT', '1')
+DISPLAY_NAME = os.environ.get('DISPLAY_NAME', 'Test Instance')
+# [END howto_operator_spanner_arguments]
+
+default_args = {
+'start_date': airflow.utils.dates.days_ago(1)
+}
+
+with models.DAG(
+'example_gcp_spanner',
+default_args=default_args,
+schedule_interval=None # Override to match your needs
+) as dag:
+# Create
+# [START howto_operator_spanner_deploy]
+spanner_instance_create_task = CloudSpannerInstanceDeployOperator(
+project_id=PROJECT_ID,
+instance_id=INSTANCE_ID,
+configuration_name=CONFIG_NAME,
+node_count=int(NODE_COUNT),
+display_name=DISPLAY_NAME,
+task_id='spanner_instance_create_task'
+)
+# [END howto_operator_spanner_deploy]
+
+# Update
+spanner_instance_update_task = CloudSpannerInstanceDeployOperator(
+project_id=PROJECT_ID,
+instance_id=INSTANCE_ID,
+configuration_name=CONFIG_NAME,
+node_count=int(NODE_COUNT) + 1,
+display_name=DISPLAY_NAME + '_updated',
+task_id='spanner_instance_update_task'
+)
+
+# [START howto_operator_spanner_delete]
+spanner_instance_delete_task = CloudSpannerInstanceDeleteOperator(
+project_id=PROJECT_ID,
+instance_id=INSTANCE_ID,
+task_id='spanner_instance_delete_task'
+)
+# [END howto_operator_spanner_delete]
+
+spanner_instance_create_task >> spanner_instance_update_task \
+>> spanner_instance_delete_task
diff --git a/airflow/contrib/hooks/gcp_spanner_hook.py
b/airflow/contrib/hooks/gcp_spanner_hook.py
new file mode 100644
index 00..fc73562e8b
--- /dev/null
+++ b/airflow/contrib/hooks/gcp_spanner_hook.py
@@ -0,0 +1,183 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may no
[jira] [Reopened] (AIRFLOW-2770) kubernetes: add support for dag folder in the docker image
[ https://issues.apache.org/jira/browse/AIRFLOW-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik reopened AIRFLOW-2770: - > kubernetes: add support for dag folder in the docker image > -- > > Key: AIRFLOW-2770 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2770 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rurui Ye >Assignee: Rurui Ye >Priority: Critical > Fix For: 1.10.2 > > > currently the kube executor need to provider dag_volume_chain or git repo in > the config file, but if the user has build dag into their docker image, they > doesn't need to provider these two options, and they can manager their dag > version by manager the docker image version. > So I suppose we can add the a new configuration as > kube.config.dag_folder_path along with dag_volume_chain and git repo. with > this config, we can run the worker just from the dags in docker image. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] kaxil commented on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i…
kaxil commented on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i… URL: https://github.com/apache/incubator-airflow/pull/3683#issuecomment-446804832 @dimberman Definitely, I have added 1.10.2 tag on Jira. Will cherry-pick this on weekends as it has transitive dependencies on other commits This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-2770) kubernetes: add support for dag folder in the docker image
[ https://issues.apache.org/jira/browse/AIRFLOW-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik resolved AIRFLOW-2770. - Resolution: Fixed Fix Version/s: 1.10.2 > kubernetes: add support for dag folder in the docker image > -- > > Key: AIRFLOW-2770 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2770 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rurui Ye >Assignee: Rurui Ye >Priority: Critical > Fix For: 1.10.2 > > > currently the kube executor need to provider dag_volume_chain or git repo in > the config file, but if the user has build dag into their docker image, they > doesn't need to provider these two options, and they can manager their dag > version by manager the docker image version. > So I suppose we can add the a new configuration as > kube.config.dag_folder_path along with dag_volume_chain and git repo. with > this config, we can run the worker just from the dags in docker image. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] jzucker2 commented on issue #3770: [AIRFLOW-3281] Fix Kubernetes operator with git-sync
jzucker2 commented on issue #3770: [AIRFLOW-3281] Fix Kubernetes operator with git-sync URL: https://github.com/apache/incubator-airflow/pull/3770#issuecomment-446802767 @odracci there are some merge conflicts in here FYI This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] megaserg commented on issue #2740: [AIRFLOW-1768] add if to show trigger only if not paused
megaserg commented on issue #2740: [AIRFLOW-1768] add if to show trigger only if not paused URL: https://github.com/apache/incubator-airflow/pull/2740#issuecomment-446786983 Still an issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] mdziemianko commented on issue #2450: [Airflow-1413] Fix FTPSensor failing on error message with unexpected text.
mdziemianko commented on issue #2450: [Airflow-1413] Fix FTPSensor failing on error message with unexpected text. URL: https://github.com/apache/incubator-airflow/pull/2450#issuecomment-446781725 > (Better late than never with a review right? Sorry it took so long!) lol:) I forgot about this PR... will do the changes. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] odracci commented on issue #3770: [AIRFLOW-3281] Fix Kubernetes operator with git-sync
odracci commented on issue #3770: [AIRFLOW-3281] Fix Kubernetes operator with git-sync URL: https://github.com/apache/incubator-airflow/pull/3770#issuecomment-446781808 @BrechtDeVlieger thanks a lot! I just fixed the code and pushed. Let's see what travis will say This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3506) Fail to query log from elasticsearch
[ https://issues.apache.org/jira/browse/AIRFLOW-3506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ping Zhang updated AIRFLOW-3506: Description: We have noticed that the elasticsearch_logging_backend use match to query the log_id from elasticsearch, which ends up getting more results from elasticsearch. It should use match_phrase to query the log_id as a phrase. (was: We have noticed that the elasticsearch_logging_backend use match to query the log_id from elasticsearch, which ended up getting more results from elasticsearch. It should use match_phrase to query the log_id as a phrase.) > Fail to query log from elasticsearch > > > Key: AIRFLOW-3506 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3506 > Project: Apache Airflow > Issue Type: Bug > Components: logging >Reporter: Ping Zhang >Assignee: Ping Zhang >Priority: Major > Labels: easyfix, newbie > > We have noticed that the elasticsearch_logging_backend use match to query the > log_id from elasticsearch, which ends up getting more results from > elasticsearch. It should use match_phrase to query the log_id as a phrase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3506) Fail to query log from elasticsearch
Ping Zhang created AIRFLOW-3506: --- Summary: Fail to query log from elasticsearch Key: AIRFLOW-3506 URL: https://issues.apache.org/jira/browse/AIRFLOW-3506 Project: Apache Airflow Issue Type: Bug Components: logging Reporter: Ping Zhang Assignee: Ping Zhang We have noticed that the elasticsearch_logging_backend use match to query the log_id from elasticsearch, which ended up getting more results from elasticsearch. It should use match_phrase to query the log_id as a phrase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] BrechtDeVlieger commented on issue #3770: [AIRFLOW-3281] Fix Kubernetes operator with git-sync
BrechtDeVlieger commented on issue #3770: [AIRFLOW-3281] Fix Kubernetes operator with git-sync URL: https://github.com/apache/incubator-airflow/pull/3770#issuecomment-446736668 @odracci I tried to test your branch manually (I couldn't find a quick way to run the integration tests locally) and I found that you used an unexisting directory `example_dags_kubernetes` in the ci script. Therefore no dags appeared on in the ui. When I changed this to `example_dags` (had to change this in a couple of files) then the example dags appeared in the ui and could be scheduled successfully. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Valkrysa commented on issue #2398: [AIRFLOW-450] Remove http example operator
Valkrysa commented on issue #2398: [AIRFLOW-450] Remove http example operator URL: https://github.com/apache/incubator-airflow/pull/2398#issuecomment-446714403 Run into the issue today that a fresh installation of python3 cannot successfully init the db in airflow as long as this example-http-operator is here. I solved the issue by manually removing the example-http-operator, however I believe a user following the natural installation procedures should not have to do this as they might not always understand it's okay to just delete the file. My suggestion is that example-http-operator either needs to be removed from the repo or fixed so that it does not cause the entire init db script to fail. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] BrechtDeVlieger commented on issue #3770: [AIRFLOW-3281] Fix Kubernetes operator with git-sync
BrechtDeVlieger commented on issue #3770: [AIRFLOW-3281] Fix Kubernetes operator with git-sync URL: https://github.com/apache/incubator-airflow/pull/3770#issuecomment-446710067 I'm not sure if it's related to this issue, but a few weeks ago git-sync v3.0.1 was released where the permission requirements of 'runAsUser=0' are no longer required. It solved some of the discovery issues for me, as I don't run airflow as root. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3505) Change name of 'dags_in_docker' field to 'dags
[ https://issues.apache.org/jira/browse/AIRFLOW-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman updated AIRFLOW-3505: - Summary: Change name of 'dags_in_docker' field to 'dags (was: Change name of 'image) > Change name of 'dags_in_docker' field to 'dags > -- > > Key: AIRFLOW-3505 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3505 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Daniel Imberman >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3505) Change name of 'dags_in_docker' field to 'dags_in_image'
[
https://issues.apache.org/jira/browse/AIRFLOW-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719304#comment-16719304
]
ASF GitHub Bot commented on AIRFLOW-3505:
-
dimberman opened a new pull request #4311: [AIRFLOW-3505] replace
'dags_in_docker' with 'dags_in_image'
URL: https://github.com/apache/incubator-airflow/pull/4311
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3505
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
As kubernetes is moving away from docker to OCI, it will be more correct to
use the 'dags_in_image' name to be more container system agnostic
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Change name of 'dags_in_docker' field to 'dags_in_image'
>
>
> Key: AIRFLOW-3505
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3505
> Project: Apache Airflow
> Issue Type: Improvement
>Reporter: Daniel Imberman
>Priority: Minor
>
> As kubernetes is moving away from docker to OCI, it will be more correct to
> use the 'dags_in_image' name to be more container system agnostic
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] dimberman opened a new pull request #4311: [AIRFLOW-3505] replace 'dags_in_docker' with 'dags_in_image'
dimberman opened a new pull request #4311: [AIRFLOW-3505] replace
'dags_in_docker' with 'dags_in_image'
URL: https://github.com/apache/incubator-airflow/pull/4311
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3505
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
As kubernetes is moving away from docker to OCI, it will be more correct to
use the 'dags_in_image' name to be more container system agnostic
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-3505) Change name of 'dags_in_docker' field to 'dags_in_image'
[ https://issues.apache.org/jira/browse/AIRFLOW-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman updated AIRFLOW-3505: - Description: As kubernetes is moving away from docker to OCI, it will be more correct to use the 'dags_in_image' name to be more container system agnostic > Change name of 'dags_in_docker' field to 'dags_in_image' > > > Key: AIRFLOW-3505 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3505 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Daniel Imberman >Priority: Minor > > As kubernetes is moving away from docker to OCI, it will be more correct to > use the 'dags_in_image' name to be more container system agnostic -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3505) Change name of 'image
Daniel Imberman created AIRFLOW-3505: Summary: Change name of 'image Key: AIRFLOW-3505 URL: https://issues.apache.org/jira/browse/AIRFLOW-3505 Project: Apache Airflow Issue Type: Improvement Reporter: Daniel Imberman -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3505) Change name of 'dags_in_docker' field to 'dags_in_image'
[ https://issues.apache.org/jira/browse/AIRFLOW-3505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman updated AIRFLOW-3505: - Summary: Change name of 'dags_in_docker' field to 'dags_in_image' (was: Change name of 'dags_in_docker' field to 'dags) > Change name of 'dags_in_docker' field to 'dags_in_image' > > > Key: AIRFLOW-3505 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3505 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Daniel Imberman >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3223) RBAC with GitHub Authentication
[ https://issues.apache.org/jira/browse/AIRFLOW-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Phanindhra reassigned AIRFLOW-3223: --- Assignee: Sai Phanindhra > RBAC with GitHub Authentication > > > Key: AIRFLOW-3223 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3223 > Project: Apache Airflow > Issue Type: Wish > Components: authentication >Affects Versions: 1.10.0 >Reporter: Vikram Fugro >Assignee: Sai Phanindhra >Priority: Major > > With airflow 1.10 released having RBAC support, I was wondering how I do > configure GitHub Auth with airflow's RBAC. In which case, I believe we don't > have to create any users using airflow. Are there any notes on this? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-2770) kubernetes: add support for dag folder in the docker image
[ https://issues.apache.org/jira/browse/AIRFLOW-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng closed AIRFLOW-2770. - Resolution: Fixed > kubernetes: add support for dag folder in the docker image > -- > > Key: AIRFLOW-2770 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2770 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Rurui Ye >Assignee: Rurui Ye >Priority: Critical > > currently the kube executor need to provider dag_volume_chain or git repo in > the config file, but if the user has build dag into their docker image, they > doesn't need to provider these two options, and they can manager their dag > version by manager the docker image version. > So I suppose we can add the a new configuration as > kube.config.dag_folder_path along with dag_volume_chain and git repo. with > this config, we can run the worker just from the dags in docker image. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3502) Add config option to control celery pool used
[ https://issues.apache.org/jira/browse/AIRFLOW-3502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Feng closed AIRFLOW-3502. - Resolution: Fixed Assignee: Gabriel Silk > Add config option to control celery pool used > - > > Key: AIRFLOW-3502 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3502 > Project: Apache Airflow > Issue Type: Improvement > Components: celery >Reporter: Gabriel Silk >Assignee: Gabriel Silk >Priority: Minor > > This adds a config option for the "pool" to allow uses to specify "prefork" > vs "solo" etc. This is particularly useful with infrastructures that don't > play nicely with celery's default prefork multi-processing model. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3502) Add config option to control celery pool used
[
https://issues.apache.org/jira/browse/AIRFLOW-3502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719266#comment-16719266
]
ASF GitHub Bot commented on AIRFLOW-3502:
-
feng-tao closed pull request #4308: [AIRFLOW-3502] Add celery config option for
setting "pool"
URL: https://github.com/apache/incubator-airflow/pull/4308
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index f2342ee148..cd414d2821 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -1052,6 +1052,9 @@ def worker(args):
'loglevel': conf.get('core', 'LOGGING_LEVEL'),
}
+if conf.has_option("celery", "pool"):
+options["pool"] = conf.get("celery", "pool")
+
if args.daemon:
pid, stdout, stderr, log_file = setup_locations("worker",
args.pid,
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add config option to control celery pool used
> -
>
> Key: AIRFLOW-3502
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3502
> Project: Apache Airflow
> Issue Type: Improvement
> Components: celery
>Reporter: Gabriel Silk
>Priority: Minor
>
> This adds a config option for the "pool" to allow uses to specify "prefork"
> vs "solo" etc. This is particularly useful with infrastructures that don't
> play nicely with celery's default prefork multi-processing model.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] feng-tao closed pull request #4308: [AIRFLOW-3502] Add celery config option for setting "pool"
feng-tao closed pull request #4308: [AIRFLOW-3502] Add celery config option for
setting "pool"
URL: https://github.com/apache/incubator-airflow/pull/4308
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index f2342ee148..cd414d2821 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -1052,6 +1052,9 @@ def worker(args):
'loglevel': conf.get('core', 'LOGGING_LEVEL'),
}
+if conf.has_option("celery", "pool"):
+options["pool"] = conf.get("celery", "pool")
+
if args.daemon:
pid, stdout, stderr, log_file = setup_locations("worker",
args.pid,
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] feng-tao commented on issue #4295: AIRFLOW-3452 removed an unused/dangerous display-none
feng-tao commented on issue #4295: AIRFLOW-3452 removed an unused/dangerous display-none URL: https://github.com/apache/incubator-airflow/pull/4295#issuecomment-446682155 👀 @jgao54 , not sure if Joy has any context of this issue. If not, we could merge this pr. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4308: [AIRFLOW-3502] Add celery config option for setting "pool"
codecov-io edited a comment on issue #4308: [AIRFLOW-3502] Add celery config option for setting "pool" URL: https://github.com/apache/incubator-airflow/pull/4308#issuecomment-446681483 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4308?src=pr&el=h1) Report > Merging [#4308](https://codecov.io/gh/apache/incubator-airflow/pull/4308?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/57e355c516b6deaa97ca7ac7ec83e5fc0f50ab87?src=pr&el=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4308?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4308 +/- ## == - Coverage78.1% 78.09% -0.01% == Files 201 201 Lines 1646416466 +2 == Hits1285912859 - Misses 3605 3607 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4308?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/4308/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `64.43% <0%> (-0.16%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4308?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4308?src=pr&el=footer). Last update [57e355c...4245ce6](https://codecov.io/gh/apache/incubator-airflow/pull/4308?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4308: [AIRFLOW-3502] Add celery config option for setting "pool"
codecov-io commented on issue #4308: [AIRFLOW-3502] Add celery config option for setting "pool" URL: https://github.com/apache/incubator-airflow/pull/4308#issuecomment-446681483 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4308?src=pr&el=h1) Report > Merging [#4308](https://codecov.io/gh/apache/incubator-airflow/pull/4308?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/57e355c516b6deaa97ca7ac7ec83e5fc0f50ab87?src=pr&el=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4308?src=pr&el=tree) ```diff @@Coverage Diff @@ ## master#4308 +/- ## == - Coverage78.1% 78.09% -0.01% == Files 201 201 Lines 1646416466 +2 == Hits1285912859 - Misses 3605 3607 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4308?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/4308/diff?src=pr&el=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `64.43% <0%> (-0.16%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4308?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4308?src=pr&el=footer). Last update [57e355c...4245ce6](https://codecov.io/gh/apache/incubator-airflow/pull/4308?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2770) kubernetes: add support for dag folder in the docker image
[
https://issues.apache.org/jira/browse/AIRFLOW-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719255#comment-16719255
]
ASF GitHub Bot commented on AIRFLOW-2770:
-
feng-tao closed pull request #3683: [AIRFLOW-2770] kubernetes: add support for
dag folder in the docker i…
URL: https://github.com/apache/incubator-airflow/pull/3683
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/config_templates/default_airflow.cfg
b/airflow/config_templates/default_airflow.cfg
index 828db54a64..35a8809690 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -572,6 +572,10 @@ namespace = default
# The name of the Kubernetes ConfigMap Containing the Airflow Configuration
(this file)
airflow_configmap =
+# For docker image already contains DAGs, this is set to `True`, and the
worker will search for dags in dags_folder,
+# otherwise use git sync or dags volumn chaim to mount DAGs
+dags_in_docker = FALSE
+
# For either git sync or volume mounted DAGs, the worker will look in this
subpath for DAGs
dags_volume_subpath =
diff --git a/airflow/contrib/executors/kubernetes_executor.py
b/airflow/contrib/executors/kubernetes_executor.py
index 788d925c38..a1df83582e 100644
--- a/airflow/contrib/executors/kubernetes_executor.py
+++ b/airflow/contrib/executors/kubernetes_executor.py
@@ -128,6 +128,10 @@ def __init__(self):
self.kubernetes_section, 'worker_service_account_name')
self.image_pull_secrets = conf.get(self.kubernetes_section,
'image_pull_secrets')
+# NOTE: user can build the dags into the docker image directly,
+# this will set to True if so
+self.dags_in_docker = conf.get(self.kubernetes_section,
'dags_in_docker')
+
# NOTE: `git_repo` and `git_branch` must be specified together as a
pair
# The http URL of the git repository to clone from
self.git_repo = conf.get(self.kubernetes_section, 'git_repo')
@@ -195,10 +199,12 @@ def __init__(self):
self._validate()
def _validate(self):
-if not self.dags_volume_claim and (not self.git_repo or not
self.git_branch):
+if not self.dags_volume_claim and not self.dags_in_docker \
+and (not self.git_repo or not self.git_branch):
raise AirflowConfigException(
'In kubernetes mode the following must be set in the
`kubernetes` '
-'config section: `dags_volume_claim` or `git_repo and
git_branch`')
+'config section: `dags_volume_claim` or `git_repo and
git_branch` '
+'or `dags_in_docker`')
class KubernetesJobWatcher(multiprocessing.Process, LoggingMixin, object):
diff --git a/airflow/contrib/kubernetes/worker_configuration.py
b/airflow/contrib/kubernetes/worker_configuration.py
index c9f86b047a..febae8ccb6 100644
--- a/airflow/contrib/kubernetes/worker_configuration.py
+++ b/airflow/contrib/kubernetes/worker_configuration.py
@@ -38,7 +38,7 @@ def __init__(self, kube_config):
def _get_init_containers(self, volume_mounts):
"""When using git to retrieve the DAGs, use the GitSync Init
Container"""
# If we're using volume claims to mount the dags, no init container is
needed
-if self.kube_config.dags_volume_claim:
+if self.kube_config.dags_volume_claim or
self.kube_config.dags_in_docker:
return []
# Otherwise, define a git-sync init container
@@ -121,32 +121,19 @@ def _construct_volume(name, claim):
return volume
volumes = [
-_construct_volume(
-dags_volume_name,
-self.kube_config.dags_volume_claim
-),
_construct_volume(
logs_volume_name,
self.kube_config.logs_volume_claim
)
]
-dag_volume_mount_path = ""
-
-if self.kube_config.dags_volume_claim:
-dag_volume_mount_path = self.worker_airflow_dags
-else:
-dag_volume_mount_path = os.path.join(
-self.worker_airflow_dags,
-self.kube_config.git_subpath
+if not self.kube_config.dags_in_docker:
+volumes.append(
+_construct_volume(
+dags_volume_name,
+self.kube_config.dags_volume_claim
+)
)
-dags_volume_mount = {
-'name': dags_volume_name,
-'mountPath': dag_volume_mount_path,
-'readOnly': True,
-}
-if self.kube_config.dags_volume_subpath:
-dags_volume_mount['subPath'] = self.kub
[GitHub] feng-tao commented on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i…
feng-tao commented on issue #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i… URL: https://github.com/apache/incubator-airflow/pull/3683#issuecomment-446677558 thanks. I trust @dimberman's review on this k8s pr. Thanks @yeluolei . This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao closed pull request #3683: [AIRFLOW-2770] kubernetes: add support for dag folder in the docker i…
feng-tao closed pull request #3683: [AIRFLOW-2770] kubernetes: add support for
dag folder in the docker i…
URL: https://github.com/apache/incubator-airflow/pull/3683
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/config_templates/default_airflow.cfg
b/airflow/config_templates/default_airflow.cfg
index 828db54a64..35a8809690 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -572,6 +572,10 @@ namespace = default
# The name of the Kubernetes ConfigMap Containing the Airflow Configuration
(this file)
airflow_configmap =
+# For docker image already contains DAGs, this is set to `True`, and the
worker will search for dags in dags_folder,
+# otherwise use git sync or dags volumn chaim to mount DAGs
+dags_in_docker = FALSE
+
# For either git sync or volume mounted DAGs, the worker will look in this
subpath for DAGs
dags_volume_subpath =
diff --git a/airflow/contrib/executors/kubernetes_executor.py
b/airflow/contrib/executors/kubernetes_executor.py
index 788d925c38..a1df83582e 100644
--- a/airflow/contrib/executors/kubernetes_executor.py
+++ b/airflow/contrib/executors/kubernetes_executor.py
@@ -128,6 +128,10 @@ def __init__(self):
self.kubernetes_section, 'worker_service_account_name')
self.image_pull_secrets = conf.get(self.kubernetes_section,
'image_pull_secrets')
+# NOTE: user can build the dags into the docker image directly,
+# this will set to True if so
+self.dags_in_docker = conf.get(self.kubernetes_section,
'dags_in_docker')
+
# NOTE: `git_repo` and `git_branch` must be specified together as a
pair
# The http URL of the git repository to clone from
self.git_repo = conf.get(self.kubernetes_section, 'git_repo')
@@ -195,10 +199,12 @@ def __init__(self):
self._validate()
def _validate(self):
-if not self.dags_volume_claim and (not self.git_repo or not
self.git_branch):
+if not self.dags_volume_claim and not self.dags_in_docker \
+and (not self.git_repo or not self.git_branch):
raise AirflowConfigException(
'In kubernetes mode the following must be set in the
`kubernetes` '
-'config section: `dags_volume_claim` or `git_repo and
git_branch`')
+'config section: `dags_volume_claim` or `git_repo and
git_branch` '
+'or `dags_in_docker`')
class KubernetesJobWatcher(multiprocessing.Process, LoggingMixin, object):
diff --git a/airflow/contrib/kubernetes/worker_configuration.py
b/airflow/contrib/kubernetes/worker_configuration.py
index c9f86b047a..febae8ccb6 100644
--- a/airflow/contrib/kubernetes/worker_configuration.py
+++ b/airflow/contrib/kubernetes/worker_configuration.py
@@ -38,7 +38,7 @@ def __init__(self, kube_config):
def _get_init_containers(self, volume_mounts):
"""When using git to retrieve the DAGs, use the GitSync Init
Container"""
# If we're using volume claims to mount the dags, no init container is
needed
-if self.kube_config.dags_volume_claim:
+if self.kube_config.dags_volume_claim or
self.kube_config.dags_in_docker:
return []
# Otherwise, define a git-sync init container
@@ -121,32 +121,19 @@ def _construct_volume(name, claim):
return volume
volumes = [
-_construct_volume(
-dags_volume_name,
-self.kube_config.dags_volume_claim
-),
_construct_volume(
logs_volume_name,
self.kube_config.logs_volume_claim
)
]
-dag_volume_mount_path = ""
-
-if self.kube_config.dags_volume_claim:
-dag_volume_mount_path = self.worker_airflow_dags
-else:
-dag_volume_mount_path = os.path.join(
-self.worker_airflow_dags,
-self.kube_config.git_subpath
+if not self.kube_config.dags_in_docker:
+volumes.append(
+_construct_volume(
+dags_volume_name,
+self.kube_config.dags_volume_claim
+)
)
-dags_volume_mount = {
-'name': dags_volume_name,
-'mountPath': dag_volume_mount_path,
-'readOnly': True,
-}
-if self.kube_config.dags_volume_subpath:
-dags_volume_mount['subPath'] = self.kube_config.dags_volume_subpath
logs_volume_mount = {
'name': logs_volume_name,
@@ -156,10 +143,28 @@ def _construct_volume(name, claim):
logs_volume_mount['subPath'] = self.kube_config.logs_volume_subpath
volume_mount
[jira] [Commented] (AIRFLOW-1536) DaemonContext uses default umask 0
[ https://issues.apache.org/jira/browse/AIRFLOW-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719251#comment-16719251 ] jack commented on AIRFLOW-1536: --- this could be a venerability though I'm not sure it's urgent to fix. > DaemonContext uses default umask 0 > -- > > Key: AIRFLOW-1536 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1536 > Project: Apache Airflow > Issue Type: Bug > Components: cli, security >Reporter: Timothy O'Keefe >Priority: Major > > All DaemonContext instances used for worker, scheduler, webserver, flower, > etc. do not supply a umask argument. See here for example: > https://github.com/apache/incubator-airflow/blob/b0669b532a7be9aa34a4390951deaa25897c62e6/airflow/bin/cli.py#L869 > As a result, the DaemonContext will use the default umask=0 which leaves user > data exposed. A BashOperator for example that writes any files would have > permissions rw-rw-rw- as would any airflow logs. > I believe the umask should either be configurable, or inherited from the > parent shell, or both. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-3210) Changing defaults types in BigQuery Hook break BigQuery operator
[ https://issues.apache.org/jira/browse/AIRFLOW-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-3210. Resolution: Fixed Fix Version/s: 2.0.0 > Changing defaults types in BigQuery Hook break BigQuery operator > > > Key: AIRFLOW-3210 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3210 > Project: Apache Airflow > Issue Type: Bug > Components: contrib, gcp >Reporter: Sergei Guschin >Priority: Major > Fix For: 2.0.0 > > > Changes in BigQuery Hook break BigQuery operator run_query() and all DAGs > which accommodate current type (Boolean or value): > [BigQuery operator > set|https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/bigquery_operator.py#L115-L121]: > destination_dataset_table=False, > udf_config=False, > [New BigQuery hook > expects|https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/bigquery_hook.py#L645-L650]: > (udf_config, 'userDefinedFunctionResources', None, list), > (destination_dataset_table, 'destinationTable', None, dict), -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao commented on issue #4308: [AIRFLOW-3502] Add celery config option for setting "pool"
feng-tao commented on issue #4308: [AIRFLOW-3502] Add celery config option for setting "pool" URL: https://github.com/apache/incubator-airflow/pull/4308#issuecomment-446675451 lgtm, thanks @gsilk . will commit once ci is clear. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-352) filter_by_owner is not working when use ldap authentication
[
https://issues.apache.org/jira/browse/AIRFLOW-352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor resolved AIRFLOW-352.
---
Resolution: Duplicate
> filter_by_owner is not working when use ldap authentication
> ---
>
> Key: AIRFLOW-352
> URL: https://issues.apache.org/jira/browse/AIRFLOW-352
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib, security, webserver
>Affects Versions: 1.7.1.3
> Environment: ubuntu 14.04 LTS , ldap without encryption
>Reporter: peter pang
>Priority: Major
> Labels: security
>
> I set airflow.cfg as follows:
> {noformat}
> [webserver]
> filter_by_owner = True
> authenticate = TRUE
> auth_backend = airflow.contrib.auth.backends.ldap_auth
> [ldap]
> uri = ldap://xx.xx.xx.xx
> user_filter = objectClass=*
> user_name_attr = uid
> superuser_filter =
> memberOf=CN=airflow-super-users,OU=Groups,OU=RWC,OU=US,OU=NORAM,DC=example,DC=com
> data_profiler_filter =
> memberOf=CN=airflow-data-profilers,OU=Groups,OU=RWC,OU=US,OU=NORAM,DC=example,DC=com
> bind_user = cn=admin,dc=example,dc=com
> bind_password = secret
> basedn = dc=example,dc=com
> cacert = /etc/ca/ldap_ca.crt
> search_scope=SUBTREE
> {noformat}
> then I run the webUI , and I can login with superuser and data_profiler user.
> But after login with data profiler user, entered the data profiler user home
> view , there's no dags listed with the same dag owner. It seems the
> filter_by_owner setting is not working.
> Debug into the views.py --> class HomeView(AdminIndexView):
> {color:red}current_user.username{color} always get{color:red} "None"{color}.
> It seems we can't get username directly.
> so , continue debug into the ldap_auth.py --> class LdapUser(models.User):
> I added a method to return username
> {code}
> def get_username(self):
> return self.user.username
> {code}
> then back to view.py , replace 'current_user.username' to
> {color:red}'current_user.get_username()'{color} , the user filter can work
> now!
> I don't know exactly why, but the modification can work...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-353) when multiple tasks removed update state fails
[ https://issues.apache.org/jira/browse/AIRFLOW-353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-353. --- Resolution: Fixed Fix Version/s: 1.8.0 > when multiple tasks removed update state fails > -- > > Key: AIRFLOW-353 > URL: https://issues.apache.org/jira/browse/AIRFLOW-353 > Project: Apache Airflow > Issue Type: Bug > Components: core >Reporter: Yiqing Jin >Assignee: Yiqing Jin >Priority: Major > Fix For: 1.8.0 > > > if multiple tasks gets removed during dag run update_state may not work > properly -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] feng-tao closed pull request #4310: [AIRFLOW-XXX] add ARMEDANGELS to the list of customers
feng-tao closed pull request #4310: [AIRFLOW-XXX] add ARMEDANGELS to the list of customers URL: https://github.com/apache/incubator-airflow/pull/4310 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/README.md b/README.md index 97431d03dd..ceb652e9fe 100644 --- a/README.md +++ b/README.md @@ -117,6 +117,7 @@ Currently **officially** using Airflow: 1. [AltX](https://www.getaltx.com/about) [[@pedromduarte](https://github.com/pedromduarte)] 1. [Apigee](https://apigee.com) [[@btallman](https://github.com/btallman)] 1. [ARGO Labs](http://www.argolabs.org) [[California Data Collaborative](https://github.com/California-Data-Collaborative)] +1. [ARMEDANGELS](https://www.armedangels.de) [[@swiffer](https://github.com/swiffer)] 1. [Arquivei](https://www.arquivei.com.br/) [[@arquivei](https://github.com/arquivei)] 1. [Astronomer](http://www.astronomer.io) [[@schnie](https://github.com/schnie), [@andscoop](https://github.com/andscoop), [@tedmiston](https://github.com/tedmiston), [@benjamingregory](https://github.com/benjamingregory)] 1. [Auth0](https://auth0.com) [[@sicarul](https://github.com/sicarul)] This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on issue #4310: [AIRFLOW-XXX] add ARMEDANGELS to the list of customers
feng-tao commented on issue #4310: [AIRFLOW-XXX] add ARMEDANGELS to the list of customers URL: https://github.com/apache/incubator-airflow/pull/4310#issuecomment-446673360 lgtm This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] feng-tao commented on issue #4310: [AIRFLOW-XXX] add ARMEDANGELS to the list of customers
feng-tao commented on issue #4310: [AIRFLOW-XXX] add ARMEDANGELS to the list of customers URL: https://github.com/apache/incubator-airflow/pull/4310#issuecomment-446673611 thanks @swiffer, next week we could use [ci-skip] to skip ci as this doesn't require go through ci. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3210) Changing defaults types in BigQuery Hook break BigQuery operator
[ https://issues.apache.org/jira/browse/AIRFLOW-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719224#comment-16719224 ] jack commented on AIRFLOW-3210: --- I think it was solved by https://github.com/apache/incubator-airflow/pull/4274 > Changing defaults types in BigQuery Hook break BigQuery operator > > > Key: AIRFLOW-3210 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3210 > Project: Apache Airflow > Issue Type: Bug > Components: contrib, gcp >Reporter: Sergei Guschin >Priority: Major > > Changes in BigQuery Hook break BigQuery operator run_query() and all DAGs > which accommodate current type (Boolean or value): > [BigQuery operator > set|https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/bigquery_operator.py#L115-L121]: > destination_dataset_table=False, > udf_config=False, > [New BigQuery hook > expects|https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/bigquery_hook.py#L645-L650]: > (udf_config, 'userDefinedFunctionResources', None, list), > (destination_dataset_table, 'destinationTable', None, dict), -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-353) when multiple tasks removed update state fails
[ https://issues.apache.org/jira/browse/AIRFLOW-353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719230#comment-16719230 ] jack commented on AIRFLOW-353: -- Did [https://github.com/apache/incubator-airflow/pull/1675] resolve this issue? > when multiple tasks removed update state fails > -- > > Key: AIRFLOW-353 > URL: https://issues.apache.org/jira/browse/AIRFLOW-353 > Project: Apache Airflow > Issue Type: Bug > Components: core >Reporter: Yiqing Jin >Assignee: Yiqing Jin >Priority: Major > > if multiple tasks gets removed during dag run update_state may not work > properly -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-352) filter_by_owner is not working when use ldap authentication
[
https://issues.apache.org/jira/browse/AIRFLOW-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719226#comment-16719226
]
jack commented on AIRFLOW-352:
--
Duplicate of https://issues.apache.org/jira/browse/AIRFLOW-1552
> filter_by_owner is not working when use ldap authentication
> ---
>
> Key: AIRFLOW-352
> URL: https://issues.apache.org/jira/browse/AIRFLOW-352
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib, security, webserver
>Affects Versions: 1.7.1.3
> Environment: ubuntu 14.04 LTS , ldap without encryption
>Reporter: peter pang
>Priority: Major
> Labels: security
>
> I set airflow.cfg as follows:
> {noformat}
> [webserver]
> filter_by_owner = True
> authenticate = TRUE
> auth_backend = airflow.contrib.auth.backends.ldap_auth
> [ldap]
> uri = ldap://xx.xx.xx.xx
> user_filter = objectClass=*
> user_name_attr = uid
> superuser_filter =
> memberOf=CN=airflow-super-users,OU=Groups,OU=RWC,OU=US,OU=NORAM,DC=example,DC=com
> data_profiler_filter =
> memberOf=CN=airflow-data-profilers,OU=Groups,OU=RWC,OU=US,OU=NORAM,DC=example,DC=com
> bind_user = cn=admin,dc=example,dc=com
> bind_password = secret
> basedn = dc=example,dc=com
> cacert = /etc/ca/ldap_ca.crt
> search_scope=SUBTREE
> {noformat}
> then I run the webUI , and I can login with superuser and data_profiler user.
> But after login with data profiler user, entered the data profiler user home
> view , there's no dags listed with the same dag owner. It seems the
> filter_by_owner setting is not working.
> Debug into the views.py --> class HomeView(AdminIndexView):
> {color:red}current_user.username{color} always get{color:red} "None"{color}.
> It seems we can't get username directly.
> so , continue debug into the ldap_auth.py --> class LdapUser(models.User):
> I added a method to return username
> {code}
> def get_username(self):
> return self.user.username
> {code}
> then back to view.py , replace 'current_user.username' to
> {color:red}'current_user.get_username()'{color} , the user filter can work
> now!
> I don't know exactly why, but the modification can work...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-691) Add SSH keepalive option to ssh_hook
[ https://issues.apache.org/jira/browse/AIRFLOW-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-691. --- Resolution: Fixed Fix Version/s: 1.8.0 > Add SSH keepalive option to ssh_hook > > > Key: AIRFLOW-691 > URL: https://issues.apache.org/jira/browse/AIRFLOW-691 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, hooks >Reporter: Daniel van der Ende >Assignee: Daniel van der Ende >Priority: Minor > Fix For: 1.8.0 > > > In situations with long running commands that are executed via the SSH_hook, > it is necessary to set the SSH keep alive option, with a corresponding > interval at which to ensure the connection stays alive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] gsilk commented on issue #4308: [AIRFLOW-3502] Add celery config option for setting "pool"
gsilk commented on issue #4308: [AIRFLOW-3502] Add celery config option for setting "pool" URL: https://github.com/apache/incubator-airflow/pull/4308#issuecomment-446667800 Thanks for catching that, I've updated the PR This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-490) Scope should be auto-added to the connection per GCP hook
[ https://issues.apache.org/jira/browse/AIRFLOW-490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719220#comment-16719220 ] jack commented on AIRFLOW-490: -- Sounds like a reasonable idea > Scope should be auto-added to the connection per GCP hook > - > > Key: AIRFLOW-490 > URL: https://issues.apache.org/jira/browse/AIRFLOW-490 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Major > > The GCP hooks should auto add the scope they need to work to the GCP > connection so a user don't need specify the obvious scopes, like: > https://www.googleapis.com/auth/cloud-platform > We should keep the scope field for extra scopes requested for special cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3419) S3_hook.select_key is broken on Python3
[
https://issues.apache.org/jira/browse/AIRFLOW-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719217#comment-16719217
]
Ash Berlin-Taylor commented on AIRFLOW-3419:
My memory is that Python2 doesn't have a distinction between bytes and str, so
I don't see how this can affect Python2?
> S3_hook.select_key is broken on Python3
> ---
>
> Key: AIRFLOW-3419
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3419
> Project: Apache Airflow
> Issue Type: Bug
> Components: boto3, hooks
>Affects Versions: 1.10.1
>Reporter: Maria Rebelka
>Priority: Major
>
> Hello,
> Using select_key throws an error:
> {quote}text = S3Hook('aws_conn').select_key(key='my_key',
> bucket_name='my_bucket',
> expression='SELECT * FROM S3Object s',
> expression_type='SQL',
> input_serialization={'JSON': \{'Type':
> 'DOCUMENT'}},
> output_serialization={'JSON': {}}){quote}
> Traceback (most recent call last):
> {quote} File "db.py", line 31, in
> output_serialization={'JSON': {}})
> File "/usr/local/lib/python3.4/site-packages/airflow/hooks/S3_hook.py",
> line 262, in select_key
> for event in response['Payload']
> TypeError: sequence item 0: expected str instance, bytes found{quote}
> Seems that the problem is in this line:
> S3_hook.py, line 262: return ''.join(event['Records']['Payload']
> which probably should be return
> ''.join(event['Records']['Payload'].decode('utf-8')
> From example in Amazon blog:
> https://aws.amazon.com/blogs/aws/s3-glacier-select/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-473) Allow the usage of the same connection "service key" for GCP DataFlow
[ https://issues.apache.org/jira/browse/AIRFLOW-473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719216#comment-16719216 ] jack commented on AIRFLOW-473: -- Is this still an issue? > Allow the usage of the same connection "service key" for GCP DataFlow > - > > Key: AIRFLOW-473 > URL: https://issues.apache.org/jira/browse/AIRFLOW-473 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Reporter: Alex Van Boxel >Assignee: Alex Van Boxel >Priority: Major > > We need a way to pass the service key to the Cloud DataFlow jar file on > places where the local service account is not available (for example in a > kubernetes cluster). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-691) Add SSH keepalive option to ssh_hook
[ https://issues.apache.org/jira/browse/AIRFLOW-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719213#comment-16719213 ] jack commented on AIRFLOW-691: -- Fixed with https://github.com/apache/incubator-airflow/pull/1937 > Add SSH keepalive option to ssh_hook > > > Key: AIRFLOW-691 > URL: https://issues.apache.org/jira/browse/AIRFLOW-691 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, hooks >Reporter: Daniel van der Ende >Assignee: Daniel van der Ende >Priority: Minor > > In situations with long running commands that are executed via the SSH_hook, > it is necessary to set the SSH keep alive option, with a corresponding > interval at which to ensure the connection stays alive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3419) S3_hook.select_key is broken on Python3
[
https://issues.apache.org/jira/browse/AIRFLOW-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719209#comment-16719209
]
jack commented on AIRFLOW-3419:
---
I'm not sure this is the reason... the decode is the same for Python 3 and
Python 2.7
> S3_hook.select_key is broken on Python3
> ---
>
> Key: AIRFLOW-3419
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3419
> Project: Apache Airflow
> Issue Type: Bug
> Components: boto3, hooks
>Affects Versions: 1.10.1
>Reporter: Maria Rebelka
>Priority: Major
>
> Hello,
> Using select_key throws an error:
> {quote}text = S3Hook('aws_conn').select_key(key='my_key',
> bucket_name='my_bucket',
> expression='SELECT * FROM S3Object s',
> expression_type='SQL',
> input_serialization={'JSON': \{'Type':
> 'DOCUMENT'}},
> output_serialization={'JSON': {}}){quote}
> Traceback (most recent call last):
> {quote} File "db.py", line 31, in
> output_serialization={'JSON': {}})
> File "/usr/local/lib/python3.4/site-packages/airflow/hooks/S3_hook.py",
> line 262, in select_key
> for event in response['Payload']
> TypeError: sequence item 0: expected str instance, bytes found{quote}
> Seems that the problem is in this line:
> S3_hook.py, line 262: return ''.join(event['Records']['Payload']
> which probably should be return
> ''.join(event['Records']['Payload'].decode('utf-8')
> From example in Amazon blog:
> https://aws.amazon.com/blogs/aws/s3-glacier-select/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] stale[bot] commented on issue #3101: AIRFLOW-2179: Make parametrable the IP on which the worker log server binds to
stale[bot] commented on issue #3101: AIRFLOW-2179: Make parametrable the IP on which the worker log server binds to URL: https://github.com/apache/incubator-airflow/pull/3101#issuecomment-446647090 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] NielsZeilemaker commented on issue #4283: [AIRFLOW-3450] Remove unnecessary sigint handler
NielsZeilemaker commented on issue #4283: [AIRFLOW-3450] Remove unnecessary sigint handler URL: https://github.com/apache/incubator-airflow/pull/4283#issuecomment-446640912 @fokko in the case of Cloud Composer it is indeed the health check failing and hence a docker sends a sigterm. The actual exception which is in this case being propagated is the SystemExit exception being thrown by the sys.exit(0). It is a bit confusing, as the sigint_handler will be called on the main_thread and intermixed with other code. So the stacktrace isn't really a logical sequence of events anymore. Catching the sigterm and attempting a nice shutdown seems like a good idea, but I don't know if the current sys.exit(0) is the best implementation. Maybe setting a flag somewhere is a better option. Catching the sigint (which could be a ctrl+c) is also good idea, as otherwise the scheduler will probably not respond to ctrl+c anymore. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kppullin commented on issue #4308: [AIRFLOW-3501] Add celery config option for setting "pool"
kppullin commented on issue #4308: [AIRFLOW-3501] Add celery config option for
setting "pool"
URL:
https://github.com/apache/incubator-airflow/pull/4308#issuecomment-446626284
@gsilk - Should this be tagged with AIRFLOW-3502 ("Add config option to
control celery pool used" instead of AIRFLOW-3501 ("Add config option to load
dags in an image with the kubernetes executor.")?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] XD-DENG commented on issue #4309: [AIRFLOW-3504] Extend/refine the functionality of "/health" endpoint
XD-DENG commented on issue #4309: [AIRFLOW-3504] Extend/refine the functionality of "/health" endpoint URL: https://github.com/apache/incubator-airflow/pull/4309#issuecomment-446625410 1 test (out of 9) failed due to transient error. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] swiffer opened a new pull request #4310: add ARMEDANGELS to the list of customers
swiffer opened a new pull request #4310: add ARMEDANGELS to the list of
customers
URL: https://github.com/apache/incubator-airflow/pull/4310
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
### Code Quality
- [ ] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-3504) Extend/refine the functionality of "/health" endpoint
[ https://issues.apache.org/jira/browse/AIRFLOW-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodong DENG updated AIRFLOW-3504: --- Summary: Extend/refine the functionality of "/health" endpoint (was: Refine the functionality of "/health" endpoint) > Extend/refine the functionality of "/health" endpoint > - > > Key: AIRFLOW-3504 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3504 > Project: Apache Airflow > Issue Type: Improvement > Components: webapp >Reporter: Xiaodong DENG >Assignee: Xiaodong DENG >Priority: Minor > > For both www/ and www_rbac/, there is "/health" endpoint, which is supposed > to help users know the "health" condition of the Airflow application. > But the functionality of it at this moment is very limited. It can only help > check if the webserver is running, but actually the "health" of the > application may be bad even though this endpoint returns "The server is > healthy!" (scheduler may be down, database backend may be disconnected, etc) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3504) Refine the functionality of "/health" endpoint
[
https://issues.apache.org/jira/browse/AIRFLOW-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719033#comment-16719033
]
ASF GitHub Bot commented on AIRFLOW-3504:
-
XD-DENG opened a new pull request #4309: [AIRFLOW-3504] Extend/refine the
functionality of "/health" endpoint
URL: https://github.com/apache/incubator-airflow/pull/4309
### Jira
- https://issues.apache.org/jira/browse/AIRFLOW-3504
### Description
The Issue That This PR Tried to Address
For both www/ and www_rbac/, there is "/health" endpoint, which is supposed
to help users know the "health" condition of the Airflow application.
But the functionality of it at this moment is very limited. It can only help
check if the webserver is running, but actually the "health" of the application
may be bad even though this endpoint returns "The server is healthy!"
(scheduler may be down, database backend may be disconnected, etc)
What Have Been Done in This PR
Extend the functionality of "`/health`" endpoint:
1. Check if webserver is up;
2. Check if database backend can be connected;
3. Check the latest scheduler run
This is done for both `/www` and `/www_rbac`. Tests are added accordingly as
well.
Note that no authentication is required to access "`/health`" endpoint (no
sensitive information will be exposed through it).
(Deleted an unnecessary line in `airflow/www/views.py` as well)
How Final Result Looks Like
The "`/health`" endpoint will return a JSON like
```
{
'webserver':'The server is healthy!',
'database':'available',
'scheduler':{
'latest_scheduler_run':'2018-12-26 17:15:11+00:00'
}
}
```
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine the functionality of "/health" endpoint
> --
>
> Key: AIRFLOW-3504
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3504
> Project: Apache Airflow
> Issue Type: Improvement
> Components: webapp
>Reporter: Xiaodong DENG
>Assignee: Xiaodong DENG
>Priority: Minor
>
> For both www/ and www_rbac/, there is "/health" endpoint, which is supposed
> to help users know the "health" condition of the Airflow application.
> But the functionality of it at this moment is very limited. It can only help
> check if the webserver is running, but actually the "health" of the
> application may be bad even though this endpoint returns "The server is
> healthy!" (scheduler may be down, database backend may be disconnected, etc)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] XD-DENG opened a new pull request #4309: [AIRFLOW-3504] Extend/refine the functionality of "/health" endpoint
XD-DENG opened a new pull request #4309: [AIRFLOW-3504] Extend/refine the
functionality of "/health" endpoint
URL: https://github.com/apache/incubator-airflow/pull/4309
### Jira
- https://issues.apache.org/jira/browse/AIRFLOW-3504
### Description
The Issue That This PR Tried to Address
For both www/ and www_rbac/, there is "/health" endpoint, which is supposed
to help users know the "health" condition of the Airflow application.
But the functionality of it at this moment is very limited. It can only help
check if the webserver is running, but actually the "health" of the application
may be bad even though this endpoint returns "The server is healthy!"
(scheduler may be down, database backend may be disconnected, etc)
What Have Been Done in This PR
Extend the functionality of "`/health`" endpoint:
1. Check if webserver is up;
2. Check if database backend can be connected;
3. Check the latest scheduler run
This is done for both `/www` and `/www_rbac`. Tests are added accordingly as
well.
Note that no authentication is required to access "`/health`" endpoint (no
sensitive information will be exposed through it).
(Deleted an unnecessary line in `airflow/www/views.py` as well)
How Final Result Looks Like
The "`/health`" endpoint will return a JSON like
```
{
'webserver':'The server is healthy!',
'database':'available',
'scheduler':{
'latest_scheduler_run':'2018-12-26 17:15:11+00:00'
}
}
```
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Code Quality
- [x] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-3504) Refine the functionality of "/health" endpoint
Xiaodong DENG created AIRFLOW-3504: -- Summary: Refine the functionality of "/health" endpoint Key: AIRFLOW-3504 URL: https://issues.apache.org/jira/browse/AIRFLOW-3504 Project: Apache Airflow Issue Type: Improvement Components: webapp Reporter: Xiaodong DENG Assignee: Xiaodong DENG For both www/ and www_rbac/, there is "/health" endpoint, which is supposed to help users know the "health" condition of the Airflow application. But the functionality of it at this moment is very limited. It can only help check if the webserver is running, but actually the "health" of the application may be bad even though this endpoint returns "The server is healthy!" (scheduler may be down, database backend may be disconnected, etc) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ashb commented on issue #4068: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow
ashb commented on issue #4068: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow URL: https://github.com/apache/incubator-airflow/pull/4068#issuecomment-446580223 Do you ever do anything else than Submit the job and the immediately start a sensor to poll for it? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-3500) Make task duration display user friendly
[ https://issues.apache.org/jira/browse/AIRFLOW-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ofer Zelig resolved AIRFLOW-3500. - Resolution: Implemented > Make task duration display user friendly > > > Key: AIRFLOW-3500 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3500 > Project: Apache Airflow > Issue Type: Improvement > Components: DAG >Affects Versions: 1.10.1 >Reporter: Ofer Zelig >Assignee: Ofer Zelig >Priority: Major > Fix For: 1.10.2 > > > When hovering over a task (in Graph mode), the duration it took is displayed > as a plain number, which doesn't say what the number is (it's actually > seconds). > When you see something like 2716 it's impractical to know how much time it > actually took, unless you can quickly do the math in your head. > Change the display to read days/hours/minutes/seconds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3503) GoogleCloudStorageHook delete return success when nothing was done
lot created AIRFLOW-3503:
Summary: GoogleCloudStorageHook delete return success when
nothing was done
Key: AIRFLOW-3503
URL: https://issues.apache.org/jira/browse/AIRFLOW-3503
Project: Apache Airflow
Issue Type: Bug
Affects Versions: 1.10.1
Reporter: lot
I'm loading files to BigQuery from Storage using:
{{gcs_export_uri = BQ_TABLE_NAME + '/' + EXEC_TIMESTAMP_PATH + '/*'
gcs_to_bigquery_op = GoogleCloudStorageToBigQueryOperator( dag=dag,
task_id='load_products_to_BigQuery', bucket=GCS_BUCKET_ID,
destination_project_dataset_table=table_name_template,
source_format='NEWLINE_DELIMITED_JSON', source_objects=[gcs_export_uri],
src_fmt_configs=\{'ignoreUnknownValues': True},
create_disposition='CREATE_IF_NEEDED', write_disposition='WRITE_TRUNCATE',
skip_leading_rows = 1, google_cloud_storage_conn_id=CONNECTION_ID,
bigquery_conn_id=CONNECTION_ID)}}
After that I want to delete the files so I do:
{{def delete_folder():}}
{{ """}}
{{ Delete files Google cloud storage}}
{{ """}}
{{ hook = GoogleCloudStorageHook(}}
{{ google_cloud_storage_conn_id=CONNECTION_ID)}}
{{ hook.delete(}}
{{ bucket=GCS_BUCKET_ID,}}
{{ object=gcs_export_uri)}}
{{This runs with PythonOperator.}}
{{The task marked as Success even though nothing was deleted.}}
{{Log:}}
[2018-12-12 11:31:29,247] \{base_task_runner.py:98} INFO - Subtask: [2018-12-12
11:31:29,247] \{transport.py:151} INFO - Attempting refresh to obtain initial
access_token [2018-12-12 11:31:29,249] \{base_task_runner.py:98} INFO -
Subtask: [2018-12-12 11:31:29,249] \{client.py:795} INFO - Refreshing
access_token [2018-12-12 11:31:29,584] \{base_task_runner.py:98} INFO -
Subtask: [2018-12-12 11:31:29,583] \{python_operator.py:90} INFO - Done.
Returned value was: None
I expect the function to fail and return something like "file was not found" if
there is nothing to delete Or let the user decide with specific flag if he
wants the function to fail or success if files were not found.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] oelesinsc24 commented on issue #4068: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow
oelesinsc24 commented on issue #4068: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow URL: https://github.com/apache/incubator-airflow/pull/4068#issuecomment-446561057 > > I would recommend submitting a glue job and then using the sensor to poll for updates. > > Could you explain your thinking? If this is what 90% of people will want, then does it make more sense having it be the default and built as a single operator? I guess if 90% of people want this then we can add it. However, in my experience with AWS Glue, it makes sense to submit the job and then poll for it This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Reopened] (AIRFLOW-2217) Add Slack Webhook Hook/Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor reopened AIRFLOW-2217: Re-opening to correct Fix Version > Add Slack Webhook Hook/Operator > --- > > Key: AIRFLOW-2217 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2217 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib, hooks, operators >Reporter: Daniel van der Ende >Assignee: Daniel van der Ende >Priority: Minor > Fix For: 2.0.0 > > > Slack offers several ways to interact with it. Airflow currently has support > for the full Slack API with the [Slack > hook|https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/slack_hook.py] > This, however, can be a bit heavy-handed for simple posting of messages. > Slack also offers the possibility of using an [Incoming > webhook|https://api.slack.com/incoming-webhooks] > It would be nice to have a hook in Airflow to use the incoming webhook API > offered by Slack. A lot of use cases for integrating Slack in Airflow are > oriented on posting error or success messages to a Slack channel based on the > outcome of a task instance. The Webhook API is perfect for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (AIRFLOW-1236) Slack Operator uses deprecated API, and should use Connection
[ https://issues.apache.org/jira/browse/AIRFLOW-1236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor closed AIRFLOW-1236. -- Resolution: Duplicate > Slack Operator uses deprecated API, and should use Connection > - > > Key: AIRFLOW-1236 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1236 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Erik Forsberg >Priority: Major > > The SlackAPIPostOperator requires legacy Slack API tokens, and hardcodes said > token into the DAG instead of using a Connection. > Provide an operator that uses the Incoming Webhook API instead, and stores > the webhook URL in a Connection. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2217) Add Slack Webhook Hook/Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-2217. Resolution: Fixed Fix Version/s: 1.10.0 > Add Slack Webhook Hook/Operator > --- > > Key: AIRFLOW-2217 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2217 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib, hooks, operators >Reporter: Daniel van der Ende >Assignee: Daniel van der Ende >Priority: Minor > Fix For: 2.0.0, 1.10.0 > > > Slack offers several ways to interact with it. Airflow currently has support > for the full Slack API with the [Slack > hook|https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/slack_hook.py] > This, however, can be a bit heavy-handed for simple posting of messages. > Slack also offers the possibility of using an [Incoming > webhook|https://api.slack.com/incoming-webhooks] > It would be nice to have a hook in Airflow to use the incoming webhook API > offered by Slack. A lot of use cases for integrating Slack in Airflow are > oriented on posting error or success messages to a Slack channel based on the > outcome of a task instance. The Webhook API is perfect for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io edited a comment on issue #4286: [AIRFLOW-3310] Google Cloud Spanner deploy / delete operators
codecov-io edited a comment on issue #4286: [AIRFLOW-3310] Google Cloud Spanner deploy / delete operators URL: https://github.com/apache/incubator-airflow/pull/4286#issuecomment-445408820 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4286?src=pr&el=h1) Report > Merging [#4286](https://codecov.io/gh/apache/incubator-airflow/pull/4286?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/57e355c516b6deaa97ca7ac7ec83e5fc0f50ab87?src=pr&el=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4286?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master #4286 +/- ## == Coverage78.1% 78.1% == Files 201 201 Lines 16464 16464 == Hits12859 12859 Misses 36053605 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4286?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4286?src=pr&el=footer). Last update [57e355c...daf545a](https://codecov.io/gh/apache/incubator-airflow/pull/4286?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #4225: [AIRFLOW-3383] Rotate fernet keys.
ashb commented on a change in pull request #4225: [AIRFLOW-3383] Rotate fernet keys. URL: https://github.com/apache/incubator-airflow/pull/4225#discussion_r240969490 ## File path: airflow/bin/cli.py ## @@ -614,6 +614,16 @@ def next_execution(args): print(None) +@cli_utils.action_logging +def rotate_credentials(args): Review comment: Code looks good, only comment is that I think this name needs changing - even knowning what the PR was for I thought this was about rotating the credentials _in_ the connection. How about `reencrypt_connections`, `rotate_fernet_key`, or `rotate_encryption_keys`? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on issue #4305: [AIRFLOW-3397] Fix integrety error in rbac AirflowSecurityManager
ashb commented on issue #4305: [AIRFLOW-3397] Fix integrety error in rbac AirflowSecurityManager URL: https://github.com/apache/incubator-airflow/pull/4305#issuecomment-446548249 @feng-tao sounds good - you know this code better than anyone :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] sprzedwojski commented on a change in pull request #4286: [AIRFLOW-3310] Google Cloud Spanner deploy / delete operators
sprzedwojski commented on a change in pull request #4286: [AIRFLOW-3310] Google Cloud Spanner deploy / delete operators URL: https://github.com/apache/incubator-airflow/pull/4286#discussion_r240959767 ## File path: airflow/contrib/hooks/gcp_spanner_hook.py ## @@ -0,0 +1,183 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from google.longrunning.operations_grpc_pb2 import Operation # noqa: F401 +from typing import Optional, Callable # noqa: F401 + +from google.api_core.exceptions import GoogleAPICallError +from google.cloud.spanner_v1.client import Client +from google.cloud.spanner_v1.instance import Instance # noqa: F401 + +from airflow.contrib.hooks.gcp_api_base_hook import GoogleCloudBaseHook + + +# noinspection PyAbstractClass +class CloudSpannerHook(GoogleCloudBaseHook): +""" +Hook for Google Cloud Spanner APIs. +""" +_client = None + +def __init__(self, + gcp_conn_id='google_cloud_default', + delegate_to=None): +super(CloudSpannerHook, self).__init__(gcp_conn_id, delegate_to) + +def get_client(self, project_id): +# type: (str) -> Client +""" +Provides a client for interacting with Cloud Spanner API. + +:param project_id: The ID of the project which owns the instances, tables and data. +:type project_id: str +:return: Client for interacting with Cloud Spanner API. See: + https://googleapis.github.io/google-cloud-python/latest/spanner/client-api.html#google.cloud.spanner_v1.client.Client +:rtype: object +""" +if not self._client: +self._client = Client(project=project_id, credentials=self._get_credentials()) +return self._client + +def get_instance(self, project_id, instance_id): +# type: (str, str) -> Optional[Instance] +""" +Gets information about a particular instance. + +:param project_id: The ID of the project which owns the instances, tables and data. +:type project_id: str +:param instance_id: The ID of the instance. +:type instance_id: str +:return: Representation of a Cloud Spanner Instance. See: + https://googleapis.github.io/google-cloud-python/latest/spanner/instance-api.html#google.cloud.spanner_v1.instance.Instance +:rtype: object +""" +client = self.get_client(project_id) +instance = client.instance(instance_id) +if not instance.exists(): +return None +return instance + +def create_instance(self, project_id, instance_id, configuration_name, node_count, +display_name): +# type: (str, str, str, int, str) -> bool +""" +Creates a new Cloud Spanner instance. + +:param project_id: The ID of the project which owns the instances, tables and +data. +:type project_id: str +:param instance_id: The ID of the instance. +:type instance_id: str +:param configuration_name: Name of the instance configuration defining how the +instance will be created. Required for instances which do not yet exist. +:type configuration_name: str +:param node_count: (Optional) Number of nodes allocated to the instance. +:type node_count: int +:param display_name: (Optional) The display name for the instance in the Cloud +Console UI. (Must be between 4 and 30 characters.) If this value is not set +in the constructor, will fall back to the instance ID. +:type display_name: str +:return: True if the operation succeeded, raises an exception otherwise. +:rtype: bool +""" +return self._apply_to_instance(project_id, instance_id, configuration_name, + node_count, display_name, lambda x: x.create()) + +def update_instance(self, project_id, instance_id, configuration_name, node_count, +display_name): +# type: (str, str, str, int, str) -> bool +""" +Updates an existing Cloud Spanner instance. + +:param project_id: Th
[jira] [Updated] (AIRFLOW-3327) BiqQuery job checking doesn't include location, which api requires outside US/EU
[ https://issues.apache.org/jira/browse/AIRFLOW-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-3327: Fix Version/s: 1.10.2 > BiqQuery job checking doesn't include location, which api requires outside > US/EU > > > Key: AIRFLOW-3327 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3327 > Project: Apache Airflow > Issue Type: Bug >Reporter: Daniel Swiegers >Assignee: Kaxil Naik >Priority: Minor > Labels: google-cloud-bigquery > Fix For: 1.10.2 > > Original Estimate: 24h > Remaining Estimate: 24h > > We use this api but don't set / pass through the geographical location. > Which is required in areas other than US and EU. > Can be seen in contrib/hooks/big_query_hook.py poll_job_complete > [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get] > |The geographic location of the job. Required except for US and EU. See > details at > https://cloud.google.com/bigquery/docs/locations#specifying_your_location.| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3327) BiqQuery job checking doesn't include location, which api requires outside US/EU
[ https://issues.apache.org/jira/browse/AIRFLOW-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718740#comment-16718740 ] Kaxil Naik commented on AIRFLOW-3327: - I have assigned this to myself [~dsdinter] :-) Will work on it and should resolve it by this weekend > BiqQuery job checking doesn't include location, which api requires outside > US/EU > > > Key: AIRFLOW-3327 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3327 > Project: Apache Airflow > Issue Type: Bug >Reporter: Daniel Swiegers >Assignee: Kaxil Naik >Priority: Minor > Labels: google-cloud-bigquery > Original Estimate: 24h > Remaining Estimate: 24h > > We use this api but don't set / pass through the geographical location. > Which is required in areas other than US and EU. > Can be seen in contrib/hooks/big_query_hook.py poll_job_complete > [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get] > |The geographic location of the job. Required except for US and EU. See > details at > https://cloud.google.com/bigquery/docs/locations#specifying_your_location.| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-3327) BiqQuery job checking doesn't include location, which api requires outside US/EU
[ https://issues.apache.org/jira/browse/AIRFLOW-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik reassigned AIRFLOW-3327: --- Assignee: Kaxil Naik > BiqQuery job checking doesn't include location, which api requires outside > US/EU > > > Key: AIRFLOW-3327 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3327 > Project: Apache Airflow > Issue Type: Bug >Reporter: Daniel Swiegers >Assignee: Kaxil Naik >Priority: Minor > Labels: google-cloud-bigquery > Original Estimate: 24h > Remaining Estimate: 24h > > We use this api but don't set / pass through the geographical location. > Which is required in areas other than US and EU. > Can be seen in contrib/hooks/big_query_hook.py poll_job_complete > [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get] > |The geographic location of the job. Required except for US and EU. See > details at > https://cloud.google.com/bigquery/docs/locations#specifying_your_location.| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3327) BiqQuery job checking doesn't include location, which api requires outside US/EU
[ https://issues.apache.org/jira/browse/AIRFLOW-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718724#comment-16718724 ] David Sabater commented on AIRFLOW-3327: It would be great for someone from the community to pick this up. Maybe magic [~kaxilnaik] can help? :) > BiqQuery job checking doesn't include location, which api requires outside > US/EU > > > Key: AIRFLOW-3327 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3327 > Project: Apache Airflow > Issue Type: Bug >Reporter: Daniel Swiegers >Assignee: Kaxil Naik >Priority: Minor > Labels: google-cloud-bigquery > Original Estimate: 24h > Remaining Estimate: 24h > > We use this api but don't set / pass through the geographical location. > Which is required in areas other than US and EU. > Can be seen in contrib/hooks/big_query_hook.py poll_job_complete > [https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/get] > |The geographic location of the job. Required except for US and EU. See > details at > https://cloud.google.com/bigquery/docs/locations#specifying_your_location.| -- This message was sent by Atlassian JIRA (v7.6.3#76005)
