[GitHub] vardancse edited a comment on issue #3994: [AIRFLOW-3136] Add retry_number to TaskInstance Key property to avoid race condition
vardancse edited a comment on issue #3994: [AIRFLOW-3136] Add retry_number to TaskInstance Key property to avoid race condition URL: https://github.com/apache/incubator-airflow/pull/3994#issuecomment-428557928 @ashb For us, we can easily reproduce by running a DAG which has a task bound to fail with retries set >0(preferably 4 or 5) and retry_delay set to 1 second, let me know if you're able to repro else I'll share sample DAG file. Also, our email_on_retry and email_on_failure is marked true, so once we see this issue of race condition, we will get an email something like below > Try 2 out of 4 > Exception: > Executor reports task instance %s finished (%s) although the task says its %s. Was the task killed externally? > Log: Link > Host: 72a8e9235db3 > Log file: /usr/local/airflow/logs/1fac4a1d57e551320a49403c_ufr09_qwpbfy/ACPComputeGateway_0_UFR09Task/2018-10-03T05:45:05.000221.log > Mark success: Link > EDIT-1: We have also done 1 testing at our side, where we were not able to reproduce issue when dag_folder consist less number of files but as soon as it touched 40+ Dags in our dag_folder, we were able to reproduce, probably increasing dags in dag_folder increasing 1 cycle of dags processing. EDIT-2: Dag which was used to reproduce this issue can be found [here](https://github.com/vardancse/incubator-airflow/blob/race-condition-temp/airflow/example_dags/tutorial.py) and this is actually the originally shipped example dag with few changes mentioned [here](https://github.com/vardancse/incubator-airflow/pull/1/files). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] deagon commented on a change in pull request #3977: [AIRFLOW-3132] Add option for DockerOperator
deagon commented on a change in pull request #3977: [AIRFLOW-3132] Add option for DockerOperator URL: https://github.com/apache/incubator-airflow/pull/3977#discussion_r224676820 ## File path: airflow/operators/docker_operator.py ## @@ -47,6 +47,10 @@ class DockerOperator(BaseOperator): :param api_version: Remote API version. Set to ``auto`` to automatically detect the server's version. :type api_version: str +:type auto_remove: bool Review comment: You're right. I'm going to modify it soon. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3177) Change scheduler_heartbeat metric from gauge to counter
[
https://issues.apache.org/jira/browse/AIRFLOW-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647515#comment-16647515
]
ASF GitHub Bot commented on AIRFLOW-3177:
-
Fokko closed pull request #4027: [AIRFLOW-3177] Change scheduler_heartbeat from
gauge to counter
URL: https://github.com/apache/incubator-airflow/pull/4027
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/UPDATING.md b/UPDATING.md
index 74337f3fe8..5e1402576b 100644
--- a/UPDATING.md
+++ b/UPDATING.md
@@ -52,6 +52,10 @@ To delete a user:
airflow users --delete --username jondoe
```
+### StatsD Metrics
+
+The `scheduler_heartbeat` metric has been changed from a gauge to a counter.
Each loop of the scheduler will increment the counter by 1. This provides a
higher degree of visibility and allows for better integration with Prometheus
using the [StatsD Exporter](https://github.com/prometheus/statsd_exporter).
Scheduler upness can be determined by graphing and alerting using a rate. If
the scheduler goes down, the rate will drop to 0.
+
### Custom auth backends interface change
We have updated the version of flask-login we depend upon, and as a result any
diff --git a/airflow/jobs.py b/airflow/jobs.py
index b224f75545..3922939a86 100644
--- a/airflow/jobs.py
+++ b/airflow/jobs.py
@@ -1895,7 +1895,7 @@ def process_file(self, file_path, pickle_dags=False,
session=None):
@provide_session
def heartbeat_callback(self, session=None):
-Stats.gauge('scheduler_heartbeat', 1, 1)
+Stats.incr('scheduler_heartbeat', 1, 1)
class BackfillJob(BaseJob):
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Change scheduler_heartbeat metric from gauge to counter
> ---
>
> Key: AIRFLOW-3177
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3177
> Project: Apache Airflow
> Issue Type: Improvement
> Components: scheduler
>Affects Versions: 2.0.0
>Reporter: Greg Neiheisel
>Assignee: Greg Neiheisel
>Priority: Minor
> Fix For: 1.10.1
>
>
> Currently, the scheduler_heartbeat metric exposed with the statsd integration
> is a gauge. I'm proposing to change the gauge to a counter for a better
> integration with Prometheus via the
> [statsd_exporter|[https://github.com/prometheus/statsd_exporter].]
> Rather than pointing Airflow at an actual statsd server, you can point it at
> this exporter, which will accumulate the metrics and expose them to be
> scraped by Prometheus at /metrics. The problem is that once this value is set
> when the scheduler runs its first loop, it will always be exposed to
> Prometheus as 1. The scheduler can crash, or be turned off and the statsd
> exporter will report a 1 until it is restarted and rebuilds its internal
> state.
> By turning this metric into a counter, we can detect an issue with the
> scheduler by graphing and alerting using a rate. If the rate of change of the
> counter drops below what it should be at (determined by the
> scheduler_heartbeat_secs setting), we can fire an alert.
> This should be helpful for adoption in Kubernetes environments where
> Prometheus is pretty much the standard.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3177) Change scheduler_heartbeat metric from gauge to counter
[ https://issues.apache.org/jira/browse/AIRFLOW-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-3177: -- Fix Version/s: 1.10.1 > Change scheduler_heartbeat metric from gauge to counter > --- > > Key: AIRFLOW-3177 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3177 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.0 >Reporter: Greg Neiheisel >Assignee: Greg Neiheisel >Priority: Minor > Fix For: 1.10.1 > > > Currently, the scheduler_heartbeat metric exposed with the statsd integration > is a gauge. I'm proposing to change the gauge to a counter for a better > integration with Prometheus via the > [statsd_exporter|[https://github.com/prometheus/statsd_exporter].] > Rather than pointing Airflow at an actual statsd server, you can point it at > this exporter, which will accumulate the metrics and expose them to be > scraped by Prometheus at /metrics. The problem is that once this value is set > when the scheduler runs its first loop, it will always be exposed to > Prometheus as 1. The scheduler can crash, or be turned off and the statsd > exporter will report a 1 until it is restarted and rebuilds its internal > state. > By turning this metric into a counter, we can detect an issue with the > scheduler by graphing and alerting using a rate. If the rate of change of the > counter drops below what it should be at (determined by the > scheduler_heartbeat_secs setting), we can fire an alert. > This should be helpful for adoption in Kubernetes environments where > Prometheus is pretty much the standard. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] Fokko closed pull request #4027: [AIRFLOW-3177] Change scheduler_heartbeat from gauge to counter
Fokko closed pull request #4027: [AIRFLOW-3177] Change scheduler_heartbeat from
gauge to counter
URL: https://github.com/apache/incubator-airflow/pull/4027
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/UPDATING.md b/UPDATING.md
index 74337f3fe8..5e1402576b 100644
--- a/UPDATING.md
+++ b/UPDATING.md
@@ -52,6 +52,10 @@ To delete a user:
airflow users --delete --username jondoe
```
+### StatsD Metrics
+
+The `scheduler_heartbeat` metric has been changed from a gauge to a counter.
Each loop of the scheduler will increment the counter by 1. This provides a
higher degree of visibility and allows for better integration with Prometheus
using the [StatsD Exporter](https://github.com/prometheus/statsd_exporter).
Scheduler upness can be determined by graphing and alerting using a rate. If
the scheduler goes down, the rate will drop to 0.
+
### Custom auth backends interface change
We have updated the version of flask-login we depend upon, and as a result any
diff --git a/airflow/jobs.py b/airflow/jobs.py
index b224f75545..3922939a86 100644
--- a/airflow/jobs.py
+++ b/airflow/jobs.py
@@ -1895,7 +1895,7 @@ def process_file(self, file_path, pickle_dags=False,
session=None):
@provide_session
def heartbeat_callback(self, session=None):
-Stats.gauge('scheduler_heartbeat', 1, 1)
+Stats.incr('scheduler_heartbeat', 1, 1)
class BackfillJob(BaseJob):
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-3177) Change scheduler_heartbeat metric from gauge to counter
[ https://issues.apache.org/jira/browse/AIRFLOW-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-3177: -- Affects Version/s: 2.0.0 > Change scheduler_heartbeat metric from gauge to counter > --- > > Key: AIRFLOW-3177 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3177 > Project: Apache Airflow > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.0 >Reporter: Greg Neiheisel >Assignee: Greg Neiheisel >Priority: Minor > Fix For: 1.10.1 > > > Currently, the scheduler_heartbeat metric exposed with the statsd integration > is a gauge. I'm proposing to change the gauge to a counter for a better > integration with Prometheus via the > [statsd_exporter|[https://github.com/prometheus/statsd_exporter].] > Rather than pointing Airflow at an actual statsd server, you can point it at > this exporter, which will accumulate the metrics and expose them to be > scraped by Prometheus at /metrics. The problem is that once this value is set > when the scheduler runs its first loop, it will always be exposed to > Prometheus as 1. The scheduler can crash, or be turned off and the statsd > exporter will report a 1 until it is restarted and rebuilds its internal > state. > By turning this metric into a counter, we can detect an issue with the > scheduler by graphing and alerting using a rate. If the rate of change of the > counter drops below what it should be at (determined by the > scheduler_heartbeat_secs setting), we can fire an alert. > This should be helpful for adoption in Kubernetes environments where > Prometheus is pretty much the standard. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] Fokko commented on issue #4027: [AIRFLOW-3177] Change scheduler_heartbeat from gauge to counter
Fokko commented on issue #4027: [AIRFLOW-3177] Change scheduler_heartbeat from gauge to counter URL: https://github.com/apache/incubator-airflow/pull/4027#issuecomment-429211898 Thanks @schnie Merging! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3977: [AIRFLOW-3132] Add option for DockerOperator
Fokko commented on a change in pull request #3977: [AIRFLOW-3132] Add option for DockerOperator URL: https://github.com/apache/incubator-airflow/pull/3977#discussion_r224675921 ## File path: airflow/operators/docker_operator.py ## @@ -47,6 +47,10 @@ class DockerOperator(BaseOperator): :param api_version: Remote API version. Set to ``auto`` to automatically detect the server's version. :type api_version: str +:type auto_remove: bool Review comment: Please switch the order. First `:param`, then `:type`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on issue #3975: [AIRFLOW-XXX] Pin docker requirement version
Fokko commented on issue #3975: [AIRFLOW-XXX] Pin docker requirement version URL: https://github.com/apache/incubator-airflow/pull/3975#issuecomment-429211053 Please create a Jira ticket for this, and mention the incompatibility with the `create_container` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on issue #4032: [AIRFLOW-2636] - fix task duration tab NoneType exception
Fokko commented on issue #4032: [AIRFLOW-2636] - fix task duration tab NoneType exception URL: https://github.com/apache/incubator-airflow/pull/4032#issuecomment-429210924 Thanks @helenweng-stripe Could you update the code for the rbac UI as well? https://github.com/apache/incubator-airflow/blob/19515e816785a6be828ec0191b9a7d83c9648110/airflow/www_rbac/views.py#L1393-L1395 The non-rbac UI will be deprecated for Airflow 2.0. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-3168) mysql not always done set up in Travis by the time it's called
[ https://issues.apache.org/jira/browse/AIRFLOW-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated AIRFLOW-3168: -- Fix Version/s: 2.0.0 > mysql not always done set up in Travis by the time it's called > -- > > Key: AIRFLOW-3168 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3168 > Project: Apache Airflow > Issue Type: Task > Components: ci >Reporter: Dima Kamalov >Assignee: Dima Kamalov >Priority: Minor > Fix For: 2.0.0 > > > seem to cause a build failure once every 5-10 tries, see e.g. > [https://travis-ci.org/apache/incubator-airflow/jobs/437669380] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] Fokko commented on issue #4033: Update airflow.gif file with a slower version
Fokko commented on issue #4033: Update airflow.gif file with a slower version URL: https://github.com/apache/incubator-airflow/pull/4033#issuecomment-429209719 Thanks @PaulVelthuis93 Can you rework the commit message, by prepending it with `[AIRFLOW-3187]`. https://help.github.com/articles/changing-a-commit-message/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators
Fokko commented on a change in pull request #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators URL: https://github.com/apache/incubator-airflow/pull/3828#discussion_r224674155 ## File path: airflow/contrib/operators/s3_to_sftp_operator.py ## @@ -0,0 +1,85 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +from airflow.models import BaseOperator +from airflow.hooks.S3_hook import S3Hook +from airflow.contrib.hooks.ssh_hook import SSHHook +from tempfile import NamedTemporaryFile +from urllib.parse import urlparse +from airflow.utils.decorators import apply_defaults + + +def get_s3_key(s3_key): Review comment: Can we make this a static function, part of the class? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators
Fokko commented on a change in pull request #3828: [AIRFLOW-2993] s3_to_sftp
and sftp_to_s3 operators
URL: https://github.com/apache/incubator-airflow/pull/3828#discussion_r224674421
##
File path: airflow/contrib/operators/sftp_to_s3_operator.py
##
@@ -0,0 +1,90 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.models import BaseOperator
+from airflow.hooks.S3_hook import S3Hook
+from airflow.contrib.hooks.ssh_hook import SSHHook
+from tempfile import NamedTemporaryFile
+from urllib.parse import urlparse
+from airflow.utils.decorators import apply_defaults
+
+
+def get_s3_key(s3_key):
+"""This parses the correct format for S3 keys
+regardless of how the S3 url is passed."""
+
+parsed_s3_key = urlparse(s3_key)
+return parsed_s3_key.path.lstrip('/')
+
+
+class SFTPToS3Operator(BaseOperator):
+"""
+This operator enables the transferring of files from a SFTP server to
Amazon S3
+:param sftp_conn_id:The sftp connection id. The name or identifier for
+establishing a connection to the SFTP server.
+:type sftp_conn_id: string
+:param sftp_path: The sftp remote path. This is the specified file
+path for downloading the file from the SFTP server.
+:type sftp_path:string
+:param s3_conn_id: The s3 connnection id. The name or identifier for
+establishing a connection to S3
+:type s3_conn_id: string
+:param s3_bucket: The targeted s3 bucket. This is the S3 bucket
+to where the file is uploaded.
+:type s3_bucket:string
+:param s3_key: The targeted s3 key. This is the specified path
+for uploading the file to S3.
+:type s3_key: string
+"""
+
+template_fields = ('s3_key', 'sftp_path')
+
+@apply_defaults
+def __init__(self,
+ s3_bucket,
+ s3_key,
+ sftp_path,
+ sftp_conn_id='ssh_default',
+ s3_conn_id='aws_default',
+ *args,
+ **kwargs):
+super(SFTPToS3Operator, self).__init__(*args, **kwargs)
+self.sftp_conn_id = sftp_conn_id
+self.sftp_path = sftp_path
+self.s3_bucket = s3_bucket
+self.s3_key = s3_key
+self.s3_conn_id = s3_conn_id
+
+def execute(self, context):
+self.s3_key = get_s3_key(self.s3_key)
+ssh_hook = SSHHook(ssh_conn_id=self.sftp_conn_id)
+s3_hook = S3Hook(self.s3_conn_id)
+
+ssh_client = ssh_hook.get_conn()
Review comment:
`sftp_client = ssh_hook.get_conn().open_sftp()`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] Fokko commented on a change in pull request #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators
Fokko commented on a change in pull request #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators URL: https://github.com/apache/incubator-airflow/pull/3828#discussion_r224674375 ## File path: airflow/contrib/operators/sftp_to_s3_operator.py ## @@ -0,0 +1,90 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +from airflow.models import BaseOperator +from airflow.hooks.S3_hook import S3Hook +from airflow.contrib.hooks.ssh_hook import SSHHook +from tempfile import NamedTemporaryFile +from urllib.parse import urlparse +from airflow.utils.decorators import apply_defaults + + +def get_s3_key(s3_key): Review comment: Make this one static as well This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators
Fokko commented on a change in pull request #3828: [AIRFLOW-2993] s3_to_sftp
and sftp_to_s3 operators
URL: https://github.com/apache/incubator-airflow/pull/3828#discussion_r224674466
##
File path: tests/contrib/operators/test_s3_to_sftp_operator.py
##
@@ -0,0 +1,167 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+
+from airflow import configuration
+from airflow import models
+from airflow.contrib.operators.s3_to_sftp_operator import S3ToSFTPOperator
+from airflow.contrib.operators.ssh_operator import SSHOperator
+from airflow.models import DAG, TaskInstance
+from airflow.settings import Session
+from airflow.utils import timezone
+from airflow.utils.timezone import datetime
+import boto3
+from moto import mock_s3
+
+
+TASK_ID = 'test_s3_to_sftp'
+BUCKET = 'test-s3-bucket'
+S3_KEY = 'test/test_1_file.csv'
+SFTP_PATH = '/tmp/remote_path.txt'
+SFTP_CONN_ID = 'ssh_default'
+S3_CONN_ID = 'aws_default'
+LOCAL_FILE_PATH = '/tmp/test_s3_upload'
+
+SFTP_MOCK_FILE = 'test_sftp_file.csv'
+S3_MOCK_FILES = 'test_1_file.csv'
+
+TEST_DAG_ID = 'unit_tests'
+DEFAULT_DATE = datetime(2018, 1, 1)
+
+
+def reset(dag_id=TEST_DAG_ID):
+session = Session()
+tis = session.query(models.TaskInstance).filter_by(dag_id=dag_id)
+tis.delete()
+session.commit()
+session.close()
+
+
+reset()
+
+
+class S3ToSFTPOperatorTest(unittest.TestCase):
+@mock_s3
+def setUp(self):
+configuration.load_test_config()
+from airflow.contrib.hooks.ssh_hook import SSHHook
+from airflow.hooks.S3_hook import S3Hook
+
+hook = SSHHook(ssh_conn_id='ssh_default')
+s3_hook = S3Hook('aws_default')
+hook.no_host_key_check = True
+args = {
+'owner': 'airflow',
+'start_date': DEFAULT_DATE,
+'provide_context': True
+}
+dag = DAG(TEST_DAG_ID + 'test_schedule_dag_once', default_args=args)
+dag.schedule_interval = '@once'
+
+self.hook = hook
+self.s3_hook = s3_hook
+
+self.ssh_client = self.hook.get_conn()
+self.sftp_client = self.ssh_client.open_sftp()
+
+self.dag = dag
+self.s3_bucket = BUCKET
+self.sftp_path = SFTP_PATH
+self.s3_key = S3_KEY
+
+@mock_s3
+def test_s3_to_sftp_operation(self):
+# Setting
+configuration.conf.set("core", "enable_xcom_pickling", "True")
+test_remote_file_content = \
+"This is remote file content \n which is also multiline " \
+"another line here \n this is last line. EOF"
+
+# Test for creation of s3 bucket
+conn = boto3.client('s3')
+conn.create_bucket(Bucket=self.s3_bucket)
+self.assertTrue((self.s3_hook.check_for_bucket(self.s3_bucket)))
+
+with open(LOCAL_FILE_PATH, 'w') as f:
+f.write(test_remote_file_content)
+self.s3_hook.load_file(LOCAL_FILE_PATH, self.s3_key,
bucket_name=BUCKET)
+
+# Check if object was created in s3
+objects_in_dest_bucket = conn.list_objects(Bucket=self.s3_bucket,
+ Prefix=self.s3_key)
+# there should be object found, and there should only be one object
found
+self.assertEqual(len(objects_in_dest_bucket['Contents']), 1)
+
+# the object found should be consistent with dest_key specified earlier
+self.assertEqual(objects_in_dest_bucket['Contents'][0]['Key'],
self.s3_key)
+
+# get remote file to local
+run_task = S3ToSFTPOperator(
+s3_bucket=BUCKET,
+s3_key=S3_KEY,
+sftp_path=SFTP_PATH,
+sftp_conn_id=SFTP_CONN_ID,
+s3_conn_id=S3_CONN_ID,
+task_id=TASK_ID,
+dag=self.dag
+)
+self.assertIsNotNone(run_task)
+
+run_task.execute(None)
+
+# Check that the file is created remotely
+check_file_task = SSHOperator(
+task_id="test_check_file",
+ssh_hook=self.hook,
+command="cat {0}".format(self.sftp_path),
+do_xcom_push=True,
+dag=self.dag
+)
+self.assertIsNotNone(check_file_task)
+
[GitHub] Fokko commented on a change in pull request #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators
Fokko commented on a change in pull request #3828: [AIRFLOW-2993] s3_to_sftp
and sftp_to_s3 operators
URL: https://github.com/apache/incubator-airflow/pull/3828#discussion_r224674347
##
File path: airflow/contrib/operators/s3_to_sftp_operator.py
##
@@ -0,0 +1,85 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.models import BaseOperator
+from airflow.hooks.S3_hook import S3Hook
+from airflow.contrib.hooks.ssh_hook import SSHHook
+from tempfile import NamedTemporaryFile
+from urllib.parse import urlparse
+from airflow.utils.decorators import apply_defaults
+
+
+def get_s3_key(s3_key):
+"""This parses the correct format for S3 keys
+regardless of how the S3 url is passed."""
+
+parsed_s3_key = urlparse(s3_key)
+return parsed_s3_key.path.lstrip('/')
+
+
+class S3ToSFTPOperator(BaseOperator):
+"""
+This operator enables the transferring of files from S3 to a SFTP server
+:param sftp_conn_id:The sftp connection id. The name or
+identifier for establishing a connection to the
SFTP server.
+:type sftp_conn_id: string
+:param sftp_path: The sftp remote path. This is the specified
+file path for uploading file to the SFTP server.
+:type sftp_path:string
+:param s3_conn_id: The s3 connnection id. The name or identifier for
establishing
+a connection to S3
+:type s3_conn_id: string
+:param s3_bucket: The targeted s3 bucket. This is the S3 bucket
+from where the file is downloaded.
+:type s3_bucket:string
+:param s3_key: The targeted s3 key. This is the specified file
path
+for downloading the file from S3.
+:type s3_key: string
+"""
+
+template_fields = ('s3_key', 'sftp_path')
+
+@apply_defaults
+def __init__(self,
+ s3_bucket,
+ s3_key,
+ sftp_path,
+ sftp_conn_id='ssh_default',
+ s3_conn_id='aws_default',
+ *args,
+ **kwargs):
+super(S3ToSFTPOperator, self).__init__(*args, **kwargs)
+self.sftp_conn_id = sftp_conn_id
+self.sftp_path = sftp_path
+self.s3_bucket = s3_bucket
+self.s3_key = s3_key
+self.s3_conn_id = s3_conn_id
+
+def execute(self, context):
+self.s3_key = get_s3_key(self.s3_key)
+ssh_hook = SSHHook(ssh_conn_id=self.sftp_conn_id)
+s3_hook = S3Hook(self.s3_conn_id)
+
+s3_client = s3_hook.get_conn()
+ssh_client = ssh_hook.get_conn()
Review comment:
Maybe wrap this in one line, since the `ssh_client` isn't used apart from
the next line?
`sftp_client = ssh_hook.get_conn().open_sftp()`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] Fokko commented on issue #4037: [AIRFLOW-3191] Fix not being able to specify execution_date when creating dagrun
Fokko commented on issue #4037: [AIRFLOW-3191] Fix not being able to specify execution_date when creating dagrun URL: https://github.com/apache/incubator-airflow/pull/4037#issuecomment-429207894 I find this one hard to review since I'm not super familiar with the UI. Someone else to verify this? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop
Fokko commented on a change in pull request #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop URL: https://github.com/apache/incubator-airflow/pull/3873#discussion_r224668793 ## File path: airflow/jobs.py ## @@ -358,6 +358,8 @@ def _launch_process(result_queue, :type thread_name: unicode :return: the process that was launched :rtype: multiprocessing.Process +:param zombies: zombie task instances to kill +:type zombies: SimpleTaskInstance Review comment: Nit, this should be `list[SimpleTaskInstance ]` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop
Fokko commented on a change in pull request #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop URL: https://github.com/apache/incubator-airflow/pull/3873#discussion_r217959560 ## File path: tests/jobs.py ## @@ -2160,7 +2174,7 @@ def test_scheduler_start_date(self): """ Test that the scheduler respects start_dates, even when DAGS have run """ - +session = settings.Session() Review comment: Can we do this with a `with settings.Session() as session:` statement, that will make it much cleaner and less error prone This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop
Fokko commented on a change in pull request #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop URL: https://github.com/apache/incubator-airflow/pull/3873#discussion_r224668809 ## File path: airflow/jobs.py ## @@ -304,14 +300,16 @@ class DagFileProcessor(AbstractDagFileProcessor, LoggingMixin): # Counter that increments everytime an instance of this class is created class_creation_counter = 0 -def __init__(self, file_path, pickle_dags, dag_id_white_list): +def __init__(self, file_path, pickle_dags, dag_id_white_list, zombies): """ :param file_path: a Python file containing Airflow DAG definitions :type file_path: unicode :param pickle_dags: whether to serialize the DAG objects to the DB :type pickle_dags: bool :param dag_id_whitelist: If specified, only look at these DAG ID's :type dag_id_whitelist: list[unicode] +:param zombies: zombie task instances to kill +:type zombies: SimpleTaskInstance Review comment: Nit, this should be `list[SimpleTaskInstance ]` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop
Fokko commented on a change in pull request #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop URL: https://github.com/apache/incubator-airflow/pull/3873#discussion_r224670598 ## File path: tests/jobs.py ## @@ -1264,6 +1262,7 @@ def setUp(self): session.query(models.DagRun).delete() session.query(models.ImportError).delete() session.commit() +session.close() Review comment: Same as above. Can we do this with a with settings.Session() as session: statement, that will make it much cleaner and less error prone This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4038: [AIRFLOW-1970] Let empty Fernet key or special `no encryption` phrase.
codecov-io commented on issue #4038: [AIRFLOW-1970] Let empty Fernet key or special `no encryption` phrase. URL: https://github.com/apache/incubator-airflow/pull/4038#issuecomment-429203577 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr=h1) Report > Merging [#4038](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7115883fafca912e46ef0d67f69320f57305bbb3?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `70%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4038 +/- ## == + Coverage 75.77% 75.77% +<.01% == Files 199 199 Lines 1594615951 +5 == + Hits1208312087 +4 - Misses 3863 3864 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4038/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.68% <70%> (-0.03%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr=footer). Last update [7115883...d0e95b6](https://codecov.io/gh/apache/incubator-airflow/pull/4038?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3192) Remove deprecated post_execute logic
[
https://issues.apache.org/jira/browse/AIRFLOW-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647458#comment-16647458
]
ASF GitHub Bot commented on AIRFLOW-3192:
-
Fokko opened a new pull request #4040: [AIRFLOW-3192] Remove deprecated
post_execute logic
URL: https://github.com/apache/incubator-airflow/pull/4040
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-3192\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3192
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-3192\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Remove deprecated post_execute logic
>
>
> Key: AIRFLOW-3192
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3192
> Project: Apache Airflow
> Issue Type: Improvement
> Components: models
>Affects Versions: 1.10.0
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
> Fix For: 2.0.0
>
>
> Remove the deprecated post_execute logic for Airflow 2.0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] Fokko opened a new pull request #4040: [AIRFLOW-3192] Remove deprecated post_execute logic
Fokko opened a new pull request #4040: [AIRFLOW-3192] Remove deprecated
post_execute logic
URL: https://github.com/apache/incubator-airflow/pull/4040
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-3192\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3192
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-3192\], code changes always need a Jira issue.
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] codecov-io edited a comment on issue #4039: [AIRFLOW-3046] Fix ECS Operator mistakenly reports success
codecov-io edited a comment on issue #4039: [AIRFLOW-3046] Fix ECS Operator mistakenly reports success URL: https://github.com/apache/incubator-airflow/pull/4039#issuecomment-429179730 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=h1) Report > Merging [#4039](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7115883fafca912e46ef0d67f69320f57305bbb3?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4039 +/- ## === Coverage 75.77% 75.77% === Files 199 199 Lines 1594615946 === Hits1208312083 Misses 3863 3863 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=footer). Last update [7115883...d44c828](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #4015: [AIRFLOW-2789] Create single node DataProc cluster
Fokko commented on a change in pull request #4015: [AIRFLOW-2789] Create single node DataProc cluster URL: https://github.com/apache/incubator-airflow/pull/4015#discussion_r224668410 ## File path: airflow/contrib/operators/dataproc_operator.py ## @@ -170,6 +172,7 @@ def __init__(self, idle_delete_ttl=None, auto_delete_time=None, auto_delete_ttl=None, + single_node=False, Review comment: Right now if feels a bit redundant te me. In this case you would need to set both the number of workers, and the `single_node` to True. Since the number of workers is a required argument, I'd say that setting `num_workers` would already suggest that you want a single node cluster. I'd set the `single_node = num_workers == 0` to still be explicit in the code, but having a less redundant interface to the developer. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #4015: [AIRFLOW-2789] Create single node DataProc cluster
Fokko commented on a change in pull request #4015: [AIRFLOW-2789] Create single node DataProc cluster URL: https://github.com/apache/incubator-airflow/pull/4015#discussion_r224668060 ## File path: airflow/contrib/operators/dataproc_operator.py ## @@ -205,10 +208,19 @@ def __init__(self, self.idle_delete_ttl = idle_delete_ttl self.auto_delete_time = auto_delete_time self.auto_delete_ttl = auto_delete_ttl +self.single_node = single_node assert not (self.custom_image and self.image_version), \ "custom_image and image_version can't be both set" +assert ( +self.single_node and +self.num_preemptible_workers + self.num_workers == 0 +) or ( +not self.single_node and +self.num_preemptible_workers + self.num_workers != 0 +) Review comment: Maybe add a helpful message for the end user here? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #4038: [AIRFLOW-1970] Let empty Fernet key or special `no encryption` phrase.
Fokko commented on a change in pull request #4038: [AIRFLOW-1970] Let empty Fernet key or special `no encryption` phrase. URL: https://github.com/apache/incubator-airflow/pull/4038#discussion_r224667522 ## File path: airflow/models.py ## @@ -150,6 +150,10 @@ def get_fernet(): :raises: AirflowException if there's a problem trying to load Fernet """ global _fernet +log = LoggingMixin().log +no_encryption_phrase = \ Review comment: Why this `no_encryption_phrase`. If you don't want to use Fernet encryption, omitting the key would make more sense to me, instead of setting it to a value like `cryptography_not_found_storing_passwords_in_plain_text`. Maybe get rid of this logic? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links.
Fokko commented on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links. URL: https://github.com/apache/incubator-airflow/pull/4036#issuecomment-429199442 @jgao54 Any thoughts on this? :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links.
Fokko commented on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links. URL: https://github.com/apache/incubator-airflow/pull/4036#issuecomment-429199045 @oliviersm199 It could be that you've worked from an old master. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on issue #4039: [AIRFLOW-3046] Fix ECS Operator mistakenly reports success
Fokko commented on issue #4039: [AIRFLOW-3046] Fix ECS Operator mistakenly reports success URL: https://github.com/apache/incubator-airflow/pull/4039#issuecomment-429198663 Good point @danmactough. Thanks for the explanaition, like I said. I'm not an AWS guru :-) Will merge on a green CI. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] danmactough commented on issue #4039: [AIRFLOW-3046] Fix ECS Operator mistakenly reports success
danmactough commented on issue #4039: [AIRFLOW-3046] Fix ECS Operator
mistakenly reports success
URL:
https://github.com/apache/incubator-airflow/pull/4039#issuecomment-429198267
> If this response is not successful, then mark the task as failed. What do
you think @danmactough ?
I _think_ this is the only possible way, @Fokko, with the responses we get
from AWS. The response we get doesn't have any failures, doesn't have a
non-zero exitCode (it has no exitCode), and there's no explicit error. Here's a
sample response:
```
[2018-09-12 01:02:02,712] {ecs_operator.py:112} INFO - ECS Task stopped,
check status: {
'tasks': [
{
'taskArn':
'arn:aws:ecs:us-east-1::task/32d43a1d-fbc7-4659-815d-9133bde11cdc',
'clusterArn': 'arn:aws:ecs:us-east-1::cluster/processing',
'taskDefinitionArn':
'arn:aws:ecs:us-east-1::task-definition/foobar-testing_dataEngineering_rd:76',
'containerInstanceArn':
'arn:aws:ecs:us-east-1::container-instance/7431f0a6-8fc5-4eff-8196-32f77d286a61',
'overrides': {
'containerOverrides': [
{
'name': 'foobar-testing',
'command': [
'./bin/generate-features.sh',
'2018-09-11'
]
}
]
},
'lastStatus': 'STOPPED',
'desiredStatus': 'STOPPED',
'cpu': '4096',
'memory': '6',
'containers': [
{
'containerArn':
'arn:aws:ecs:us-east-1::container/0d5cc553-f894-4f9a-b17c-9f80f7ce8d0a',
'taskArn':
'arn:aws:ecs:us-east-1::task/32d43a1d-fbc7-4659-815d-9133bde11cdc',
'name': 'foobar-testing',
'lastStatus': 'RUNNING',
'networkBindings': [],
'networkInterfaces': [],
'healthStatus': 'UNKNOWN'
}
],
'startedBy': 'Airflow',
'version': 3,
'stoppedReason': 'Host EC2 (instance i-02cf23bbd5ae26194)
terminated.',
'connectivity': 'CONNECTED',
'connectivityAt': datetime.datetime(2018, 9, 12, 0, 6, 30, 245000,
tzinfo=tzlocal()),
'pullStartedAt': datetime.datetime(2018, 9, 12, 0, 6, 32, 748000,
tzinfo=tzlocal()),
'pullStoppedAt': datetime.datetime(2018, 9, 12, 0, 6, 59, 748000,
tzinfo=tzlocal()),
'createdAt': datetime.datetime(2018, 9, 12, 0, 6, 30, 245000,
tzinfo=tzlocal()),
'startedAt': datetime.datetime(2018, 9, 12, 0, 7, 0, 748000,
tzinfo=tzlocal()),
'stoppingAt': datetime.datetime(2018, 9, 12, 1, 2, 0, 91000,
tzinfo=tzlocal()),
'stoppedAt': datetime.datetime(2018, 9, 12, 1, 2, 0, 91000,
tzinfo=tzlocal()),
'group': 'family:foobar-testing_dataEngineering_rd',
'launchType': 'EC2',
'attachments': [],
'healthStatus': 'UNKNOWN'
}
],
'failures': [],
'ResponseMetadata': {
'RequestId': '758c791f-b627-11e8-83f7-2b76f4796ed2',
'HTTPStatusCode': 200,
'HTTPHeaders': {
'server': 'Server',
'date': 'Wed, 12 Sep 2018 01:02:02 GMT',
'content-type': 'application/x-amz-json-1.1',
'content-length': '1412',
'connection': 'keep-alive',
'x-amzn-requestid': '758c791f-b627-11e8-83f7-2b76f4796ed2'
},
'RetryAttempts': 0
}
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] danmactough commented on a change in pull request #4039: [AIRFLOW-3046] Fix ECS Operator mistakenly reports success
danmactough commented on a change in pull request #4039: [AIRFLOW-3046] Fix ECS
Operator mistakenly reports success
URL: https://github.com/apache/incubator-airflow/pull/4039#discussion_r224666163
##
File path: airflow/contrib/operators/ecs_operator.py
##
@@ -139,6 +140,10 @@ def _check_success_task(self):
raise AirflowException(response)
for task in response['tasks']:
+if re.match(r'Host .+?\(.+?\) terminated\.',
task.get('stoppedReason', '')):
Review comment:
> I would like to edit that check to match "terminated" or "stopped" though,
having seen that documentation, now.
Amended with that change and added a comment as you suggested, as well,
@Fokko
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#discussion_r224665899 ## File path: airflow/contrib/operators/bigquery_get_data.py ## @@ -78,6 +80,7 @@ def __init__(self, selected_fields=None, bigquery_conn_id='bigquery_default', delegate_to=None, + do_xcom_push=True, Review comment: @XD-DENG Thanks for checking, lets go for `False` then. It makes also more sense to me since it might clutter up the Database quickly. We should still add a line to `Updating.md` and target the fix for Airflow 2.0 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3192) Remove deprecated post_execute logic
Fokko Driesprong created AIRFLOW-3192: - Summary: Remove deprecated post_execute logic Key: AIRFLOW-3192 URL: https://issues.apache.org/jira/browse/AIRFLOW-3192 Project: Apache Airflow Issue Type: Improvement Components: models Affects Versions: 1.10.0 Reporter: Fokko Driesprong Assignee: Fokko Driesprong Fix For: 2.0.0 Remove the deprecated post_execute logic for Airflow 2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#discussion_r224665405 ## File path: airflow/contrib/operators/bigquery_get_data.py ## @@ -78,6 +80,7 @@ def __init__(self, selected_fields=None, bigquery_conn_id='bigquery_default', delegate_to=None, + do_xcom_push=True, Review comment: @XD-DENG I fully agree. It would be even nicer to add the logic somewhere around here: https://github.com/apache/incubator-airflow/blob/b85210d8596c1e05fbf3da2941aaca8170ae871e/airflow/models.py#L1643 Instead of duplicating it in every operator. This makes it very easy for newcomers to forget about xcom. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#discussion_r224664481 ## File path: airflow/contrib/operators/gcs_to_bq.py ## @@ -248,7 +252,7 @@ def execute(self, context): time_partitioning=self.time_partitioning, cluster_fields=self.cluster_fields) -if self.max_id_key: +if self.do_xcom_push and self.max_id_key: Review comment: I'm not really sure if the `do_xcom_push` flag adds a lot of value here. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#discussion_r224664329 ## File path: airflow/contrib/operators/dataproc_operator.py ## @@ -394,7 +394,7 @@ def execute(self, context): self.cluster_name ) self._wait_for_done(service) -return True +return None Review comment: Change this to `return` without the `None` please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#discussion_r224664795 ## File path: airflow/contrib/operators/ssh_operator.py ## @@ -45,7 +45,7 @@ class SSHOperator(BaseOperator): :type command: str :param timeout: timeout (in seconds) for executing the command. :type timeout: int -:param do_xcom_push: return the stdout which also get set in xcom by airflow platform +:param do_xcom_push: return the stdout which also get set in XCOM Review comment: Nice one This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
XD-DENG commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#discussion_r224664921 ## File path: airflow/contrib/operators/bigquery_get_data.py ## @@ -78,6 +80,7 @@ def __init__(self, selected_fields=None, bigquery_conn_id='bigquery_default', delegate_to=None, + do_xcom_push=True, Review comment: Did another quick check: seems default value of most existing `xcom_push`/`do_xcom_push` arguments are `False`. We should make all of them consistent no matter we decide to use `True` or `False`. BTW, now I understand why @itscaro put `do_xcom_push` as the variable name since it is used in some of the operators. This may cause confusion. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#discussion_r224664588 ## File path: airflow/contrib/operators/gcs_to_s3.py ## @@ -88,11 +91,17 @@ def __init__(self, self.dest_s3_key = dest_s3_key self.dest_verify = dest_verify self.replace = replace +self.do_xcom_push = do_xcom_push def execute(self, context): +# this operator calls super and uses the returned value, so we set do_xcom_push to Review comment: This is nasty, we should be able to return this, or pass the `do_xcom_push` to the super. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#discussion_r224664350 ## File path: airflow/contrib/operators/dataproc_operator.py ## @@ -412,7 +412,7 @@ def execute(self, context): self.cluster_name ) self._wait_for_done(service) -return True +return None Review comment: Change this to `return` without the `None` please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#discussion_r224664695 ## File path: airflow/contrib/operators/mlengine_operator.py ## @@ -387,6 +400,9 @@ class MLEngineVersionOperator(BaseOperator): For this to work, the service account making the request must have domain-wide delegation enabled. :type delegate_to: str + Review comment: Trim this newline please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#discussion_r224664626 ## File path: airflow/contrib/operators/jenkins_job_trigger_operator.py ## @@ -243,7 +247,9 @@ def execute(self, context): 'this exception for unknown parameters' 'You can also check logs for more details on this exception ' '(jenkins_url/log/rss)', str(err)) -if build_info: -# If we can we return the url of the job -# for later use (like retrieving an artifact) -return build_info['url'] + +if self.do_xcom_push: Review comment: Please collapse this into a single if `if self.do_xcom_push && build_info` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#discussion_r224664655 ## File path: airflow/contrib/operators/mlengine_operator.py ## @@ -151,6 +151,9 @@ class MLEngineBatchPredictionOperator(BaseOperator): have doamin-wide delegation enabled. :type delegate_to: str Review comment: Trim this newline please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] XD-DENG commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
XD-DENG commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#discussion_r224664461 ## File path: airflow/contrib/operators/bigquery_get_data.py ## @@ -78,6 +80,7 @@ def __init__(self, selected_fields=None, bigquery_conn_id='bigquery_default', delegate_to=None, + do_xcom_push=True, Review comment: Personally I think `False` is making more sense. Was this discussed by other folks? In addition, if that's the case and we go for `True`, other changes would be needed, for example, the default value of `xcom_push` in [`BashOperator`](https://github.com/apache/incubator-airflow/blob/master/airflow/operators/bash_operator.py) is `False` currently. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
Fokko commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#discussion_r224664023 ## File path: airflow/contrib/operators/bigquery_get_data.py ## @@ -78,6 +80,7 @@ def __init__(self, selected_fields=None, bigquery_conn_id='bigquery_default', delegate_to=None, + do_xcom_push=True, Review comment: `True` is default behaviour, so then we need to add a line to the `Updating.md` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] danmactough commented on a change in pull request #4039: [AIRFLOW-3046] Fix ECS Operator mistakenly reports success
danmactough commented on a change in pull request #4039: [AIRFLOW-3046] Fix ECS
Operator mistakenly reports success
URL: https://github.com/apache/incubator-airflow/pull/4039#discussion_r224664053
##
File path: airflow/contrib/operators/ecs_operator.py
##
@@ -139,6 +140,10 @@ def _check_success_task(self):
raise AirflowException(response)
for task in response['tasks']:
+if re.match(r'Host .+?\(.+?\) terminated\.',
task.get('stoppedReason', '')):
Review comment:
> Do you know what other `stoppedReasons` might be?
That's a great question. The [only documentation I can
find](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/stopped-task-errors.html)
identifies "**_some_** possible reasons and their explanations," which doesn't
give a lot of confidence that the list is complete.
I would like to edit that check to match "terminated" or "stopped" though,
having seen that documentation, now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] Fokko commented on a change in pull request #4039: [AIRFLOW-3046] Fix ECS Operator mistakenly reports success
Fokko commented on a change in pull request #4039: [AIRFLOW-3046] Fix ECS
Operator mistakenly reports success
URL: https://github.com/apache/incubator-airflow/pull/4039#discussion_r224663367
##
File path: airflow/contrib/operators/ecs_operator.py
##
@@ -139,6 +140,10 @@ def _check_success_task(self):
raise AirflowException(response)
for task in response['tasks']:
+if re.match(r'Host .+?\(.+?\) terminated\.',
task.get('stoppedReason', '')):
Review comment:
Maybe add a comment here to elaborate where this regex matches on. Do you
know what other `stoppedReasons` might be? Just to be sure that it doesn't
conflict with other reasons.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] codecov-io edited a comment on issue #4039: [AIRFLOW-3046] Fix ECS Operator mistakenly reports success
codecov-io edited a comment on issue #4039: [AIRFLOW-3046] Fix ECS Operator mistakenly reports success URL: https://github.com/apache/incubator-airflow/pull/4039#issuecomment-429179730 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=h1) Report > Merging [#4039](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7115883fafca912e46ef0d67f69320f57305bbb3?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4039 +/- ## == - Coverage 75.77% 75.76% -0.01% == Files 199 199 Lines 1594615946 == - Hits1208312082 -1 - Misses 3863 3864 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4039/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.67% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=footer). Last update [7115883...6437400](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4039: [AIRFLOW-3046] Fix ECS Operator mistakenly reports success
codecov-io edited a comment on issue #4039: [AIRFLOW-3046] Fix ECS Operator mistakenly reports success URL: https://github.com/apache/incubator-airflow/pull/4039#issuecomment-429179730 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=h1) Report > Merging [#4039](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7115883fafca912e46ef0d67f69320f57305bbb3?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4039 +/- ## == - Coverage 75.77% 75.76% -0.01% == Files 199 199 Lines 1594615946 == - Hits1208312082 -1 - Misses 3863 3864 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4039/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.67% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=footer). Last update [7115883...6437400](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4039: [AIRFLOW-3046] Fix ECS Operator mistakenly reports success
codecov-io commented on issue #4039: [AIRFLOW-3046] Fix ECS Operator mistakenly reports success URL: https://github.com/apache/incubator-airflow/pull/4039#issuecomment-429179730 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=h1) Report > Merging [#4039](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7115883fafca912e46ef0d67f69320f57305bbb3?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4039 +/- ## === Coverage 75.77% 75.77% === Files 199 199 Lines 1594615946 === Hits1208312083 Misses 3863 3863 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=footer). Last update [7115883...9b343ee](https://codecov.io/gh/apache/incubator-airflow/pull/4039?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3046) ECS Operator mistakenly reports success when task is killed due to EC2 host termination
[
https://issues.apache.org/jira/browse/AIRFLOW-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647276#comment-16647276
]
ASF GitHub Bot commented on AIRFLOW-3046:
-
danmactough opened a new pull request #4039: [AIRFLOW-3046] Fix ECS Operator
mistakenly reports success
URL: https://github.com/apache/incubator-airflow/pull/4039
Supercedes #3938
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-3046) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3046
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
We have ECS clusters made up of EC2 spot fleets. Among other things, this
means hosts can be terminated on short notice. When this happens, all tasks
(and associated containers) get terminated, as well.
We expect that when that happens for Airflow task instances using the ECS
Operator, those instances will be marked as failures and retried.
Instead, they are marked as successful.
As a result, the immediate downstream task fails, causing the scheduled DAG
run to fail.
Here's an example of the Airflow log output when this happens:
```
[2018-09-12 01:02:02,712] {ecs_operator.py:112} INFO - ECS Task stopped,
check status: {'tasks': [{'taskArn':
'arn:aws:ecs:us-east-1::task/32d43a1d-fbc7-4659-815d-9133bde11cdc',
'clusterArn': 'arn:aws:ecs:us-east-1::cluster/processing',
'taskDefinitionArn':
'arn:aws:ecs:us-east-1::task-definition/foobar-testing_dataEngineering_rd:76',
'containerInstanceArn':
'arn:aws:ecs:us-east-1::container-instance/7431f0a6-8fc5-4eff-8196-32f77d286a61',
'overrides': {'containerOverrides': [{'name': 'foobar-testing', 'command':
['./bin/generate-features.sh', '2018-09-11']}]}, 'lastStatus': 'STOPPED',
'desiredStatus': 'STOPPED', 'cpu': '4096', 'memory': '6', 'containers':
[{'containerArn':
'arn:aws:ecs:us-east-1::container/0d5cc553-f894-4f9a-b17c-9f80f7ce8d0a',
'taskArn':
'arn:aws:ecs:us-east-1::task/32d43a1d-fbc7-4659-815d-9133bde11cdc',
'name': 'foobar-testing', 'lastStatus': 'RUNNING', 'networkBindings': [],
'networkInterfaces': [], 'healthStatus': 'UNKNOWN'}], 'startedBy': 'Airflow',
'version': 3, 'stoppedReason': 'Host EC2 (instance i-02cf23bbd5ae26194)
terminated.', 'connectivity': 'CONNECTED', 'connectivityAt':
datetime.datetime(2018, 9, 12, 0, 6, 30, 245000, tzinfo=tzlocal()),
'pullStartedAt': datetime.datetime(2018, 9, 12, 0, 6, 32, 748000,
tzinfo=tzlocal()), 'pullStoppedAt': datetime.datetime(2018, 9, 12, 0, 6, 59,
748000, tzinfo=tzlocal()), 'createdAt': datetime.datetime(2018, 9, 12, 0, 6,
30, 245000, tzinfo=tzlocal()), 'startedAt': datetime.datetime(2018, 9, 12, 0,
7, 0, 748000, tzinfo=tzlocal()), 'stoppingAt': datetime.datetime(2018, 9, 12,
1, 2, 0, 91000, tzinfo=tzlocal()), 'stoppedAt': datetime.datetime(2018, 9, 12,
1, 2, 0, 91000, tzinfo=tzlocal()), 'group':
'family:foobar-testing_dataEngineering_rd', 'launchType': 'EC2', 'attachments':
[], 'healthStatus': 'UNKNOWN'}], 'failures': [], 'ResponseMetadata':
{'RequestId': '758c791f-b627-11e8-83f7-2b76f4796ed2', 'HTTPStatusCode': 200,
'HTTPHeaders': {'server': 'Server', 'date': 'Wed, 12 Sep 2018 01:02:02 GMT',
'content-type': 'application/x-amz-json-1.1', 'content-length': '1412',
'connection': 'keep-alive', 'x-amzn-requestid':
'758c791f-b627-11e8-83f7-2b76f4796ed2'}, 'RetryAttempts': 0}}
```
This PR updates the ECS Operator to include a check for this failure
condition and raise an `AirflowException`.
I've also updated the `setup.py` to include `unittest2` in the `devel`
section when using Python 2. The tests depend on `unittest2` being installed.
### Tests
- [x] My PR adds the following unit tests
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use
[GitHub] XD-DENG commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
XD-DENG commented on a change in pull request #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#discussion_r223653754 ## File path: airflow/contrib/operators/bigquery_get_data.py ## @@ -78,6 +80,7 @@ def __init__(self, selected_fields=None, bigquery_conn_id='bigquery_default', delegate_to=None, + do_xcom_push=True, Review comment: Hi @itscaro , two comments: 1. It would be better to have consistent argument naming. In other core operators, like `BashOperator`, this argument is called `xcom_push` rather than `do_xcom_push`. 2. I believe the default value should be `False` rather than `True`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] oliviersm199 commented on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links.
oliviersm199 commented on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links. URL: https://github.com/apache/incubator-airflow/pull/4036#issuecomment-429165375 not sure why the code coverage has changed so drastically, I added some test cases but it looks like the hits and misses are changing abnormally? Did I miss something? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] itscaro commented on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
itscaro commented on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#issuecomment-429158851 Everything is good. There is just one question about naming the variable. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-1970) Database cannot be initialized if an invalid fernet key is provided
[
https://issues.apache.org/jira/browse/AIRFLOW-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647136#comment-16647136
]
ASF GitHub Bot commented on AIRFLOW-1970:
-
exploy opened a new pull request #4038: [AIRFLOW-1970] Let empty Fernet key or
special `no encryption` phrase.
URL: https://github.com/apache/incubator-airflow/pull/4038
### Jira
My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-1970) issues and references
them in the PR title.
### Description
Once the user has installed Fernet package then the application enforces
setting valid Fernet key.
This change will alter this behavior into letting empty Fernet key or
special `no encryption` phrase and interpreting those two cases as no
encryption desirable.
### Tests
My PR adds no tests as of now.
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
### Code Quality
- [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Database cannot be initialized if an invalid fernet key is provided
> ---
>
> Key: AIRFLOW-1970
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1970
> Project: Apache Airflow
> Issue Type: Bug
> Components: configuration
>Affects Versions: 1.9.0
> Environment: Python 2.7.12
> PostgreSQL 9.6.3
>Reporter: Michael Backes
>Assignee: Jarosław Śmietanka
>Priority: Major
>
> If I use an invalid fernet key in my config file, I'm not able to run
> "airflow initdb" successfully.
> For example if I have the following in my config:
> {panel:title=airflow.cfg}
> ...
> \\# Secret key to save connection passwords in the db
> fernet_key = xxx
> ...
> {panel}
> I will get the following error when running "airflow initdb":
> {panel:title=log}
> [2018-01-05 16:43:00,525] {__init__.py:45} INFO - Using executor LocalExecutor
> DB: postgresql+psycopg2://airflow_user:secret_pw@some.address:5432/airflow
> [2018-01-05 16:43:00,624] {db.py:312} INFO - Creating tables
> INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
> INFO [alembic.runtime.migration] Will assume transactional DDL.
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 27, in
> args.func(args)
> File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 897,
> in initdb
> db_utils.initdb()
> File "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line
> 114, in initdb
> schema='airflow_ci'))
> File "", line 4, in __init__
> File "/usr/local/lib64/python2.7/site-packages/sqlalchemy/orm/state.py",
> line 414, in _initialize_instance
> manager.dispatch.init_failure(self, args, kwargs)
> File
> "/usr/local/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> compat.reraise(exc_type, exc_value, exc_tb)
> File "/usr/local/lib64/python2.7/site-packages/sqlalchemy/orm/state.py",
> line 411, in _initialize_instance
> return manager.original_init(*mixed[1:], **kwargs)
> File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 578,
> in __init__
> self.extra = extra
> File "", line 1, in __set__
> File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 639,
> in set_extra
> fernet = get_fernet()
> File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 105,
> in get_fernet
> return Fernet(configuration.get('core', 'FERNET_KEY').encode('utf-8'))
> File "/usr/local/lib64/python2.7/site-packages/cryptography/fernet.py",
> line 34, in __init__
> key = base64.urlsafe_b64decode(key)
> File "/usr/lib64/python2.7/base64.py", line 119, in urlsafe_b64decode
> return b64decode(s.translate(_urlsafe_decode_translation))
> File "/usr/lib64/python2.7/base64.py", line 78, in b64decode
> raise TypeError(msg)
> TypeError: Incorrect padding
> {panel}
> I also got an error when I try to add extras to a connection, if fernet_key
> is empty in the config file. The error message was "Incorrect padding". Once
> I provided a valid key generated with the instructions given
> [here|http://airflow.readthedocs.io/en/latest/configuration.html?highlight=fernet#connections]
> and
[GitHub] exploy opened a new pull request #4038: [AIRFLOW-1970] Let empty Fernet key or special `no encryption` phrase.
exploy opened a new pull request #4038: [AIRFLOW-1970] Let empty Fernet key or special `no encryption` phrase. URL: https://github.com/apache/incubator-airflow/pull/4038 ### Jira My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW-1970) issues and references them in the PR title. ### Description Once the user has installed Fernet package then the application enforces setting valid Fernet key. This change will alter this behavior into letting empty Fernet key or special `no encryption` phrase and interpreting those two cases as no encryption desirable. ### Tests My PR adds no tests as of now. ### Commits - [x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": ### Code Quality - [ ] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-1970) Database cannot be initialized if an invalid fernet key is provided
[
https://issues.apache.org/jira/browse/AIRFLOW-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarosław Śmietanka reassigned AIRFLOW-1970:
---
Assignee: Jarosław Śmietanka
> Database cannot be initialized if an invalid fernet key is provided
> ---
>
> Key: AIRFLOW-1970
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1970
> Project: Apache Airflow
> Issue Type: Bug
> Components: configuration
>Affects Versions: 1.9.0
> Environment: Python 2.7.12
> PostgreSQL 9.6.3
>Reporter: Michael Backes
>Assignee: Jarosław Śmietanka
>Priority: Major
>
> If I use an invalid fernet key in my config file, I'm not able to run
> "airflow initdb" successfully.
> For example if I have the following in my config:
> {panel:title=airflow.cfg}
> ...
> \\# Secret key to save connection passwords in the db
> fernet_key = xxx
> ...
> {panel}
> I will get the following error when running "airflow initdb":
> {panel:title=log}
> [2018-01-05 16:43:00,525] {__init__.py:45} INFO - Using executor LocalExecutor
> DB: postgresql+psycopg2://airflow_user:secret_pw@some.address:5432/airflow
> [2018-01-05 16:43:00,624] {db.py:312} INFO - Creating tables
> INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
> INFO [alembic.runtime.migration] Will assume transactional DDL.
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 27, in
> args.func(args)
> File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 897,
> in initdb
> db_utils.initdb()
> File "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line
> 114, in initdb
> schema='airflow_ci'))
> File "", line 4, in __init__
> File "/usr/local/lib64/python2.7/site-packages/sqlalchemy/orm/state.py",
> line 414, in _initialize_instance
> manager.dispatch.init_failure(self, args, kwargs)
> File
> "/usr/local/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> compat.reraise(exc_type, exc_value, exc_tb)
> File "/usr/local/lib64/python2.7/site-packages/sqlalchemy/orm/state.py",
> line 411, in _initialize_instance
> return manager.original_init(*mixed[1:], **kwargs)
> File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 578,
> in __init__
> self.extra = extra
> File "", line 1, in __set__
> File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 639,
> in set_extra
> fernet = get_fernet()
> File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 105,
> in get_fernet
> return Fernet(configuration.get('core', 'FERNET_KEY').encode('utf-8'))
> File "/usr/local/lib64/python2.7/site-packages/cryptography/fernet.py",
> line 34, in __init__
> key = base64.urlsafe_b64decode(key)
> File "/usr/lib64/python2.7/base64.py", line 119, in urlsafe_b64decode
> return b64decode(s.translate(_urlsafe_decode_translation))
> File "/usr/lib64/python2.7/base64.py", line 78, in b64decode
> raise TypeError(msg)
> TypeError: Incorrect padding
> {panel}
> I also got an error when I try to add extras to a connection, if fernet_key
> is empty in the config file. The error message was "Incorrect padding". Once
> I provided a valid key generated with the instructions given
> [here|http://airflow.readthedocs.io/en/latest/configuration.html?highlight=fernet#connections]
> and restarted all of the airflow services it worked without any issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1970) Database cannot be initialized if an invalid fernet key is provided
[
https://issues.apache.org/jira/browse/AIRFLOW-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647090#comment-16647090
]
Jarosław Śmietanka commented on AIRFLOW-1970:
-
I've also got the same issue.
And application forces me to set encryption key only because I have a package
installed?
If I have set `fernet_key` to an empty value, then my understanding would be
that application follows my orders and does no encryption.
This case is especially valid to me while running tests (development process).
> Database cannot be initialized if an invalid fernet key is provided
> ---
>
> Key: AIRFLOW-1970
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1970
> Project: Apache Airflow
> Issue Type: Bug
> Components: configuration
>Affects Versions: 1.9.0
> Environment: Python 2.7.12
> PostgreSQL 9.6.3
>Reporter: Michael Backes
>Priority: Major
>
> If I use an invalid fernet key in my config file, I'm not able to run
> "airflow initdb" successfully.
> For example if I have the following in my config:
> {panel:title=airflow.cfg}
> ...
> \\# Secret key to save connection passwords in the db
> fernet_key = xxx
> ...
> {panel}
> I will get the following error when running "airflow initdb":
> {panel:title=log}
> [2018-01-05 16:43:00,525] {__init__.py:45} INFO - Using executor LocalExecutor
> DB: postgresql+psycopg2://airflow_user:secret_pw@some.address:5432/airflow
> [2018-01-05 16:43:00,624] {db.py:312} INFO - Creating tables
> INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
> INFO [alembic.runtime.migration] Will assume transactional DDL.
> Traceback (most recent call last):
> File "/usr/local/bin/airflow", line 27, in
> args.func(args)
> File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 897,
> in initdb
> db_utils.initdb()
> File "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line
> 114, in initdb
> schema='airflow_ci'))
> File "", line 4, in __init__
> File "/usr/local/lib64/python2.7/site-packages/sqlalchemy/orm/state.py",
> line 414, in _initialize_instance
> manager.dispatch.init_failure(self, args, kwargs)
> File
> "/usr/local/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py",
> line 66, in __exit__
> compat.reraise(exc_type, exc_value, exc_tb)
> File "/usr/local/lib64/python2.7/site-packages/sqlalchemy/orm/state.py",
> line 411, in _initialize_instance
> return manager.original_init(*mixed[1:], **kwargs)
> File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 578,
> in __init__
> self.extra = extra
> File "", line 1, in __set__
> File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 639,
> in set_extra
> fernet = get_fernet()
> File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 105,
> in get_fernet
> return Fernet(configuration.get('core', 'FERNET_KEY').encode('utf-8'))
> File "/usr/local/lib64/python2.7/site-packages/cryptography/fernet.py",
> line 34, in __init__
> key = base64.urlsafe_b64decode(key)
> File "/usr/lib64/python2.7/base64.py", line 119, in urlsafe_b64decode
> return b64decode(s.translate(_urlsafe_decode_translation))
> File "/usr/lib64/python2.7/base64.py", line 78, in b64decode
> raise TypeError(msg)
> TypeError: Incorrect padding
> {panel}
> I also got an error when I try to add extras to a connection, if fernet_key
> is empty in the config file. The error message was "Incorrect padding". Once
> I provided a valid key generated with the instructions given
> [here|http://airflow.readthedocs.io/en/latest/configuration.html?highlight=fernet#connections]
> and restarted all of the airflow services it worked without any issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] exploy commented on a change in pull request #4015: [AIRFLOW-2789] Create single node DataProc cluster
exploy commented on a change in pull request #4015: [AIRFLOW-2789] Create single node DataProc cluster URL: https://github.com/apache/incubator-airflow/pull/4015#discussion_r224614716 ## File path: airflow/contrib/operators/dataproc_operator.py ## @@ -170,6 +172,7 @@ def __init__(self, idle_delete_ttl=None, auto_delete_time=None, auto_delete_ttl=None, + single_node=False, Review comment: I based the behavior on how the `gcloud dataproc clusters create` command works. While the cluster is in a single node configuration, user has to keep in mind that resouces avaiable to the spark are not the same as in case of regular workers - single node acts both as a worker and as the master, so part of the resources is consumed by master process. And last but not least, I prefer to follow the rule ["Explicit is better than implicit"](https://www.python.org/dev/peps/pep-0020/). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] KevinYang21 commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop
KevinYang21 commented on issue #3873: [Airflow-2760] Decouple DAG parsing loop from scheduler loop URL: https://github.com/apache/incubator-airflow/pull/3873#issuecomment-429124129 @aoen @Fokko @bolkedebruin @mistercrunch @kaxil any love for this old PR? quite a big PR and thus rebase experience is not very ideal :P This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3919: [AIRFLOW-3089] Drop hard-coded url scheme in google auth redirect.
codecov-io edited a comment on issue #3919: [AIRFLOW-3089] Drop hard-coded url scheme in google auth redirect. URL: https://github.com/apache/incubator-airflow/pull/3919#issuecomment-422879617 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3919?src=pr=h1) Report > Merging [#3919](https://codecov.io/gh/apache/incubator-airflow/pull/3919?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7115883fafca912e46ef0d67f69320f57305bbb3?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3919?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3919 +/- ## == - Coverage 75.77% 75.76% -0.01% == Files 199 199 Lines 1594615946 == - Hits1208312082 -1 - Misses 3863 3864 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3919?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3919/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.67% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3919?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3919?src=pr=footer). Last update [7115883...862bd59](https://codecov.io/gh/apache/incubator-airflow/pull/3919?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3919: [AIRFLOW-3089] Drop hard-coded url scheme in google auth redirect.
codecov-io edited a comment on issue #3919: [AIRFLOW-3089] Drop hard-coded url scheme in google auth redirect. URL: https://github.com/apache/incubator-airflow/pull/3919#issuecomment-422879617 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3919?src=pr=h1) Report > Merging [#3919](https://codecov.io/gh/apache/incubator-airflow/pull/3919?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7115883fafca912e46ef0d67f69320f57305bbb3?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3919?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3919 +/- ## == - Coverage 75.77% 75.76% -0.01% == Files 199 199 Lines 1594615946 == - Hits1208312082 -1 - Misses 3863 3864 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3919?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3919/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.67% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3919?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3919?src=pr=footer). Last update [7115883...862bd59](https://codecov.io/gh/apache/incubator-airflow/pull/3919?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2976) pipy docker dependency broken
[ https://issues.apache.org/jira/browse/AIRFLOW-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646978#comment-16646978 ] Matthew Donovan commented on AIRFLOW-2976: -- The cpu_shares and mem_limit keyword arguments were both moved into the host_config argument. > pipy docker dependency broken > - > > Key: AIRFLOW-2976 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2976 > Project: Apache Airflow > Issue Type: Bug > Components: dependencies, docker >Affects Versions: 2.0.0 >Reporter: Lin, Yi-Li >Assignee: Gerardo Curiel >Priority: Major > Attachments: docker_dag.log, docker_dag.py > > > I'm trying to install airflow with docker extras but airflow's dependency > will install recent docker-py (3.5.0) from pypi which is incompatible with > current DockerOperator. > DockerOperator will complain that "create_container() got an unexpected > keyword argument 'cpu_shares'". > It looks like that interface is changed from docker-py 3.0.0 and work with > docker-py 2.7.0. > The log and dag file are in attachments. > Note, installation comment: "AIRFLOW_GPL_UNIDECODE=yes pip install > 'apache-airflow[docker,mysql]==1.10.0'" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] jmcarp commented on issue #3919: [AIRFLOW-3089] Drop hard-coded url scheme in google auth redirect.
jmcarp commented on issue #3919: [AIRFLOW-3089] Drop hard-coded url scheme in google auth redirect. URL: https://github.com/apache/incubator-airflow/pull/3919#issuecomment-429095621 Rebased. I think the docs on redirect URLs and the ProxyFix middleware in integration.rst cover this well enough, so I think this is ready for review. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] jmcarp commented on issue #3919: [AIRFLOW-3089] Drop hard-coded url scheme in google auth redirect.
jmcarp commented on issue #3919: [AIRFLOW-3089] Drop hard-coded url scheme in google auth redirect. URL: https://github.com/apache/incubator-airflow/pull/3919#issuecomment-429093179 Yes, I ran into this testing google auth on localhost, where I didn't have https set up. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4037: [AIRFLOW-3191] Fix not being able to specify execution_date when creating dagrun
codecov-io edited a comment on issue #4037: [AIRFLOW-3191] Fix not being able to specify execution_date when creating dagrun URL: https://github.com/apache/incubator-airflow/pull/4037#issuecomment-429042678 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=h1) Report > Merging [#4037](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7115883fafca912e46ef0d67f69320f57305bbb3?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4037 +/- ## == + Coverage 75.77% 75.77% +<.01% == Files 199 199 Lines 1594615947 +1 == + Hits1208312084 +1 Misses 3863 3863 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `68.87% <100%> (+0.01%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=footer). Last update [7115883...5b3b6f1](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4037: [AIRFLOW-3191] Fix not being able to specify execution_date when creating dagrun
codecov-io edited a comment on issue #4037: [AIRFLOW-3191] Fix not being able to specify execution_date when creating dagrun URL: https://github.com/apache/incubator-airflow/pull/4037#issuecomment-429042678 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=h1) Report > Merging [#4037](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7115883fafca912e46ef0d67f69320f57305bbb3?src=pr=desc) will **decrease** coverage by `0.26%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=tree) ```diff @@Coverage Diff@@ ## master #4037 +/- ## = - Coverage 75.77% 75.5% -0.27% = Files 199 199 Lines 15946 15947 +1 = - Hits12083 12041 -42 - Misses 38633906 +43 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `68.68% <100%> (-0.17%)` | :arrow_down: | | [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `32.5% <0%> (-60%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `75.71% <0%> (-5.72%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `75.8% <0%> (-4.04%)` | :arrow_down: | | [airflow/task/task\_runner/base\_task\_runner.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL2Jhc2VfdGFza19ydW5uZXIucHk=) | `77.96% <0%> (-1.7%)` | :arrow_down: | | [airflow/www\_rbac/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9hcHAucHk=) | `95.6% <0%> (-1.1%)` | :arrow_down: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `64.24% <0%> (-0.24%)` | :arrow_down: | | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `71.89% <0%> (-0.15%)` | :arrow_down: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.67% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=footer). Last update [7115883...5b3b6f1](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4037: [AIRFLOW-3191] Fix not being able to specify execution_date when creating dagrun
codecov-io edited a comment on issue #4037: [AIRFLOW-3191] Fix not being able to specify execution_date when creating dagrun URL: https://github.com/apache/incubator-airflow/pull/4037#issuecomment-429042678 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=h1) Report > Merging [#4037](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7115883fafca912e46ef0d67f69320f57305bbb3?src=pr=desc) will **decrease** coverage by `0.26%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=tree) ```diff @@Coverage Diff@@ ## master #4037 +/- ## = - Coverage 75.77% 75.5% -0.27% = Files 199 199 Lines 15946 15947 +1 = - Hits12083 12041 -42 - Misses 38633906 +43 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `68.68% <100%> (-0.17%)` | :arrow_down: | | [airflow/hooks/hdfs\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9oZGZzX2hvb2sucHk=) | `32.5% <0%> (-60%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `75.71% <0%> (-5.72%)` | :arrow_down: | | [airflow/hooks/dbapi\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9kYmFwaV9ob29rLnB5) | `75.8% <0%> (-4.04%)` | :arrow_down: | | [airflow/task/task\_runner/base\_task\_runner.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy90YXNrL3Rhc2tfcnVubmVyL2Jhc2VfdGFza19ydW5uZXIucHk=) | `77.96% <0%> (-1.7%)` | :arrow_down: | | [airflow/www\_rbac/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9hcHAucHk=) | `95.6% <0%> (-1.1%)` | :arrow_down: | | [airflow/bin/cli.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `64.24% <0%> (-0.24%)` | :arrow_down: | | [airflow/www\_rbac/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92aWV3cy5weQ==) | `71.89% <0%> (-0.15%)` | :arrow_down: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.67% <0%> (-0.05%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=footer). Last update [7115883...5b3b6f1](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links.
codecov-io edited a comment on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links. URL: https://github.com/apache/incubator-airflow/pull/4036#issuecomment-429043686 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr=h1) Report > Merging [#4036](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7115883fafca912e46ef0d67f69320f57305bbb3?src=pr=desc) will **decrease** coverage by `60.55%`. > The diff coverage is `12.5%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4036 +/- ## === - Coverage 75.77% 15.22% -60.56% === Files 199 199 Lines 1594615963 +17 === - Hits12083 2430 -9653 - Misses 386313533 +9670 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `0% <0%> (-91.36%)` | :arrow_down: | | [airflow/www\_rbac/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9hcHAucHk=) | `21.56% <20%> (-75.14%)` | :arrow_down: | | [airflow/www\_rbac/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92YWxpZGF0b3JzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/www/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmFsaWRhdG9ycy5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/api/common/experimental/get\_dag\_runs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfZGFnX3J1bnMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/www/forms.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvZm9ybXMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | ... and [153 more](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr=footer). Last update [7115883...5341ea5](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links.
codecov-io commented on issue #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links. URL: https://github.com/apache/incubator-airflow/pull/4036#issuecomment-429043686 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr=h1) Report > Merging [#4036](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7115883fafca912e46ef0d67f69320f57305bbb3?src=pr=desc) will **decrease** coverage by `60.55%`. > The diff coverage is `12.5%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4036 +/- ## === - Coverage 75.77% 15.22% -60.56% === Files 199 199 Lines 1594615963 +17 === - Hits12083 2430 -9653 - Misses 386313533 +9670 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/plugins\_manager.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy9wbHVnaW5zX21hbmFnZXIucHk=) | `0% <0%> (-91.36%)` | :arrow_down: | | [airflow/www\_rbac/app.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy9hcHAucHk=) | `21.56% <20%> (-75.14%)` | :arrow_down: | | [airflow/www\_rbac/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy93d3dfcmJhYy92YWxpZGF0b3JzLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/utils/log/es\_task\_handler.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9sb2cvZXNfdGFza19oYW5kbGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/subdags/subdag.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3Mvc3ViZGFncy9zdWJkYWcucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/www/validators.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmFsaWRhdG9ycy5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/example\_docker\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9kb2NrZXJfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/example\_dags/example\_subdag\_operator.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy9leGFtcGxlX2RhZ3MvZXhhbXBsZV9zdWJkYWdfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/api/common/experimental/get\_dag\_runs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9nZXRfZGFnX3J1bnMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/www/forms.py](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvZm9ybXMucHk=) | `0% <0%> (-100%)` | :arrow_down: | | ... and [153 more](https://codecov.io/gh/apache/incubator-airflow/pull/4036/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr=footer). Last update [7115883...5341ea5](https://codecov.io/gh/apache/incubator-airflow/pull/4036?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4037: [AIRFLOW-3191] Fix not being able to specify execution_date when creating dagrun
codecov-io edited a comment on issue #4037: [AIRFLOW-3191] Fix not being able to specify execution_date when creating dagrun URL: https://github.com/apache/incubator-airflow/pull/4037#issuecomment-429042678 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=h1) Report > Merging [#4037](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7115883fafca912e46ef0d67f69320f57305bbb3?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4037 +/- ## == + Coverage 75.77% 75.78% +0.01% == Files 199 199 Lines 1594615947 +1 == + Hits1208312086 +3 + Misses 3863 3861 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `68.87% <100%> (+0.01%)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.67% <0%> (-0.05%)` | :arrow_down: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `82.39% <0%> (+0.26%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=footer). Last update [7115883...5b3b6f1](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io commented on issue #4037: [AIRFLOW-3191] Fix not being able to specify execution_date when creating dagrun
codecov-io commented on issue #4037: [AIRFLOW-3191] Fix not being able to specify execution_date when creating dagrun URL: https://github.com/apache/incubator-airflow/pull/4037#issuecomment-429042678 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=h1) Report > Merging [#4037](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7115883fafca912e46ef0d67f69320f57305bbb3?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=tree) ```diff @@Coverage Diff @@ ## master#4037 +/- ## == + Coverage 75.77% 75.78% +0.01% == Files 199 199 Lines 1594615947 +1 == + Hits1208312086 +3 + Misses 3863 3861 -2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `68.87% <100%> (+0.01%)` | :arrow_up: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `91.67% <0%> (-0.05%)` | :arrow_down: | | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/4037/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `82.39% <0%> (+0.26%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=footer). Last update [7115883...5b3b6f1](https://codecov.io/gh/apache/incubator-airflow/pull/4037?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3191) Fix not being able to specify execution_date when creating dagrun
[
https://issues.apache.org/jira/browse/AIRFLOW-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646746#comment-16646746
]
ASF GitHub Bot commented on AIRFLOW-3191:
-
aoen opened a new pull request #4037: [AIRFLOW-3191] Fix not being able to
specify execution_date when creating dagrun
URL: https://github.com/apache/incubator-airflow/pull/4037
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3191
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
Creating a dagrun using the flask web UI is not possible because
execution_date is hidden. This is because the column type is actually DATETIME
where Flask only has a converter for DateTime (it's the same type, it's a
casing issue).
Fix is to add a form override: form_overrides =
dict(execution_date=DateTimeField)
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
UI change, no tests for it.
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [X] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fix not being able to specify execution_date when creating dagrun
> -
>
> Key: AIRFLOW-3191
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3191
> Project: Apache Airflow
> Issue Type: Bug
> Components: webserver
>Affects Versions: 1.10.0
>Reporter: Dan Davydov
>Assignee: Dan Davydov
>Priority: Major
>
> Creating a dagrun using the flask web UI is not possible because
> execution_date is hidden. This is because the column type is actually
> DATETIME where Flask only has a converter for DateTime (it's the same type,
> it's a casing issue).
> Fix is to add a form override: form_overrides =
> dict(execution_date=DateTimeField)
> Only applies to the non-rbac webserver.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] aoen opened a new pull request #4037: [AIRFLOW-3191] Fix not being able to specify execution_date when creating dagrun
aoen opened a new pull request #4037: [AIRFLOW-3191] Fix not being able to
specify execution_date when creating dagrun
URL: https://github.com/apache/incubator-airflow/pull/4037
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-3191
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
Creating a dagrun using the flask web UI is not possible because
execution_date is hidden. This is because the column type is actually DATETIME
where Flask only has a converter for DateTime (it's the same type, it's a
casing issue).
Fix is to add a form override: form_overrides =
dict(execution_date=DateTimeField)
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
UI change, no tests for it.
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [X] Passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (AIRFLOW-3191) Fix not being able to specify execution_date when creating dagrun
[ https://issues.apache.org/jira/browse/AIRFLOW-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Davydov updated AIRFLOW-3191: - Description: Creating a dagrun using the flask web UI is not possible because execution_date is hidden. This is because the column type is actually DATETIME where Flask only has a converter for DateTime (it's the same type, it's a casing issue). Fix is to add a form override: form_overrides = dict(execution_date=DateTimeField) Only applies to the non-rbac webserver. was: Creating a dagrun using the flask web UI is not possible because execution_date is hidden. This is because the column type is actually DATETIME where Flask only has a converter for DateTime (it's the same type, it's a casing issue). Fix is to add a form override: form_overrides = dict(execution_date=DateTimeField) > Fix not being able to specify execution_date when creating dagrun > - > > Key: AIRFLOW-3191 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3191 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Affects Versions: 1.10.0 >Reporter: Dan Davydov >Assignee: Dan Davydov >Priority: Major > > Creating a dagrun using the flask web UI is not possible because > execution_date is hidden. This is because the column type is actually > DATETIME where Flask only has a converter for DateTime (it's the same type, > it's a casing issue). > Fix is to add a form override: form_overrides = > dict(execution_date=DateTimeField) > Only applies to the non-rbac webserver. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3191) Can not specify execution_date when creating dagrun
Dan Davydov created AIRFLOW-3191: Summary: Can not specify execution_date when creating dagrun Key: AIRFLOW-3191 URL: https://issues.apache.org/jira/browse/AIRFLOW-3191 Project: Apache Airflow Issue Type: Bug Components: webserver Affects Versions: 1.10.0 Reporter: Dan Davydov Assignee: Dan Davydov Creating a dagrun using the flask web UI is not possible because execution_date is hidden. This is because the column type is actually DATETIME where Flask only has a converter for DateTime (it's the same type, it's a casing issue). Fix is to add a form override: form_overrides = dict(execution_date=DateTimeField) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3191) Fix not being able to specify execution_date when creating dagrun
[ https://issues.apache.org/jira/browse/AIRFLOW-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Davydov updated AIRFLOW-3191: - Summary: Fix not being able to specify execution_date when creating dagrun (was: Can not specify execution_date when creating dagrun) > Fix not being able to specify execution_date when creating dagrun > - > > Key: AIRFLOW-3191 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3191 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Affects Versions: 1.10.0 >Reporter: Dan Davydov >Assignee: Dan Davydov >Priority: Major > > Creating a dagrun using the flask web UI is not possible because > execution_date is hidden. This is because the column type is actually > DATETIME where Flask only has a converter for DateTime (it's the same type, > it's a casing issue). > Fix is to add a form override: form_overrides = > dict(execution_date=DateTimeField) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] oliviersm199 commented on a change in pull request #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links.
oliviersm199 commented on a change in pull request #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links. URL: https://github.com/apache/incubator-airflow/pull/4036#discussion_r224515375 ## File path: airflow/plugins_manager.py ## @@ -49,6 +49,8 @@ class AirflowPlugin(object): admin_views = [] flask_blueprints = [] menu_links = [] +role_based_views = [] +role_based_menu_items = [] Review comment: Changed it to use appbuilder_views etc. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ron819 commented on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators
ron819 commented on issue #3981: [AIRFLOW-3133] Implement xcom_push flag for contrib's operators URL: https://github.com/apache/incubator-airflow/pull/3981#issuecomment-429003474 Is this ready to be merged? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ashb commented on a change in pull request #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links.
ashb commented on a change in pull request #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links. URL: https://github.com/apache/incubator-airflow/pull/4036#discussion_r224486669 ## File path: airflow/plugins_manager.py ## @@ -49,6 +49,8 @@ class AirflowPlugin(object): admin_views = [] flask_blueprints = [] menu_links = [] +role_based_views = [] +role_based_menu_items = [] Review comment: Lets call these `appbuilder_views` etc. The "role" is just the reason we switched to FAB. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] oliviersm199 opened a new pull request #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links.
oliviersm199 opened a new pull request #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links. URL: https://github.com/apache/incubator-airflow/pull/4036 ### Description Airflow Users that wish to create plugins for the new www_rbac UI can not add plugin views or links. This PR fixes that by letting a user specify their plugins for www_rbac and maintains backwards compatibility with the existing plugins system. Adds two additional members of an AirflowPlugin class to cover role based views and role based menu items. Specifies that the user can create a dictionary which has the metadata necessary to create a FlaskAppBuilder BaseView or MenuItem. This lets the plugins manager extend to the new www_rbac GUI. ### Tests Adds unit tests under the plugins manager to test that the www_rbac views and menu links get added successfully. ### Documentation Demonstrates in the plugins page how a user would create plugins for the new UI with an example. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-2744) RBAC app doesn't integrate plugins (blueprints etc)
[ https://issues.apache.org/jira/browse/AIRFLOW-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646558#comment-16646558 ] ASF GitHub Bot commented on AIRFLOW-2744: - oliviersm199 opened a new pull request #4036: [AIRFLOW-2744] Allow RBAC to accept plugins for views and links. URL: https://github.com/apache/incubator-airflow/pull/4036 ### Description Airflow Users that wish to create plugins for the new www_rbac UI can not add plugin views or links. This PR fixes that by letting a user specify their plugins for www_rbac and maintains backwards compatibility with the existing plugins system. Adds two additional members of an AirflowPlugin class to cover role based views and role based menu items. Specifies that the user can create a dictionary which has the metadata necessary to create a FlaskAppBuilder BaseView or MenuItem. This lets the plugins manager extend to the new www_rbac GUI. ### Tests Adds unit tests under the plugins manager to test that the www_rbac views and menu links get added successfully. ### Documentation Demonstrates in the plugins page how a user would create plugins for the new UI with an example. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > RBAC app doesn't integrate plugins (blueprints etc) > --- > > Key: AIRFLOW-2744 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2744 > Project: Apache Airflow > Issue Type: Bug > Components: webapp, webserver >Affects Versions: 2.0.0 >Reporter: David Dossett >Priority: Major > > In the current 1.10.0rc tag, the new RBAC app doesn't integrate any plugins > created by a user extending Airflow. In the old www/app.py you had the > [integrate_plugins|https://github.com/apache/incubator-airflow/blob/f1083cbada337731ed0b7e27b09eee7a26c8189a/airflow/www/app.py#L126] > function. But currently the > [www_rbac/app.py|https://github.com/apache/incubator-airflow/blob/f1083cbada337731ed0b7e27b09eee7a26c8189a/airflow/www_rbac/app.py] > doesn't pull in any plugins from the plugin_manager. So nothing you do to > extend Airflow's webapp will work. > I think adding the code for registering the blueprints and menu links is a > pretty simple fix. I'm not sure how the FAB system is handling the same > functionality as Flask-Admin views though. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] Fokko commented on issue #4035: [AIRFLOW-3190] Make flake8 compliant
Fokko commented on issue #4035: [AIRFLOW-3190] Make flake8 compliant URL: https://github.com/apache/incubator-airflow/pull/4035#issuecomment-428980180 @kaxil The day of Flake8 compliant Airflow has come. I had some spare time in the plane This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3190) Enforce Flake8
[ https://issues.apache.org/jira/browse/AIRFLOW-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646512#comment-16646512 ] ASF GitHub Bot commented on AIRFLOW-3190: - Fokko opened a new pull request #4035: [AIRFLOW-3190] Make flake8 compliant URL: https://github.com/apache/incubator-airflow/pull/4035 Enforce Flake8 over the entire project Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-3190\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3190 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-3190\], code changes always need a Jira issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Enforce Flake8 > -- > > Key: AIRFLOW-3190 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3190 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Fokko Driesprong >Priority: Major > > Enforce Flake8 over the entire project -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] Fokko opened a new pull request #4035: [AIRFLOW-3190] Make flake8 compliant
Fokko opened a new pull request #4035: [AIRFLOW-3190] Make flake8 compliant URL: https://github.com/apache/incubator-airflow/pull/4035 Enforce Flake8 over the entire project Make sure you have checked _all_ steps below. ### Jira - [x] My PR addresses the following [Airflow Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "\[AIRFLOW-3190\] My Airflow PR" - https://issues.apache.org/jira/browse/AIRFLOW-3190 - In case you are fixing a typo in the documentation you can prepend your commit with \[AIRFLOW-3190\], code changes always need a Jira issue. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wmorris75 commented on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators
wmorris75 commented on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators URL: https://github.com/apache/incubator-airflow/pull/3828#issuecomment-428957248 > @wmorris75, do you plan update PR and fix comment's issues? The updates have been made with the inclusion of the tests along with addressing all the comments made earlier. It is now ready for final review and approval. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] amichai07 opened a new pull request #4034: Update UPDATING.md
amichai07 opened a new pull request #4034: Update UPDATING.md URL: https://github.com/apache/incubator-airflow/pull/4034 I think the code may be clearer when words are kind of obscure This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3190) Enforce Flake8
Fokko Driesprong created AIRFLOW-3190: - Summary: Enforce Flake8 Key: AIRFLOW-3190 URL: https://issues.apache.org/jira/browse/AIRFLOW-3190 Project: Apache Airflow Issue Type: Improvement Reporter: Fokko Driesprong Enforce Flake8 over the entire project -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] vardancse edited a comment on issue #3994: [AIRFLOW-3136] Add retry_number to TaskInstance Key property to avoid race condition
vardancse edited a comment on issue #3994: [AIRFLOW-3136] Add retry_number to TaskInstance Key property to avoid race condition URL: https://github.com/apache/incubator-airflow/pull/3994#issuecomment-428557928 @ashb For us, we can easily reproduce by running a DAG which has a task bound to fail with retries set >0(preferably 4 or 5) and retry_delay set to 1 second, let me know if you're able to repro else I'll share sample DAG file. Also, our email_on_retry and email_on_failure is marked true, so once we see this issue of race condition, we will get an email something like below > Try 2 out of 4 > Exception: > Executor reports task instance %s finished (%s) although the task says its %s. Was the task killed externally? > Log: Link > Host: 72a8e9235db3 > Log file: /usr/local/airflow/logs/1fac4a1d57e551320a49403c_ufr09_qwpbfy/ACPComputeGateway_0_UFR09Task/2018-10-03T05:45:05.000221.log > Mark success: Link > EDIT-1: We have also done 1 testing at our side, where we were not able to reproduce issue when dag_folder consist less number of files but as soon as it touched 40+ Dags in our dag_folder, we were able to reproduce, probably increasing dags in dag_folder increasing 1 cycle of dags processing. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3189) DbAPI get_uri returns invalid uri if schema is None
Thomas Haederle created AIRFLOW-3189: Summary: DbAPI get_uri returns invalid uri if schema is None Key: AIRFLOW-3189 URL: https://issues.apache.org/jira/browse/AIRFLOW-3189 Project: Apache Airflow Issue Type: Bug Components: hooks Affects Versions: 1.10.0 Reporter: Thomas Haederle the current implementation of get_uri attaches the schema name to the URI even if no schema was specified. This leads to errors in downstream functions such as returning an invalid sqlalchemy engine. we should add a simple check, such when the schema is None, it should not be appended to the URI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3188) Browse -> DagRuns should link directly to right date on DAG pages
[
https://issues.apache.org/jira/browse/AIRFLOW-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ash Berlin-Taylor updated AIRFLOW-3188:
---
Description:
The list of DAG runs for a given tag can be reached by clicking on one of the
circles in the DAG runs page, which will bring up a filtered list of DAG run.
The Dag id column on this page ({{/admin/dagrun}} in non-RBAC UI) is a link to
the dag "detail" pages but to the most recent run -- the link should be updated
to link to the right dated dag run.
The same fix should be applied to the Dag id column on the Task Instances list
({{/admin/taskinstance}})
was:
The list of DAG runs for a given tag can be reached by clicking on one of the
circles in the DAG runs page, which will bring up a filtered list of DAG run.
The Dag id column on this page is a link to the dag "detail" pages but to the
most recent run -- the link should be updated to link to the right dated dag
run.
> Browse -> DagRuns should link directly to right date on DAG pages
> -
>
> Key: AIRFLOW-3188
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3188
> Project: Apache Airflow
> Issue Type: Improvement
> Components: webserver
>Affects Versions: 1.10.0, 2.0.0
>Reporter: Ash Berlin-Taylor
>Priority: Minor
> Labels: easyfix
>
> The list of DAG runs for a given tag can be reached by clicking on one of the
> circles in the DAG runs page, which will bring up a filtered list of DAG run.
> The Dag id column on this page ({{/admin/dagrun}} in non-RBAC UI) is a link
> to the dag "detail" pages but to the most recent run -- the link should be
> updated to link to the right dated dag run.
> The same fix should be applied to the Dag id column on the Task Instances
> list ({{/admin/taskinstance}})
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3188) Browse -> DagRuns should link directly to right date on DAG pages
Ash Berlin-Taylor created AIRFLOW-3188: -- Summary: Browse -> DagRuns should link directly to right date on DAG pages Key: AIRFLOW-3188 URL: https://issues.apache.org/jira/browse/AIRFLOW-3188 Project: Apache Airflow Issue Type: Improvement Components: webserver Affects Versions: 1.10.0, 2.0.0 Reporter: Ash Berlin-Taylor The list of DAG runs for a given tag can be reached by clicking on one of the circles in the DAG runs page, which will bring up a filtered list of DAG run. The Dag id column on this page is a link to the dag "detail" pages but to the most recent run -- the link should be updated to link to the right dated dag run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] PaulVelthuis93 commented on issue #4033: Update airflow.gif file with a slower version
PaulVelthuis93 commented on issue #4033: Update airflow.gif file with a slower version URL: https://github.com/apache/incubator-airflow/pull/4033#issuecomment-428910152 Hello Fokko, Here is the link to the Jira issue: https://issues.apache.org/jira/browse/AIRFLOW-3187 Have I done the linking correctly, or should it be linked differently with Jira? best, Paul Velthuis This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-3187) AIRFLOW-4033
Paul Velthuis created AIRFLOW-3187: -- Summary: AIRFLOW-4033 Key: AIRFLOW-3187 URL: https://issues.apache.org/jira/browse/AIRFLOW-3187 Project: Apache Airflow Issue Type: Wish Components: Documentation Reporter: Paul Velthuis Assignee: Paul Velthuis Attachments: airflow.gif The Airflow.gif is extremely fast, this can be annoying. For this reason I would like to propose to make it slower. Also for people who can't handle fast changing pictures this can lead to pain in the eyes. Please also find the new gif attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3187) Update airflow.gif file with a slower version
[ https://issues.apache.org/jira/browse/AIRFLOW-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Velthuis updated AIRFLOW-3187: --- Summary: Update airflow.gif file with a slower version (was: AIRFLOW-4033) > Update airflow.gif file with a slower version > - > > Key: AIRFLOW-3187 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3187 > Project: Apache Airflow > Issue Type: Wish > Components: Documentation >Reporter: Paul Velthuis >Assignee: Paul Velthuis >Priority: Major > Labels: documentation > Attachments: airflow.gif > > > The Airflow.gif is extremely fast, this can be annoying. For this reason I > would like to propose to make it slower. Also for people who can't handle > fast changing pictures this can lead to pain in the eyes. > > Please also find the new gif attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io edited a comment on issue #4033: Update airflow.gif file with a slower version
codecov-io edited a comment on issue #4033: Update airflow.gif file with a slower version URL: https://github.com/apache/incubator-airflow/pull/4033#issuecomment-428904897 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4033?src=pr=h1) Report > Merging [#4033](https://codecov.io/gh/apache/incubator-airflow/pull/4033?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7115883fafca912e46ef0d67f69320f57305bbb3?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4033?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4033 +/- ## === Coverage 75.77% 75.77% === Files 199 199 Lines 1594615946 === Hits1208312083 Misses 3863 3863 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4033?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4033?src=pr=footer). Last update [7115883...838fa4f](https://codecov.io/gh/apache/incubator-airflow/pull/4033?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #4033: Update airflow.gif file with a slower version
codecov-io edited a comment on issue #4033: Update airflow.gif file with a slower version URL: https://github.com/apache/incubator-airflow/pull/4033#issuecomment-428904897 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4033?src=pr=h1) Report > Merging [#4033](https://codecov.io/gh/apache/incubator-airflow/pull/4033?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/7115883fafca912e46ef0d67f69320f57305bbb3?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4033?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4033 +/- ## === Coverage 75.77% 75.77% === Files 199 199 Lines 1594615946 === Hits1208312083 Misses 3863 3863 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4033?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4033?src=pr=footer). Last update [7115883...838fa4f](https://codecov.io/gh/apache/incubator-airflow/pull/4033?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
