[GitHub] [airflow] seelmann commented on a change in pull request #6210: [AIRFLOW-5567] BaseAsyncOperator
seelmann commented on a change in pull request #6210: [AIRFLOW-5567] BaseAsyncOperator URL: https://github.com/apache/airflow/pull/6210#discussion_r336318983 ## File path: airflow/models/base_async_operator.py ## @@ -0,0 +1,161 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Base Asynchronous Operator for kicking off a long running +operations and polling for completion with reschedule mode. +""" + +from abc import abstractmethod +from typing import Dict, List, Optional, Union + +from airflow.models import SkipMixin, TaskReschedule +from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY +from airflow.sensors.base_sensor_operator import BaseSensorOperator +from airflow.utils.decorators import apply_defaults + + Review comment: If you need to persist state you may consider to add it to `task_reschedule` table? This was added in the initial reschedule PR https://github.com/apache/airflow/pull/3596/. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jaketf commented on issue #6210: [AIRFLOW-5567] BaseAsyncOperator
jaketf commented on issue #6210: [AIRFLOW-5567] BaseAsyncOperator URL: https://github.com/apache/airflow/pull/6210#issuecomment-543467902 @JonnyIncognito I'm a gcp-er don't have aws access. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] frankcash commented on issue #6360: [AIRFLOW-XXXX] Include code tags for sample file names
frankcash commented on issue #6360: [AIRFLOW-] Include code tags for sample file names URL: https://github.com/apache/airflow/pull/6360#issuecomment-543465011 Thank you This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] JonnyIncognito commented on a change in pull request #6210: [AIRFLOW-5567] BaseAsyncOperator
JonnyIncognito commented on a change in pull request #6210: [AIRFLOW-5567] BaseAsyncOperator URL: https://github.com/apache/airflow/pull/6210#discussion_r336291080 ## File path: airflow/models/base_async_operator.py ## @@ -0,0 +1,161 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Base Asynchronous Operator for kicking off a long running +operations and polling for completion with reschedule mode. +""" + +from abc import abstractmethod +from typing import Dict, List, Optional, Union + +from airflow.models import SkipMixin, TaskReschedule +from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY +from airflow.sensors.base_sensor_operator import BaseSensorOperator +from airflow.utils.decorators import apply_defaults + + Review comment: @Fokko reviewed and approved PR #4408 where `State.UP_FOR_RESCHEDULE` was added, so he may also have an opinion? Given the purpose that this was added - for asynchronous sensors - and that it was added so relatively recently in 1.10.2, I can't think of a reason we shouldn't make the change to retain XCom on reschedule. Hopefully we can find a clean way to do that! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-5648) Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG
[
https://issues.apache.org/jira/browse/AIRFLOW-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qian Yu reassigned AIRFLOW-5648:
Assignee: Qian Yu
> Add ClearTaskOperator to allow clearing/re-running tasks from within a DAG
> --
>
> Key: AIRFLOW-5648
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5648
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
>Affects Versions: 1.10.5
>Reporter: Qian Yu
>Assignee: Qian Yu
>Priority: Major
> Labels: ClearTaskOperator, airflow, clear, duplicate, operator,
> re-run, rerun, task
>
> There are use cases where some external conditions have changed and a section
> of the DAG needs to be re-run (after they have already finished previously).
> Here's such an example I recently encountered:
> We have a DAG that runs the following in the morning for execution_date T.
> The preliminary result of task J is needed at 9am in the morning. K, L and M
> needs to wait for Sensor to pass so they are not done till much later in the
> evening:
> {code:java}
> A >> C >> E >> G >> H >> I >> J >> K >> L >> M >> Finish
>^ ^
>| |
> B >> D >> F> Sensor
> {code}
> Later on in the afternoon at 3pm, some external condition changes (indicated
> by Sensor). At that point, we need to re-run task A and all its downstream
> tasks that have already run (i.e. A, C, E, G, H, I, J) to reflect possible
> changes. Other finished tasks such as B, D, F do not need to be re-run. The
> new results of J is needed in the evening by downstream tasks K, L, M that
> have been waiting.
> One possible approach is to make the DAG look like this by duplicating the
> section that needs to be re-run. In the following image, A1 runs the same
> command as A, C1 runs the same command as C, etc. This mostly works, but it
> causes the DAG to look unnecessarily large. It also causes code duplication
> because the tasks A1, C1, E1, G1, H1, I1, J1 are all identical to the
> original tasks. In this simplified example, the duplication does not look too
> bad, but in the real examples I faced, task A has many downstream tasks with
> complex dependencies. Copying all of them is more difficult. Because of these
> duplication, the next time someone updates the DAG and inserts a new task in
> the middle of E and G, it'll be hard to remember to add it in between E1 and
> G1 as well.
> {code:java}
> A >> C >> E >> G >> H >> I >> J >> K >> L >> M >> Finish
>^ ^
>| |__
> B >> D >> F |
> |
> Sensor >> A1 >> C1 >> E1 >> G1 >> H1 >> I1 >> J1
> {code}
> Instead of duplicating the tasks, I'm proposing adding a ClearTaskOperator.
> This operator takes an external_task_id as its parameter. When
> ClearTaskOperator runs, it clears the state of the given external_task_id and
> all its downstream tasks. This will cause them to re-run. So the problem I'm
> facing can be tackled without duplicating all those tasks. With
> ClearTaskOperator, the DAG can look like this.
> {code:java}
> A >> C >> E >> G >> H >> I >> J >> K >> L >> M >> Finish
>^ ^
>| |
> B >> D >> F|
>|
> Sensor >> Clear_Task_A >
> {code}
> In the above DAG, Clear_Task_A is a ClearTaskOperator defined like this. When
> Clear_Task_A executes, it clears task A and all its downstream tasks (so in
> this case it causes A, C, E, G, H, I, J to be cleared and re-run).
> {code:python}
> Clear_Task_A = ClearTaskOperator(task_id="Clear_Task_A",
> external_task_id="A") {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] dstandish commented on a change in pull request #6210: [AIRFLOW-5567] BaseAsyncOperator
dstandish commented on a change in pull request #6210: [AIRFLOW-5567] BaseAsyncOperator URL: https://github.com/apache/airflow/pull/6210#discussion_r336287664 ## File path: airflow/models/base_async_operator.py ## @@ -0,0 +1,161 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Base Asynchronous Operator for kicking off a long running +operations and polling for completion with reschedule mode. +""" + +from abc import abstractmethod +from typing import Dict, List, Optional, Union + +from airflow.models import SkipMixin, TaskReschedule +from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY +from airflow.sensors.base_sensor_operator import BaseSensorOperator +from airflow.utils.decorators import apply_defaults + + Review comment: > It seems to me we definitely want to wipe XComs at the end of a DagRun as to not affect a re run of this dag for this execution date. if this is your concern, xcom can be cleared at start if dag is re-run. (indeed, that's how it works now, which is a problem for this PR) probably makes sense to find some way to clear xcom when a task instance re-run but _not_ merely after a reschedule/poke. then we can leave the xcom value in place at end of run, _and_ it is safe to re-run, and everybody's happy? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dstandish commented on issue #6163: [AIRFLOW-4574] SSHHook private_key may only be supplied in extras
dstandish commented on issue #6163: [AIRFLOW-4574] SSHHook private_key may only be supplied in extras URL: https://github.com/apache/airflow/pull/6163#issuecomment-543435917 > I am sorry that I missed this PR earlier. no worries > In the next PR I would like to divide the cleaning changes in the tests from the changes in functionality. will do next time thank you This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jaketf commented on a change in pull request #6210: [AIRFLOW-5567] BaseAsyncOperator
jaketf commented on a change in pull request #6210: [AIRFLOW-5567]
BaseAsyncOperator
URL: https://github.com/apache/airflow/pull/6210#discussion_r336286276
##
File path: tests/models/test_base_async_operator.py
##
@@ -0,0 +1,288 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+""" Tests for BaseAsyncOperator"""
+
+import random
+import unittest
+import uuid
+from datetime import timedelta
+from unittest.mock import Mock # pylint: disable=ungrouped-imports
+
+from freezegun import freeze_time
+from parameterized import parameterized
+
+from airflow import DAG, settings
+from airflow.exceptions import AirflowSensorTimeout
+from airflow.models import DagRun, TaskInstance, TaskReschedule
+from airflow.models.base_async_operator import BaseAsyncOperator
+from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY
+from airflow.operators.dummy_operator import DummyOperator
+from airflow.utils import timezone
+from airflow.utils.state import State
+from airflow.utils.timezone import datetime
+
+DEFAULT_DATE = datetime(2015, 1, 1)
+TEST_DAG_ID = 'unit_test_dag'
+DUMMY_OP = 'dummy_op'
+ASYNC_OP = 'async_op'
+
+
+def _job_id():
+"""yield a random job id."""
+return 'job_id-{}'.format(uuid.uuid4())
+
+
+ALL_ID_TYPES = [
+(_job_id(),),
+(random.randint(0, 10**10),),
+([_job_id(), _job_id()],),
+({'job1': _job_id()},),
+(None,)
+]
+
+
+class DummyAsyncOperator(BaseAsyncOperator):
+"""
+Test subclass of BaseAsyncOperator
+"""
+def __init__(self, return_value=False,
+ **kwargs):
+super().__init__(**kwargs)
+self.return_value = return_value
+
+def poke(self, context):
+"""successful on first poke"""
+return self.return_value
+
+def submit_request(self, context):
+"""pretend to submit a job w/ random id"""
+return _job_id()
+
+def process_result(self, context):
+"""attempt to get the external resource_id"""
+return self.get_external_resource_id(context)
+
+
+class TestBaseAsyncOperator(unittest.TestCase):
+"""Test cases for BaseAsyncOperator."""
+def setUp(self):
+args = {
+'owner': 'airflow',
+'start_date': DEFAULT_DATE
+}
+self.dag = DAG(TEST_DAG_ID, default_args=args)
+
+session = settings.Session()
+session.query(TaskReschedule).delete()
+session.query(DagRun).delete()
+session.query(TaskInstance).delete()
+session.commit()
+
+def _make_dag_run(self):
+return self.dag.create_dagrun(
+run_id='manual__',
+start_date=timezone.utcnow(),
+execution_date=DEFAULT_DATE,
+state=State.RUNNING
+)
+
+def _make_async_op(self, return_value, resource_id=None, **kwargs):
+poke_interval = 'poke_interval'
+timeout = 'timeout'
+if poke_interval not in kwargs:
+kwargs[poke_interval] = 0
+if timeout not in kwargs:
+kwargs[timeout] = 0
+
+async_op = DummyAsyncOperator(
+task_id=ASYNC_OP,
+return_value=return_value,
+resource_id=resource_id,
+dag=self.dag,
+**kwargs
+)
+
+dummy_op = DummyOperator(
+task_id=DUMMY_OP,
+dag=self.dag
+)
+dummy_op.set_upstream(async_op)
+return async_op
+
+@classmethod
+def _run(cls, task):
+task.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE,
ignore_ti_state=True)
+
+def test_ok(self):
+""" Test normal behavior"""
+async_op = self._make_async_op(True)
+dr = self._make_dag_run()
+
+self._run(async_op)
+tis = dr.get_task_instances()
+self.assertEqual(len(tis), 2)
+for ti in tis:
+if ti.task_id == ASYNC_OP:
+self.assertEqual(ti.state, State.SUCCESS)
+if ti.task_id == DUMMY_OP:
+self.assertEqual(ti.state, State.NONE)
+
+def test_poke_fail(self):
+""" Test failure in poke"""
+async_op = self._make_async_op(False)
+dr = self._make_dag_run()
+
+with
[GitHub] [airflow] jaketf commented on a change in pull request #6210: [AIRFLOW-5567] BaseAsyncOperator
jaketf commented on a change in pull request #6210: [AIRFLOW-5567] BaseAsyncOperator URL: https://github.com/apache/airflow/pull/6210#discussion_r336285919 ## File path: airflow/models/base_async_operator.py ## @@ -0,0 +1,161 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Base Asynchronous Operator for kicking off a long running +operations and polling for completion with reschedule mode. +""" + +from abc import abstractmethod +from typing import Dict, List, Optional, Union + +from airflow.models import SkipMixin, TaskReschedule +from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY +from airflow.sensors.base_sensor_operator import BaseSensorOperator +from airflow.utils.decorators import apply_defaults + + Review comment: What kind of consensus should we build on that kind of change for XCom in reschedule scenarios? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] jaketf commented on a change in pull request #6210: [AIRFLOW-5567] BaseAsyncOperator
jaketf commented on a change in pull request #6210: [AIRFLOW-5567] BaseAsyncOperator URL: https://github.com/apache/airflow/pull/6210#discussion_r336285752 ## File path: airflow/models/base_async_operator.py ## @@ -0,0 +1,161 @@ +# -*- coding: utf-8 -*- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Base Asynchronous Operator for kicking off a long running +operations and polling for completion with reschedule mode. +""" + +from abc import abstractmethod +from typing import Dict, List, Optional, Union + +from airflow.models import SkipMixin, TaskReschedule +from airflow.models.xcom import XCOM_EXTERNAL_RESOURCE_ID_KEY +from airflow.sensors.base_sensor_operator import BaseSensorOperator +from airflow.utils.decorators import apply_defaults + + Review comment: It seems to me we definitely want to wipe XComs at the end of a DagRun as to not affect a re run of this dag for this execution date. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6359: [AIRFLOW-XXX] Change instances "Google cloud storage" to "Google Cloud Storage".
mik-laj commented on issue #6359: [AIRFLOW-XXX] Change instances "Google cloud
storage" to "Google Cloud Storage".
URL: https://github.com/apache/airflow/pull/6359#issuecomment-543397354
This problem also occurs elsewhere. Can you fix it in the whole project?
```
./airflow/operators/gcs_to_bq.py:Loads files from Google cloud storage
into BigQuery.
./airflow/operators/gcs_to_bq.py:point the operator to a Google cloud
storage object name. The object in
./airflow/operators/gcs_to_bq.py:Google cloud storage must be a JSON
file with the schema fields in it.
./airflow/operators/gcs_to_bq.py::param source_objects: List of Google
cloud storage URIs to load from. (templated)
./airflow/operators/cassandra_to_gcs.py:data from Cassandra to Google cloud
storage in JSON format.
./airflow/operators/cassandra_to_gcs.py:Copy data from Cassandra to
Google cloud storage in JSON format
./airflow/operators/cassandra_to_gcs.py:to Google cloud storage. A
{} should be specified in the filename
./airflow/operators/local_to_gcs.py:Uploads the file to Google cloud
storage
./airflow/operators/mysql_to_gcs.py:"""Copy data from MySQL to Google
cloud storage in JSON or CSV format.
./airflow/operators/mysql_to_gcs.py:JSON/Google cloud
storage/BigQuery. Dates are converted to UTC seconds.
./airflow/operators/gcs_to_gcs.py::param source_bucket: The source
Google cloud storage bucket where the
./airflow/operators/gcs_to_gcs.py::param destination_bucket: The
destination Google cloud storage bucket
./airflow/operators/gcs_to_gcs.py:destination Google cloud storage
bucket. (templated)
./airflow/operators/sql_to_gcs.py:Google cloud storage.
./airflow/gcp/sensors/gcs.py::param bucket: The Google cloud storage
bucket where the object is.
./airflow/gcp/sensors/gcs.py:connecting to Google cloud storage.
./airflow/gcp/sensors/gcs.py::param bucket: The Google cloud storage
bucket where the object is.
./airflow/gcp/sensors/gcs.py:connecting to Google cloud storage.
./airflow/gcp/sensors/gcs.py::param bucket: The Google cloud storage
bucket where the object is.
./airflow/gcp/sensors/gcs.py:connecting to Google cloud storage.
./airflow/gcp/sensors/gcs.py::param bucket: The Google cloud storage
bucket where the objects are.
./airflow/gcp/sensors/gcs.py:to Google cloud storage.
./airflow/gcp/operators/cloud_storage_transfer_service.py::param
source_bucket: The source Google cloud storage bucket where the
./airflow/gcp/operators/cloud_storage_transfer_service.py::param
destination_bucket: The destination Google cloud storage bucket
./airflow/gcp/operators/gcs.py::param bucket: The Google cloud storage
bucket to find the objects. (templated)
./airflow/gcp/operators/gcs.py::param bucket: The Google cloud storage
bucket where the object is.
./airflow/gcp/operators/bigquery.py:point the operator to a Google cloud
storage object name. The object in
./airflow/gcp/operators/bigquery.py:Google cloud storage must be a JSON
file with the schema fields in it.
./airflow/gcp/operators/bigquery.py:point the operator to a Google cloud
storage object name. The object in
./airflow/gcp/operators/bigquery.py:Google cloud storage must be a JSON
file with the schema fields in it.
./airflow/gcp/operators/bigquery.py::param source_objects: List of
Google cloud storage URIs to point
./airflow/gcp/hooks/gcs.py::param bucket_name: The Google cloud
storage bucket where the object is.
./airflow/gcp/hooks/gcs.py::param bucket_name: The Google cloud
storage bucket where the object is.
./airflow/gcp/hooks/gcs.py::param bucket_name: The Google cloud
storage bucket where the blob_name is.
./airflow/gcp/hooks/gcs.py::param bucket_name: The Google cloud
storage bucket where the blob_name is.
./airflow/gcp/hooks/gcs.py::param bucket_name: The Google cloud
storage bucket where the blob_name is.
grep: ./logs/scheduler/latest: No such file or directory
grep: ./files/airflow-home/logs/scheduler/latest: No such file or directory
./CHANGELOG.txt:- Don't return error when writing files to Google cloud
storage.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj merged pull request #6360: [AIRFLOW-XXXX] Include code tags for sample file names
mik-laj merged pull request #6360: [AIRFLOW-] Include code tags for sample file names URL: https://github.com/apache/airflow/pull/6360 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6351: [AIRFLOW-5682] Allow labels in gcs_to_bq operator
mik-laj commented on a change in pull request #6351: [AIRFLOW-5682] Allow
labels in gcs_to_bq operator
URL: https://github.com/apache/airflow/pull/6351#discussion_r336258244
##
File path: tests/gcp/hooks/test_bigquery.py
##
@@ -732,6 +733,29 @@ def run_with_config(config):
assert mocked_rwc.call_count == 1
+class TestLabelsInRunLoad(unittest.TestCase):
+@mock.patch.object(hook.BigQueryBaseCursor, 'run_with_configuration')
+def test_run_query_with_arg(self, mocked_rwc):
+project_id = '12345'
+
+def run_with_config(config):
+self.assertEqual(
+config['labels'], {'label1': 'test1', 'label2': 'test2',
+ 'airflow-version': hook._AIRFLOW_VERSION}
+)
+mocked_rwc.side_effect = run_with_config
Review comment:
Can you use assert_called_once_with instead creating special function? This
will make the code clearer.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow-site] mik-laj commented on issue #77: Improve typography
mik-laj commented on issue #77: Improve typography URL: https://github.com/apache/airflow-site/pull/77#issuecomment-543374173 Can you do rebase? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #5942: [AIRFLOW-5339] Fix infinite wait for Spark subprocess
codecov-io commented on issue #5942: [AIRFLOW-5339] Fix infinite wait for Spark subprocess URL: https://github.com/apache/airflow/pull/5942#issuecomment-543368332 # [Codecov](https://codecov.io/gh/apache/airflow/pull/5942?src=pr=h1) Report > Merging [#5942](https://codecov.io/gh/apache/airflow/pull/5942?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/76fe45e1d127b657b1aad5c0fd657e011f5a09bc?src=pr=desc) will **decrease** coverage by `0.35%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/airflow/pull/5942?src=pr=tree) ```diff @@Coverage Diff@@ ## master #5942 +/- ## = - Coverage 80.05% 79.7% -0.36% = Files 610 616 +6 Lines 35264 35794 +530 = + Hits28232 28528 +296 - Misses 70327266 +234 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/5942?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/hooks/spark\_submit\_hook.py](https://codecov.io/gh/apache/airflow/pull/5942/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NwYXJrX3N1Ym1pdF9ob29rLnB5) | `82.42% <0%> (ø)` | :arrow_up: | | [airflow/contrib/hooks/spark\_sql\_hook.py](https://codecov.io/gh/apache/airflow/pull/5942/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL3NwYXJrX3NxbF9ob29rLnB5) | `89.39% <100%> (ø)` | :arrow_up: | | [airflow/operators/mysql\_operator.py](https://codecov.io/gh/apache/airflow/pull/5942/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfb3BlcmF0b3IucHk=) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/5942/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvbXlzcWxfdG9faGl2ZS5weQ==) | `0% <0%> (-100%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/5942/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/5942/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `47.61% <0%> (-52.39%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/5942/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/5942/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/kube\_client.py](https://codecov.io/gh/apache/airflow/pull/5942/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL2t1YmVfY2xpZW50LnB5) | `33.33% <0%> (-41.67%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/5942/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `70.14% <0%> (-28.36%)` | :arrow_down: | | ... and [39 more](https://codecov.io/gh/apache/airflow/pull/5942/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/5942?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/5942?src=pr=footer). Last update [76fe45e...18df304](https://codecov.io/gh/apache/airflow/pull/5942?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6163: [AIRFLOW-4574] SSHHook private_key may only be supplied in extras
mik-laj commented on issue #6163: [AIRFLOW-4574] SSHHook private_key may only be supplied in extras URL: https://github.com/apache/airflow/pull/6163#issuecomment-543358516 I am sorry that I missed this PR earlier. The patch looks good to me, but @ashb wants to look, so I'll wait for its review. In the next PR I would like to divide the cleaning changes in the tests from the changes in functionality. This made it easier to review the changes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #6163: [AIRFLOW-4574] SSHHook private_key may only be supplied in extras
mik-laj commented on a change in pull request #6163: [AIRFLOW-4574] SSHHook private_key may only be supplied in extras URL: https://github.com/apache/airflow/pull/6163#discussion_r336220904 ## File path: tests/contrib/hooks/test_ssh_hook.py ## @@ -176,11 +212,13 @@ def test_tunnel(self): import subprocess import socket -server_handle = subprocess.Popen(["python", "-c", HELLO_SERVER_CMD], - stdout=subprocess.PIPE) -with hook.create_tunnel(2135, 2134): +subprocess_kwargs = dict( +args=["python", "-c", HELLO_SERVER_CMD], +stdout=subprocess.PIPE, +) +with subprocess.Popen(**subprocess_kwargs) as server_handle, hook.create_tunnel(2135, 2134): Review comment: Nice!. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5687) Upgrade pip to 19.0.2
[
https://issues.apache.org/jira/browse/AIRFLOW-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954102#comment-16954102
]
ASF GitHub Bot commented on AIRFLOW-5687:
-
kaxil commented on pull request #6361: [AIRFLOW-5687] Use PEP517 while
installing Airflow
URL: https://github.com/apache/airflow/pull/6361
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5687
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
I forgot to push this commit from my fork to #6358
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade pip to 19.0.2
> -
>
> Key: AIRFLOW-5687
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5687
> Project: Apache Airflow
> Issue Type: Improvement
> Components: dependencies
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Kaxil Naik
>Priority: Major
> Fix For: 1.10.6
>
>
> Pendulum==1.4.4 errors
>
> {code}
> Complete output from command /usr/local/bin/python
> /usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py
> get_requires_for_build_wheel /tmp/tmpGuP1ar:
> Traceback (most recent call last):
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 207, in
> main()
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 197, in main
> json_out['return_val'] = hook(**hook_input['kwargs'])
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 54, in get_requires_for_build_wheel
> return hook(config_settings)
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 115, in get_requires_for_build_wheel
> return _get_build_requires(config_settings, requirements=['wheel'])
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 101, in _get_build_requires
> _run_setup()
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 85, in _run_setup
> exec(compile(code, __file__, 'exec'), locals())
> File "setup.py", line 47, in
> from build import *
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/build.py", line 7,
> in
> from pip._vendor import pytoml
> ImportError: No module named pip._vendor
> {code}
> More details: https://github.com/pypa/pip/issues/6190
> Here is the comment from the maintainer:
> https://github.com/pypa/pip/issues/6163#issuecomment-462014864 that says
> pip==19.0.2 fixes that
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] kaxil opened a new pull request #6361: [AIRFLOW-5687] Use PEP517 while installing Airflow
kaxil opened a new pull request #6361: [AIRFLOW-5687] Use PEP517 while
installing Airflow
URL: https://github.com/apache/airflow/pull/6361
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-5687
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
I forgot to push this commit from my fork to #6358
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (AIRFLOW-5687) Upgrade pip to 19.0.2
[
https://issues.apache.org/jira/browse/AIRFLOW-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kaxil Naik resolved AIRFLOW-5687.
-
Resolution: Fixed
> Upgrade pip to 19.0.2
> -
>
> Key: AIRFLOW-5687
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5687
> Project: Apache Airflow
> Issue Type: Improvement
> Components: dependencies
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Kaxil Naik
>Priority: Major
> Fix For: 1.10.6
>
>
> Pendulum==1.4.4 errors
>
> {code}
> Complete output from command /usr/local/bin/python
> /usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py
> get_requires_for_build_wheel /tmp/tmpGuP1ar:
> Traceback (most recent call last):
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 207, in
> main()
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 197, in main
> json_out['return_val'] = hook(**hook_input['kwargs'])
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 54, in get_requires_for_build_wheel
> return hook(config_settings)
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 115, in get_requires_for_build_wheel
> return _get_build_requires(config_settings, requirements=['wheel'])
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 101, in _get_build_requires
> _run_setup()
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 85, in _run_setup
> exec(compile(code, __file__, 'exec'), locals())
> File "setup.py", line 47, in
> from build import *
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/build.py", line 7,
> in
> from pip._vendor import pytoml
> ImportError: No module named pip._vendor
> {code}
> More details: https://github.com/pypa/pip/issues/6190
> Here is the comment from the maintainer:
> https://github.com/pypa/pip/issues/6163#issuecomment-462014864 that says
> pip==19.0.2 fixes that
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5687) Upgrade pip to 19.0.2
[
https://issues.apache.org/jira/browse/AIRFLOW-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954099#comment-16954099
]
ASF subversion and git services commented on AIRFLOW-5687:
--
Commit b9438963d96b8717be9ed1eff73dfd015148837d in airflow's branch
refs/heads/v1-10-test from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=b943896 ]
[AIRFLOW-5687] Upgrade pip to 19.0.2 (#6358)
(cherry picked from commit f2b7f5a85aa9cb454836c66614465d7e048820a3)
> Upgrade pip to 19.0.2
> -
>
> Key: AIRFLOW-5687
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5687
> Project: Apache Airflow
> Issue Type: Improvement
> Components: dependencies
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Kaxil Naik
>Priority: Major
> Fix For: 1.10.6
>
>
> Pendulum==1.4.4 errors
>
> {code}
> Complete output from command /usr/local/bin/python
> /usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py
> get_requires_for_build_wheel /tmp/tmpGuP1ar:
> Traceback (most recent call last):
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 207, in
> main()
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 197, in main
> json_out['return_val'] = hook(**hook_input['kwargs'])
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 54, in get_requires_for_build_wheel
> return hook(config_settings)
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 115, in get_requires_for_build_wheel
> return _get_build_requires(config_settings, requirements=['wheel'])
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 101, in _get_build_requires
> _run_setup()
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 85, in _run_setup
> exec(compile(code, __file__, 'exec'), locals())
> File "setup.py", line 47, in
> from build import *
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/build.py", line 7,
> in
> from pip._vendor import pytoml
> ImportError: No module named pip._vendor
> {code}
> More details: https://github.com/pypa/pip/issues/6190
> Here is the comment from the maintainer:
> https://github.com/pypa/pip/issues/6163#issuecomment-462014864 that says
> pip==19.0.2 fixes that
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5687) Upgrade pip to 19.0.2
[
https://issues.apache.org/jira/browse/AIRFLOW-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954095#comment-16954095
]
ASF subversion and git services commented on AIRFLOW-5687:
--
Commit f2b7f5a85aa9cb454836c66614465d7e048820a3 in airflow's branch
refs/heads/master from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f2b7f5a ]
[AIRFLOW-5687] Upgrade pip to 19.0.2 (#6358)
> Upgrade pip to 19.0.2
> -
>
> Key: AIRFLOW-5687
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5687
> Project: Apache Airflow
> Issue Type: Improvement
> Components: dependencies
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Kaxil Naik
>Priority: Major
> Fix For: 1.10.6
>
>
> Pendulum==1.4.4 errors
>
> {code}
> Complete output from command /usr/local/bin/python
> /usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py
> get_requires_for_build_wheel /tmp/tmpGuP1ar:
> Traceback (most recent call last):
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 207, in
> main()
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 197, in main
> json_out['return_val'] = hook(**hook_input['kwargs'])
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 54, in get_requires_for_build_wheel
> return hook(config_settings)
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 115, in get_requires_for_build_wheel
> return _get_build_requires(config_settings, requirements=['wheel'])
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 101, in _get_build_requires
> _run_setup()
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 85, in _run_setup
> exec(compile(code, __file__, 'exec'), locals())
> File "setup.py", line 47, in
> from build import *
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/build.py", line 7,
> in
> from pip._vendor import pytoml
> ImportError: No module named pip._vendor
> {code}
> More details: https://github.com/pypa/pip/issues/6190
> Here is the comment from the maintainer:
> https://github.com/pypa/pip/issues/6163#issuecomment-462014864 that says
> pip==19.0.2 fixes that
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5687) Upgrade pip to 19.0.2
[
https://issues.apache.org/jira/browse/AIRFLOW-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954094#comment-16954094
]
ASF GitHub Bot commented on AIRFLOW-5687:
-
kaxil commented on pull request #6358: [AIRFLOW-5687] Upgrade pip to 19.0.2
URL: https://github.com/apache/airflow/pull/6358
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade pip to 19.0.2
> -
>
> Key: AIRFLOW-5687
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5687
> Project: Apache Airflow
> Issue Type: Improvement
> Components: dependencies
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Kaxil Naik
>Priority: Major
> Fix For: 1.10.6
>
>
> Pendulum==1.4.4 errors
>
> {code}
> Complete output from command /usr/local/bin/python
> /usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py
> get_requires_for_build_wheel /tmp/tmpGuP1ar:
> Traceback (most recent call last):
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 207, in
> main()
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 197, in main
> json_out['return_val'] = hook(**hook_input['kwargs'])
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 54, in get_requires_for_build_wheel
> return hook(config_settings)
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 115, in get_requires_for_build_wheel
> return _get_build_requires(config_settings, requirements=['wheel'])
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 101, in _get_build_requires
> _run_setup()
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 85, in _run_setup
> exec(compile(code, __file__, 'exec'), locals())
> File "setup.py", line 47, in
> from build import *
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/build.py", line 7,
> in
> from pip._vendor import pytoml
> ImportError: No module named pip._vendor
> {code}
> More details: https://github.com/pypa/pip/issues/6190
> Here is the comment from the maintainer:
> https://github.com/pypa/pip/issues/6163#issuecomment-462014864 that says
> pip==19.0.2 fixes that
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] kaxil merged pull request #6358: [AIRFLOW-5687] Upgrade pip to 19.0.2
kaxil merged pull request #6358: [AIRFLOW-5687] Upgrade pip to 19.0.2 URL: https://github.com/apache/airflow/pull/6358 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-4758) Add GoogleCloudStorageToGoogleDrive Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-4758: Fix Version/s: (was: 2.0.0) 1.10.7 > Add GoogleCloudStorageToGoogleDrive Operator > > > Key: AIRFLOW-4758 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4758 > Project: Apache Airflow > Issue Type: Wish > Components: gcp, operators >Affects Versions: 1.10.3 >Reporter: jack >Priority: Major > Fix For: 1.10.7 > > > Add Operators: > GoogleCloudStorageToGoogleDrive > GoogleDriveToGoogleCloudStorage > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-4161) Big query to mysql Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kaxil Naik updated AIRFLOW-4161: Fix Version/s: (was: 2.0.0) 1.10.7 > Big query to mysql Operator > --- > > Key: AIRFLOW-4161 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4161 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Antoine Dussarps >Assignee: Antoine Dussarps >Priority: Minor > Fix For: 1.10.7 > > Attachments: bq_to_mysql_operator.py > > > [Contrib] We developed an operator that safely copy a bigquery table onto > MySql. > It can be used for speed improvement on some pre-agregated requests. > Does not work on views. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] OmerJog commented on issue #6297: [AIRFLOW-5224] Add encoding parameter to GoogleCloudStorageToBigQuery…
OmerJog commented on issue #6297: [AIRFLOW-5224] Add encoding parameter to GoogleCloudStorageToBigQuery… URL: https://github.com/apache/airflow/pull/6297#issuecomment-543325139 Jira is still open. Can this PR also be marked for 1.10.7? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] OmerJog commented on issue #5967: [AIRFLOW-5415] Enable authentication for druid query hook.
OmerJog commented on issue #5967: [AIRFLOW-5415] Enable authentication for druid query hook. URL: https://github.com/apache/airflow/pull/5967#issuecomment-543323888 @scrawfor you need to change your commit message to: [AIRFLOW-5415] Enable authentication for druid query hook. the commit message is important for the log Also I believe you'll need to add tests ([test_druid_hook.py](https://github.com/apache/airflow/blob/master/tests/hooks/test_druid_hook.py)) for this PR but wait to see what commiters think This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] OmerJog commented on issue #5388: [AIRFLOW-4490] DagRun.conf returns empty dictionary by default
OmerJog commented on issue #5388: [AIRFLOW-4490] DagRun.conf returns empty dictionary by default URL: https://github.com/apache/airflow/pull/5388#issuecomment-543316332 @dstandish tests did not run because of Falke8 error: `tests/models/test_taskinstance.py:1285:1: W293 blank line contains whitespace` can you resolve this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] OmerJog commented on issue #5559: [AIRFLOW-4930] Send druid ingestion spec as data, not json
OmerJog commented on issue #5559: [AIRFLOW-4930] Send druid ingestion spec as data, not json URL: https://github.com/apache/airflow/pull/5559#issuecomment-543314840 @Fokko maybe you can give your input on the suggestion This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] OmerJog commented on issue #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
OmerJog commented on issue #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#issuecomment-543313549 2 new operators were missed from this release? GoogleCloudStorageToGoogleDrive https://issues.apache.org/jira/browse/AIRFLOW-4758 BigQueryToMySqlOperator https://issues.apache.org/jira/browse/AIRFLOW-4161 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] frankcash opened a new pull request #6360: [AIRFLOW-XXXX] Include code tags for sample file names
frankcash opened a new pull request #6360: [AIRFLOW-] Include code tags for
sample file names
URL: https://github.com/apache/airflow/pull/6360
Improves readability of docs
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
AIRFLOW-XXX path as this a doc change
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Changes quotes around files to code formatting for visual indication that we
are discussing a file.
Proposed changes:
https://user-images.githubusercontent.com/6012231/67037992-6aad0380-f0d3-11e9-8700-b795b254674d.png;>
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason: simple RST change
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] fritzdavenport commented on issue #4660: [AIRFLOW-3819] - Allow the configuration of a global default for work…
fritzdavenport commented on issue #4660: [AIRFLOW-3819] - Allow the configuration of a global default for work… URL: https://github.com/apache/airflow/pull/4660#issuecomment-543308415 Agreed w/ @danieltahara - creating pods with no resource requests or limits, and relying on a user having namespace defaults is definitely not good practice. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] fritzdavenport edited a comment on issue #4660: [AIRFLOW-3819] - Allow the configuration of a global default for work…
fritzdavenport edited a comment on issue #4660: [AIRFLOW-3819] - Allow the configuration of a global default for work… URL: https://github.com/apache/airflow/pull/4660#issuecomment-543308415 Agreed w/ @danieltahara - creating pods with no resource requests or limits, and relying on a user having namespace defaults is definitely not good practice. This seems like a fundamentally basic feature for any pod - and that anything launching pods should account for This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5637) KubeExecutor - Resource Request and Limits
[ https://issues.apache.org/jira/browse/AIRFLOW-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953994#comment-16953994 ] Daniel Tahara commented on AIRFLOW-5637: 3819 was closed due to the recommendation to handle it on the k8s side of things. As mentioned in my comment on github, I don't actually think that's an appropriate configuration. https://github.com/apache/airflow/pull/4660#issuecomment-542982778 ``` Since LimitRanges are applied as a namespace default, the more tenants you have the less useful those ranges are going to be (because they'll have to be uselessly wide). There isn't a good reason that Airflow should be creating pods without memory limits or not providing that configuration option out of the box. It makes binpacking and cluster autoscaling much harder, since the pods will schedule effectively anywhere. ``` The existing proposed PR would be sufficient to solve the problem. > KubeExecutor - Resource Request and Limits > -- > > Key: AIRFLOW-5637 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5637 > Project: Apache Airflow > Issue Type: Improvement > Components: executor-kubernetes >Affects Versions: 1.10.5 >Reporter: F D >Assignee: Daniel Imberman >Priority: Minor > > KubeExecutor doesn't have a way to have pods be created with resource > requests or limits. > The only workaround is to set Namespace defaults, which the created pods pick > up. > I feel this could be a setting in `[kuberetes]` in `airflow.cfg`, or use the > `[operator]` defaults. > Here is the reference of the spec: > [https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.16/#resourcerequirements-v1-core] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] enricapq commented on issue #6327: [AIRFLOW-4675] Make airflow/lineage Pylint compatible
enricapq commented on issue #6327: [AIRFLOW-4675] Make airflow/lineage Pylint compatible URL: https://github.com/apache/airflow/pull/6327#issuecomment-543290651 Hi @ashb thanks for the review. I can't apply changes at the moment, I can continue working on that in 3 weeks. Feel free to close or modify it if you don't want to wait for me. Thank u for your time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on issue #6358: [AIRFLOW-5687] Upgrade pip to 19.0.2
kaxil commented on issue #6358: [AIRFLOW-5687] Upgrade pip to 19.0.2 URL: https://github.com/apache/airflow/pull/6358#issuecomment-543281463 > There are two more instances further on in that file that should be removed too? My bad, removed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #6358: [AIRFLOW-5687] Upgrade pip to 19.0.2
ashb commented on issue #6358: [AIRFLOW-5687] Upgrade pip to 19.0.2 URL: https://github.com/apache/airflow/pull/6358#issuecomment-543278655 There are two more instances further on in that file that should be removed too? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5608) Scheduler not shutting down executor(s) gracefully
[ https://issues.apache.org/jira/browse/AIRFLOW-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953940#comment-16953940 ] ASF subversion and git services commented on AIRFLOW-5608: -- Commit 2fb442478d535f1039e32e73e518eac688b2fcf9 in airflow's branch refs/heads/v1-10-stable from Ash Berlin-Taylor [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2fb4424 ] Revert "[AIRFLOW-5608] Gracefully stop executor when SIGTERM is received by SchedulerJob (#6274)" This reverts commit d0cf27c240cd8b453f4f1bd53618f358aef1c51a. > Scheduler not shutting down executor(s) gracefully > -- > > Key: AIRFLOW-5608 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5608 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.10.5 >Reporter: Kiran >Assignee: Kiran >Priority: Major > > If SIGTERM is sent to scheduler, the bug is preventing it from calling > graceful shutdown of executor(s). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5608) Scheduler not shutting down executor(s) gracefully
[ https://issues.apache.org/jira/browse/AIRFLOW-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor updated AIRFLOW-5608: --- Fix Version/s: (was: 1.10.6) > Scheduler not shutting down executor(s) gracefully > -- > > Key: AIRFLOW-5608 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5608 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.10.5 >Reporter: Kiran >Assignee: Kiran >Priority: Major > > If SIGTERM is sent to scheduler, the bug is preventing it from calling > graceful shutdown of executor(s). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5608) Scheduler not shutting down executor(s) gracefully
[ https://issues.apache.org/jira/browse/AIRFLOW-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953939#comment-16953939 ] ASF subversion and git services commented on AIRFLOW-5608: -- Commit 2fb442478d535f1039e32e73e518eac688b2fcf9 in airflow's branch refs/heads/v1-10-test from Ash Berlin-Taylor [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2fb4424 ] Revert "[AIRFLOW-5608] Gracefully stop executor when SIGTERM is received by SchedulerJob (#6274)" This reverts commit d0cf27c240cd8b453f4f1bd53618f358aef1c51a. > Scheduler not shutting down executor(s) gracefully > -- > > Key: AIRFLOW-5608 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5608 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.10.5 >Reporter: Kiran >Assignee: Kiran >Priority: Major > Fix For: 1.10.6 > > > If SIGTERM is sent to scheduler, the bug is preventing it from calling > graceful shutdown of executor(s). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes
codecov-io edited a comment on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes URL: https://github.com/apache/airflow/pull/6321#issuecomment-543213812 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=h1) Report > Merging [#6321](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/ac42428bf530c259ab1b0dca08458c1ebf49b04a?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6321 +/- ## == + Coverage 80.08% 80.08% +<.01% == Files 616 616 Lines 3579735797 == + Hits2866828669 +1 + Misses 7129 7128 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6321/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.28% <0%> (-0.51%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6321/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (ø)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6321/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90% <0%> (+5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=footer). Last update [ac42428...dd8715c](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r336103607 ## File path: CHANGELOG.txt ## @@ -1,3 +1,133 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Features + +- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546) +- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407) +- [AIRFLOW-4939] Add default_task_retries config (#5570) +- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130) +- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) +- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743) + +Improvements + +- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738) +- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737) +- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736) +- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330) +- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034) +- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194) +- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334) +- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) +- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) +- [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of the aws hook (#6303) +- [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302) +- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304) +- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309) +- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153) +- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) +- [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) +- [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062) +- [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059) +- [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948) +- [AIRFLOW-3888] HA for Hive metastore connection (#4708) +- [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873) +- [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768) +- [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214) +- [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696) +- [AIRFLOW-2891] allow configurable docker_operator container name (#5689) +- [AIRFLOW-4285] Update task dependency context definition and usage (#5079) +- [AIRFLOW-5142] Fixed flaky Cassandra test (#5758) +- [AIRFLOW-5218] Less polling of AWS Batch job status (#5825) +- [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589) +- [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339) +- [AIRFLOW-5560] Allow no confirmation on reset dags in `airflow backfill` command (#6195) +- [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879) +- [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159) +- [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154) +- [AIRFLOW-4835] Refactor operator render_template (#5461) + +Bug Fixes +" +- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator (#6078) +- [Airflow 4923] Fix Databricks hook leaks API secret in logs (#5635) +- [AIRFLOW-5133] Keep original env state in provide_gcp_credential_file (#5747) +- [AIRFLOW-5497] Update docstring in ``airflow/utils/dag_processing.py`` (#6314) +- Revert/and then rework "[AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)" to improve performance (#5908) +- [AIRFLOW-5634] Don't allow editing of DagModelView (#6308) +- [AIRFLOW-4309] Remove Broken Dag error after Dag is deleted (#6102) +- [AIRFLOW-5387] Fix "show paused" pagination bug (#6100) +- [AIRFLOW-5489] Remove unneeded assignment of variable (#6106) +- [AIRFLOW-5491] mark_tasks pydoc is incorrect (#6108) +- [AIRFLOW-5492] added missing docstrings (#6107) +- [AIRFLOW-5503] Fix tree view layout on HDPI screen (#6125) +- [AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101) +- [AIRFLOW-3857] spark_submit_hook cannot kill driver pod in Kubernetes (#4678) +- [AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent Tasks' (#5909) +- [AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185) +- [AIRFLOW-5306] Fix the display of links when they contain special characters (#5904) +- [AIRFLOW-3705] Fix PostgresHook get_conn to use conn_name_attr (#5841) +- [AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler shutdown on SIGTERM (#6237) +- [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView (#6307) +- [AIRFLOW-4833] Allow to set Jinja env options in DAG
[GitHub] [airflow] Sinsin1367 edited a comment on issue #6261: [AIRFLOW-5141] Added optional resource usage logging to KubernetesPod…
Sinsin1367 edited a comment on issue #6261: [AIRFLOW-5141] Added optional resource usage logging to KubernetesPod… URL: https://github.com/apache/airflow/pull/6261#issuecomment-543251695 > My initial thought: Just use prometheus/cadvisor for this. There are other tools that are better suited to capturing metrics than Airflow. > > What about the KubeExecutor? If (and it's still an if to me) we are to add this it should work with pods launched by the executor too. The reason for fetching usage in this feature is to report the usage interleaved with application logs. This way the user would know at what stage in their job their usage is going up or down and will watch the usage as it is going. Also it eliminates necessity of using metric monitoring frameworks. Like in our case we use datadog for our metric monitoring which is not that easy to integrate in this case. I was hoping having this as an option that is turned off by default would not hurt any existing use-cases and adds the value of getting resource usage live in the logs to know exactly what events might be responsible for a bump in usage. KubeExecutor uses PodLauncher to launch pods also. However it seems it calls the async method of podLauncher which means it would not log. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Sinsin1367 commented on issue #6261: [AIRFLOW-5141] Added optional resource usage logging to KubernetesPod…
Sinsin1367 commented on issue #6261: [AIRFLOW-5141] Added optional resource usage logging to KubernetesPod… URL: https://github.com/apache/airflow/pull/6261#issuecomment-543251695 > My initial thought: Just use prometheus/cadvisor for this. There are other tools that are better suited to capturing metrics than Airflow. > > What about the KubeExecutor? If (and it's still an if to me) we are to add this it should work with pods launched by the executor too. The reason for fetching usage in this feature is to report the usage interleaved with application logs. This way the user would know at what stage in their job their usage is going up or down and will watch the usage as it is going. Also it eliminates necessity of using metric monitoring frameworks. Like in our case we use datadog for our metric monitoring which is not that easy to integrate in this case. KubeExecutor uses PodLauncher to launch pods also. However it seems it calls the async method of podLauncher which means it would not log. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Sinsin1367 commented on a change in pull request #6261: [AIRFLOW-5141] Added optional resource usage logging to KubernetesPod…
Sinsin1367 commented on a change in pull request #6261: [AIRFLOW-5141] Added
optional resource usage logging to KubernetesPod…
URL: https://github.com/apache/airflow/pull/6261#discussion_r336086229
##
File path: airflow/kubernetes/pod_usage_metrics_logger.py
##
@@ -0,0 +1,223 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Logs resource usage of a pod"""
+import re
+import time
+from collections import defaultdict
+from datetime import datetime as dt, timedelta
+
+from kubernetes.client.models.v1_pod import V1Pod
+
+from airflow import LoggingMixin
+
+
+class PodUsageMetricsLogger(LoggingMixin):
+"""
+This class logs all the resource usage metrics of a pod during the pod
life time.
+"""
+memory_usage_units = {'Ki': 1,
+ 'Mi': 2 ** 10,
+ 'Gi': 2 ** 20,
+ 'Ti': 2 ** 30,
+ 'Pi': 2 ** 40,
+ 'Ei': 2 ** 50,
+ '': 1}
+# Sometimes memory usage is not followed by unit if it is 0
+# In these cases regex might return '' or None for unit.
+valid_memory_usage_regex =
re.compile(r'^([+]?[0-9]+?([.][0-9]+)?)(([KMGTPE][i])?)$')
+# This regex checks for a positive float number with optional Ki, Mi, Gi,
Ti, Pi, or Ei as unit
+
+cpu_usage_units = {'m': 1,
+ '': 1} # Sometimes cpu usage is not followed by the
unit when usage is 0
+valid_cpu_usage_regex = re.compile(r'^([+]?[0-9]+?([.][0-9]+)?)([m]?)$')
+# This regex check for a positive float number with optional millicpu(m)
unit
+
+def __init__(self,
+ pod_launcher: object, # (PodLauncher type) importing it
might causes circular imports
+ pod: V1Pod,
+ resource_usage_fetch_interval: int,
+ resource_usage_log_interval: int):
+"""
+Creates the usage metrics logger.
+:param pod_launcher: reference to the launcher of the metrics logger
+:param pod: the pod we are logging the usage metrics
+:param resource_usage_log_interval: logging interval in seconds
+"""
+super().__init__()
+self.pod_launcher = pod_launcher
+self.pod = pod
+self.resource_usage_fetch_interval = resource_usage_fetch_interval
+self.resource_usage_log_interval = resource_usage_log_interval
+self.max_cpu_usages = defaultdict(lambda: -1) # type: dict
+self.max_memory_usages = defaultdict(lambda: -1) # type: dict
+
+def log_pod_usage_metrics(self):
+"""
+Periodically calls metrics api for the pod, logs current usage of the
pod per container, and keeps
+track of max usage seen for all containers.
+:return:
+"""
+if self.resource_usage_fetch_interval <= 0:
+self.log.error('Parameter resource_usage_fetch_interval must be
positive. Cancelling pod usage '
+ 'monitoring thread for pod:
{0}.'.format(self.pod.metadata.name))
+return
+self.log.info('Pod usage monitoring thread started for pod:
{0}'.format(self.pod.metadata.name))
+resource_usage_api_url =
"/apis/metrics.k8s.io/v1beta1/namespaces/{0}/pods/{1}"\
+ .format(self.pod.metadata.namespace,
+ self.pod.metadata.name)
+cur_time = dt.now()
+last_fetch_time = cur_time -
timedelta(seconds=self.resource_usage_fetch_interval)
+last_log_time = cur_time -
timedelta(seconds=self.resource_usage_log_interval)
+while self.pod_launcher.pod_is_running(self.pod):
+if cur_time < last_fetch_time +
timedelta(seconds=self.resource_usage_fetch_interval):
+time.sleep(1)
Review comment:
We do not want to block main thread if the pod is not running anymore.
Imagine a task with fetch interval set to 5min, then sleeping 5min while the
pod is not running and blocking main thread is not ideal.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and
[GitHub] [airflow] kaxil commented on issue #6358: [AIRFLOW-5687] Upgrade pip to 19.0.2
kaxil commented on issue #6358: [AIRFLOW-5687] Upgrade pip to 19.0.2 URL: https://github.com/apache/airflow/pull/6358#issuecomment-543237459 > Do we need to remove the pep517 flag too? Removed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (AIRFLOW-4431) Gantt chart tab do not showing
[
https://issues.apache.org/jira/browse/AIRFLOW-4431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948337#comment-16948337
]
Diederik Perdok edited comment on AIRFLOW-4431 at 10/17/19 3:39 PM:
I ran into the same problem, so it is not fixed in 1.10.4.
For me the problem is caused by the presence of a task failure without
start_date and duration. A task_instance for the same task_id does exist, and
has state success and a valid start_date, end_date and duration. When viewing
the Gantt chart, this task failure is included, but this fails due to the
missing start_date.
The fix should not be that difficult, but I would first like to understand how
the db can be put in this state. At first I thought this could be done by
failing a task through the UI before it started, but this does not result in a
new record in task_failure. I have tried other ways, but I have not been able
to reproduce it myself. I will talk to the user who reported the problem to
me; hopefully he can tell me the exact steps that led up to this state.
was (Author: diederikwp):
I ran into the same problem, so it is not fixed in 1.10.4 (haven't tried 1.10.5
yet, but looking at the code I'm sure the same bug exists in 1.10.5 as well).
I believe the steps to reproduce are as follows:
# Start a new dag run
# Fail a task before it has had a chance to run
# Clear this task and wait for it to start running
# Try to view the Gantt chart
Now there exists a task instance which has a start_date, and a task failure for
the same task_id which does not have a start_date. This task failure is
included in the Gantt chart (because the _task instance_ does have a
start_date), but it shouldn't be.
I will submit a PR before the end of next week.
BTW while the above steps do reproduce
[AIRFLOW-4431|https://issues.apache.org/jira/browse/AIRFLOW-4431] they do not
reproduce [AIRFLOW-4432|https://issues.apache.org/jira/browse/AIRFLOW-4432],
probably because that was already fixed in
[AIRFLOW-4409|https://issues.apache.org/jira/browse/AIRFLOW-4409].
> Gantt chart tab do not showing
> --
>
> Key: AIRFLOW-4431
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4431
> Project: Apache Airflow
> Issue Type: Bug
> Components: webserver
>Affects Versions: 1.10.3
> Environment: Ubuntu 16
> Python 3.5.2
>Reporter: Dmytro Kulyk
>Assignee: Diederik Perdok
>Priority: Major
> Labels: gantt
>
> There is an error when opening Gantt chart in WebUI
> {code}
> Traceback (most recent call last):
> File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 2292, in
> wsgi_app
> response = self.full_dispatch_request()
> File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1815, in
> full_dispatch_request
> rv = self.handle_user_exception(e)
> File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1718, in
> handle_user_exception
> reraise(exc_type, exc_value, tb)
> File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 35, in
> reraise
> raise value
> File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1813, in
> full_dispatch_request
> rv = self.dispatch_request()
> File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1799, in
> dispatch_request
> return self.view_functions[rule.endpoint](**req.view_args)
> File "/usr/local/lib/python3.5/dist-packages/flask_admin/base.py", line 69,
> in inner
> return self._run_view(f, *args, **kwargs)
> File "/usr/local/lib/python3.5/dist-packages/flask_admin/base.py", line
> 368, in _run_view
> return fn(self, *args, **kwargs)
> File "/usr/local/lib/python3.5/dist-packages/flask_login/utils.py", line
> 261, in decorated_view
> return func(*args, **kwargs)
> File "/usr/local/lib/python3.5/dist-packages/airflow/www/utils.py", line
> 275, in wrapper
> return f(*args, **kwargs)
> File "/usr/local/lib/python3.5/dist-packages/airflow/utils/db.py", line 73,
> in wrapper
> return func(*args, **kwargs)
> File "/usr/local/lib/python3.5/dist-packages/airflow/www/views.py", line
> 1988, in gantt
> 'startDate': wwwutils.epoch(start_date),
> File "/usr/local/lib/python3.5/dist-packages/airflow/www/utils.py", line
> 245, in epoch
> return int(time.mktime(dttm.timetuple())) * 1000,
> AttributeError: 'NoneType' object has no attribute 'timetuple'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] ashb commented on issue #6358: [AIRFLOW-5687] Upgrade pip to 19.0.2
ashb commented on issue #6358: [AIRFLOW-5687] Upgrade pip to 19.0.2 URL: https://github.com/apache/airflow/pull/6358#issuecomment-543229473 Do we need to remove the pep517 flag too? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dgorelik opened a new pull request #6359: [AIRFLOW-XXX] Change instances "Google cloud storage" to "Google Cloud Storage".
dgorelik opened a new pull request #6359: [AIRFLOW-XXX] Change instances
"Google cloud storage" to "Google Cloud Storage".
URL: https://github.com/apache/airflow/pull/6359
GCS should be formatted as "Google Cloud Storage", so change instances
"Google cloud storage" to "Google Cloud Storage".
Make sure you have checked _all_ steps below.
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes: just changing docstring
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-5687) Upgrade pip to 19.0.2
[
https://issues.apache.org/jira/browse/AIRFLOW-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953827#comment-16953827
]
ASF GitHub Bot commented on AIRFLOW-5687:
-
kaxil commented on pull request #6358: [AIRFLOW-5687] Upgrade pip to 19.0.2
URL: https://github.com/apache/airflow/pull/6358
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
Pendulum==1.4.4 errors
Failed build: https://travis-ci.com/ashb/airflow/jobs/246690984
```
Complete output from command /usr/local/bin/python
/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py
get_requires_for_build_wheel /tmp/tmpGuP1ar:
Traceback (most recent call last):
File
"/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
line 207, in
main()
File
"/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
line 197, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File
"/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
line 54, in get_requires_for_build_wheel
return hook(config_settings)
File
"/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
line 115, in get_requires_for_build_wheel
return _get_build_requires(config_settings, requirements=['wheel'])
File
"/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
line 101, in _get_build_requires
_run_setup()
File
"/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
line 85, in _run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 47, in
from build import *
File
"/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/build.py", line 7,
in
from pip._vendor import pytoml
ImportError: No module named pip._vendor
```
More details: https://github.com/pypa/pip/issues/6190
Here is the comment from the maintainer:
https://github.com/pypa/pip/issues/6163#issuecomment-462014864 that says
pip==19.0.2 fixes that
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade pip to 19.0.2
> -
>
> Key: AIRFLOW-5687
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5687
> Project: Apache Airflow
> Issue Type: Improvement
> Components: dependencies
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Kaxil Naik
>
[jira] [Commented] (AIRFLOW-5687) Upgrade pip to 19.0.2
[
https://issues.apache.org/jira/browse/AIRFLOW-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953826#comment-16953826
]
ASF subversion and git services commented on AIRFLOW-5687:
--
Commit 65efdb92b53f56dbf3edddf3961a99a8b776bd38 in airflow's branch
refs/heads/kaxil-patch-1 from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=65efdb9 ]
[AIRFLOW-5687] Upgrade pip to 19.0.2
> Upgrade pip to 19.0.2
> -
>
> Key: AIRFLOW-5687
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5687
> Project: Apache Airflow
> Issue Type: Improvement
> Components: dependencies
>Affects Versions: 2.0.0, 1.10.6
>Reporter: Kaxil Naik
>Priority: Major
> Fix For: 1.10.6
>
>
> Pendulum==1.4.4 errors
>
> {code}
> Complete output from command /usr/local/bin/python
> /usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py
> get_requires_for_build_wheel /tmp/tmpGuP1ar:
> Traceback (most recent call last):
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 207, in
> main()
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 197, in main
> json_out['return_val'] = hook(**hook_input['kwargs'])
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
> line 54, in get_requires_for_build_wheel
> return hook(config_settings)
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 115, in get_requires_for_build_wheel
> return _get_build_requires(config_settings, requirements=['wheel'])
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 101, in _get_build_requires
> _run_setup()
> File
> "/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
> line 85, in _run_setup
> exec(compile(code, __file__, 'exec'), locals())
> File "setup.py", line 47, in
> from build import *
> File
> "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/build.py", line 7,
> in
> from pip._vendor import pytoml
> ImportError: No module named pip._vendor
> {code}
> More details: https://github.com/pypa/pip/issues/6190
> Here is the comment from the maintainer:
> https://github.com/pypa/pip/issues/6163#issuecomment-462014864 that says
> pip==19.0.2 fixes that
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] kaxil opened a new pull request #6358: [AIRFLOW-5687] Upgrade pip to 19.0.2
kaxil opened a new pull request #6358: [AIRFLOW-5687] Upgrade pip to 19.0.2
URL: https://github.com/apache/airflow/pull/6358
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
Pendulum==1.4.4 errors
Failed build: https://travis-ci.com/ashb/airflow/jobs/246690984
```
Complete output from command /usr/local/bin/python
/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py
get_requires_for_build_wheel /tmp/tmpGuP1ar:
Traceback (most recent call last):
File
"/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
line 207, in
main()
File
"/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
line 197, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File
"/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
line 54, in get_requires_for_build_wheel
return hook(config_settings)
File
"/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
line 115, in get_requires_for_build_wheel
return _get_build_requires(config_settings, requirements=['wheel'])
File
"/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
line 101, in _get_build_requires
_run_setup()
File
"/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
line 85, in _run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 47, in
from build import *
File
"/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/build.py", line 7,
in
from pip._vendor import pytoml
ImportError: No module named pip._vendor
```
More details: https://github.com/pypa/pip/issues/6190
Here is the comment from the maintainer:
https://github.com/pypa/pip/issues/6163#issuecomment-462014864 that says
pip==19.0.2 fixes that
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-5687) Upgrade pip to 19.0.2
Kaxil Naik created AIRFLOW-5687:
---
Summary: Upgrade pip to 19.0.2
Key: AIRFLOW-5687
URL: https://issues.apache.org/jira/browse/AIRFLOW-5687
Project: Apache Airflow
Issue Type: Improvement
Components: dependencies
Affects Versions: 2.0.0, 1.10.6
Reporter: Kaxil Naik
Fix For: 1.10.6
Pendulum==1.4.4 errors
{code}
Complete output from command /usr/local/bin/python
/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py
get_requires_for_build_wheel /tmp/tmpGuP1ar:
Traceback (most recent call last):
File
"/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
line 207, in
main()
File
"/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
line 197, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File
"/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py",
line 54, in get_requires_for_build_wheel
return hook(config_settings)
File
"/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
line 115, in get_requires_for_build_wheel
return _get_build_requires(config_settings, requirements=['wheel'])
File
"/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
line 101, in _get_build_requires
_run_setup()
File
"/tmp/pip-build-env-_krb5E/overlay/lib/python2.7/site-packages/setuptools/build_meta.py",
line 85, in _run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 47, in
from build import *
File "/usr/local/lib/python2.7/site-packages/pip/_vendor/pep517/build.py",
line 7, in
from pip._vendor import pytoml
ImportError: No module named pip._vendor
{code}
More details: https://github.com/pypa/pip/issues/6190
Here is the comment from the maintainer:
https://github.com/pypa/pip/issues/6163#issuecomment-462014864 that says
pip==19.0.2 fixes that
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5608) Scheduler not shutting down executor(s) gracefully
[ https://issues.apache.org/jira/browse/AIRFLOW-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953821#comment-16953821 ] ASF GitHub Bot commented on AIRFLOW-5608: - ashb commented on pull request #6357: Revert "AIRFLOW-5608: Fix bug in SchedulerJob when calling executor.end" URL: https://github.com/apache/airflow/pull/6357 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Scheduler not shutting down executor(s) gracefully > -- > > Key: AIRFLOW-5608 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5608 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.10.5 >Reporter: Kiran >Assignee: Kiran >Priority: Major > Fix For: 1.10.6 > > > If SIGTERM is sent to scheduler, the bug is preventing it from calling > graceful shutdown of executor(s). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes
codecov-io edited a comment on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes URL: https://github.com/apache/airflow/pull/6321#issuecomment-543213812 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=h1) Report > Merging [#6321](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/ac42428bf530c259ab1b0dca08458c1ebf49b04a?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6321 +/- ## == + Coverage 80.08% 80.08% +<.01% == Files 616 616 Lines 3579735797 == + Hits2866828669 +1 + Misses 7129 7128 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6321/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.28% <0%> (-0.51%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6321/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (ø)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6321/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90% <0%> (+5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=footer). Last update [ac42428...dd8715c](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes
codecov-io edited a comment on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes URL: https://github.com/apache/airflow/pull/6321#issuecomment-543213812 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=h1) Report > Merging [#6321](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/ac42428bf530c259ab1b0dca08458c1ebf49b04a?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6321 +/- ## == + Coverage 80.08% 80.08% +<.01% == Files 616 616 Lines 3579735797 == + Hits2866828669 +1 + Misses 7129 7128 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6321/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.28% <0%> (-0.51%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6321/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (ø)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6321/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90% <0%> (+5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=footer). Last update [ac42428...dd8715c](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes
codecov-io commented on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes URL: https://github.com/apache/airflow/pull/6321#issuecomment-543213812 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=h1) Report > Merging [#6321](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/ac42428bf530c259ab1b0dca08458c1ebf49b04a?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6321 +/- ## == + Coverage 80.08% 80.08% +<.01% == Files 616 616 Lines 3579735797 == + Hits2866828669 +1 + Misses 7129 7128 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/models/taskinstance.py](https://codecov.io/gh/apache/airflow/pull/6321/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvdGFza2luc3RhbmNlLnB5) | `93.28% <0%> (-0.51%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6321/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (ø)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6321/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90% <0%> (+5%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=footer). Last update [ac42428...dd8715c](https://codecov.io/gh/apache/airflow/pull/6321?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5608) Scheduler not shutting down executor(s) gracefully
[ https://issues.apache.org/jira/browse/AIRFLOW-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953822#comment-16953822 ] ASF subversion and git services commented on AIRFLOW-5608: -- Commit e35829e472c4f0b014dcbb71d1f35785fef6ca8b in airflow's branch refs/heads/master from Ash Berlin-Taylor [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e35829e ] Revert "AIRFLOW-5608: Gracefully stop executor when SIGTERM is received by SchedulerJib (#6274)" (#6357) This reverts commit 7c9b44dbfc72df951aadbda006229977beb6d47c. > Scheduler not shutting down executor(s) gracefully > -- > > Key: AIRFLOW-5608 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5608 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.10.5 >Reporter: Kiran >Assignee: Kiran >Priority: Major > Fix For: 1.10.6 > > > If SIGTERM is sent to scheduler, the bug is preventing it from calling > graceful shutdown of executor(s). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb merged pull request #6357: Revert "AIRFLOW-5608: Fix bug in SchedulerJob when calling executor.end"
ashb merged pull request #6357: Revert "AIRFLOW-5608: Fix bug in SchedulerJob when calling executor.end" URL: https://github.com/apache/airflow/pull/6357 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ToxaZ commented on issue #6336: [AIRFLOW-5073] Change SQLSensor to optionally treat NULL as keep poking
ToxaZ commented on issue #6336: [AIRFLOW-5073] Change SQLSensor to optionally treat NULL as keep poking URL: https://github.com/apache/airflow/pull/6336#issuecomment-543188183 bump Looks like CI issues does not belong to my PR This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5608) Scheduler not shutting down executor(s) gracefully
[ https://issues.apache.org/jira/browse/AIRFLOW-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953742#comment-16953742 ] ASF GitHub Bot commented on AIRFLOW-5608: - ashb commented on pull request #6357: Revert "AIRFLOW-5608: Fix bug in SchedulerJob when calling executor.end" URL: https://github.com/apache/airflow/pull/6357 Reverts apache/airflow#6274 It caused the LocalExecutor to not shutdown when given a SIGINT with this stack trace. ``` ^CTraceback (most recent call last): File "/Users/ash/code/python/incubator-airflow/airflow/jobs/scheduler_job.py", line 1309, in _execute self._execute_helper() File "/Users/ash/code/python/incubator-airflow/airflow/jobs/scheduler_job.py", line 1435, in _execute_helper sleep(sleep_length) File "/Users/ash/code/python/incubator-airflow/airflow/bin/cli.py", line 84, in sigint_handler sys.exit(0) SystemExit: 0 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/ash/.virtualenvs/airflow/bin/airflow", line 7, in exec(compile(f.read(), __file__, 'exec')) File "/Users/ash/code/python/incubator-airflow/airflow/bin/airflow", line 39, in args.func(args) File "/Users/ash/code/python/incubator-airflow/airflow/utils/cli.py", line 73, in wrapper return f(*args, **kwargs) File "/Users/ash/code/python/incubator-airflow/airflow/bin/cli.py", line 1124, in scheduler job.run() File "/Users/ash/code/python/incubator-airflow/airflow/jobs/base_job.py", line 217, in run self._execute() File "/Users/ash/code/python/incubator-airflow/airflow/jobs/scheduler_job.py", line 1314, in _execute self.executor.end() File "/Users/ash/code/python/incubator-airflow/airflow/executors/local_executor.py", line 229, in end self.impl.end() File "/Users/ash/code/python/incubator-airflow/airflow/executors/local_executor.py", line 205, in end self.queue.put((None, None)) File "", line 2, in put File "/Users/ash/.homebrew/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/managers.py", line 796, in _callmethod kind, result = conn.recv() File "/Users/ash/.homebrew/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 250, in recv buf = self._recv_bytes() File "/Users/ash/.homebrew/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes buf = self._recv(4) File "/Users/ash/.homebrew/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 383, in _recv raise EOFError EOFError ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Scheduler not shutting down executor(s) gracefully > -- > > Key: AIRFLOW-5608 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5608 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.10.5 >Reporter: Kiran >Assignee: Kiran >Priority: Major > Fix For: 1.10.6 > > > If SIGTERM is sent to scheduler, the bug is preventing it from calling > graceful shutdown of executor(s). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5608) Scheduler not shutting down executor(s) gracefully
[ https://issues.apache.org/jira/browse/AIRFLOW-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-5608. Resolution: Cannot Reproduce And the proposed (and merged, but now unmerged) PR made it worse. If you can give more detailed repro steps lets re-open this. > Scheduler not shutting down executor(s) gracefully > -- > > Key: AIRFLOW-5608 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5608 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.10.5 >Reporter: Kiran >Assignee: Kiran >Priority: Major > Fix For: 1.10.6 > > > If SIGTERM is sent to scheduler, the bug is preventing it from calling > graceful shutdown of executor(s). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (AIRFLOW-5608) Scheduler not shutting down executor(s) gracefully
[ https://issues.apache.org/jira/browse/AIRFLOW-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor reopened AIRFLOW-5608: > Scheduler not shutting down executor(s) gracefully > -- > > Key: AIRFLOW-5608 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5608 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.10.5 >Reporter: Kiran >Assignee: Kiran >Priority: Major > Fix For: 1.10.6 > > > If SIGTERM is sent to scheduler, the bug is preventing it from calling > graceful shutdown of executor(s). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-5608) Scheduler not shutting down executor(s) gracefully
[ https://issues.apache.org/jira/browse/AIRFLOW-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953740#comment-16953740 ] ASF subversion and git services commented on AIRFLOW-5608: -- Commit 84c13d2eb3e5a9300adc7d579deeb16abefca8d1 in airflow's branch refs/heads/revert-6274-feature/AIRFLOW-5608-scheduler-bug-fix from Ash Berlin-Taylor [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=84c13d2 ] Revert "AIRFLOW-5608: Gracefully stop executor when SIGTERM is received by SchedulerJib (#6274)" This reverts commit 7c9b44dbfc72df951aadbda006229977beb6d47c. > Scheduler not shutting down executor(s) gracefully > -- > > Key: AIRFLOW-5608 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5608 > Project: Apache Airflow > Issue Type: Bug > Components: scheduler >Affects Versions: 1.10.5 >Reporter: Kiran >Assignee: Kiran >Priority: Major > Fix For: 1.10.6 > > > If SIGTERM is sent to scheduler, the bug is preventing it from calling > graceful shutdown of executor(s). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] ashb opened a new pull request #6357: Revert "AIRFLOW-5608: Fix bug in SchedulerJob when calling executor.end"
ashb opened a new pull request #6357: Revert "AIRFLOW-5608: Fix bug in SchedulerJob when calling executor.end" URL: https://github.com/apache/airflow/pull/6357 Reverts apache/airflow#6274 It caused the LocalExecutor to not shutdown when given a SIGINT with this stack trace. ``` ^CTraceback (most recent call last): File "/Users/ash/code/python/incubator-airflow/airflow/jobs/scheduler_job.py", line 1309, in _execute self._execute_helper() File "/Users/ash/code/python/incubator-airflow/airflow/jobs/scheduler_job.py", line 1435, in _execute_helper sleep(sleep_length) File "/Users/ash/code/python/incubator-airflow/airflow/bin/cli.py", line 84, in sigint_handler sys.exit(0) SystemExit: 0 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/ash/.virtualenvs/airflow/bin/airflow", line 7, in exec(compile(f.read(), __file__, 'exec')) File "/Users/ash/code/python/incubator-airflow/airflow/bin/airflow", line 39, in args.func(args) File "/Users/ash/code/python/incubator-airflow/airflow/utils/cli.py", line 73, in wrapper return f(*args, **kwargs) File "/Users/ash/code/python/incubator-airflow/airflow/bin/cli.py", line 1124, in scheduler job.run() File "/Users/ash/code/python/incubator-airflow/airflow/jobs/base_job.py", line 217, in run self._execute() File "/Users/ash/code/python/incubator-airflow/airflow/jobs/scheduler_job.py", line 1314, in _execute self.executor.end() File "/Users/ash/code/python/incubator-airflow/airflow/executors/local_executor.py", line 229, in end self.impl.end() File "/Users/ash/code/python/incubator-airflow/airflow/executors/local_executor.py", line 205, in end self.queue.put((None, None)) File "", line 2, in put File "/Users/ash/.homebrew/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/managers.py", line 796, in _callmethod kind, result = conn.recv() File "/Users/ash/.homebrew/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 250, in recv buf = self._recv_bytes() File "/Users/ash/.homebrew/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes buf = self._recv(4) File "/Users/ash/.homebrew/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 383, in _recv raise EOFError EOFError ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on issue #6274: AIRFLOW-5608: Fix bug in SchedulerJob when calling executor.end
ashb commented on issue #6274: AIRFLOW-5608: Fix bug in SchedulerJob when calling executor.end URL: https://github.com/apache/airflow/pull/6274#issuecomment-543176050 @kpathak13 This PR breaks the Scheduler stopping with LocalExecutor: ``` Traceback (most recent call last): File "/Users/ash/code/python/incubator-airflow/airflow/jobs/scheduler_job.py", line 1358, in _execute self._execute_helper() File "/Users/ash/code/python/incubator-airflow/airflow/jobs/scheduler_job.py", line 1485, in _execute_helper sleep(sleep_length) File "/Users/ash/code/python/incubator-airflow/airflow/bin/cli.py", line 91, in sigint_handler sys.exit(0) SystemExit: 0 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/ash/.virtualenvs/airflow/bin/airflow", line 7, in exec(compile(f.read(), __file__, 'exec')) File "/Users/ash/code/python/incubator-airflow/airflow/bin/airflow", line 37, in args.func(args) File "/Users/ash/code/python/incubator-airflow/airflow/utils/cli.py", line 74, in wrapper return f(*args, **kwargs) File "/Users/ash/code/python/incubator-airflow/airflow/bin/cli.py", line 1042, in scheduler job.run() File "/Users/ash/code/python/incubator-airflow/airflow/jobs/base_job.py", line 222, in run self._execute() File "/Users/ash/code/python/incubator-airflow/airflow/jobs/scheduler_job.py", line 1363, in _execute self.executor.end() File "/Users/ash/code/python/incubator-airflow/airflow/executors/local_executor.py", line 233, in end self.impl.end() File "/Users/ash/code/python/incubator-airflow/airflow/executors/local_executor.py", line 209, in end self.queue.put((None, None)) File "", line 2, in put File "/Users/ash/.homebrew/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/managers.py", line 796, in _callmethod kind, result = conn.recv() File "/Users/ash/.homebrew/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 250, in recv buf = self._recv_bytes() File "/Users/ash/.homebrew/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes buf = self._recv(4) File "/Users/ash/.homebrew/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 383, in _recv raise EOFError EOFError ``` This makes what _was_ a clean shutdown for me an abnormal shutdown. I am reverting this change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5684) Docker-compose-kubernetes still used by breeze
[ https://issues.apache.org/jira/browse/AIRFLOW-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953714#comment-16953714 ] ASF subversion and git services commented on AIRFLOW-5684: -- Commit f8794d888fcb032fd620a27c1c54312145e9ffb6 in airflow's branch refs/heads/master from Jarek Potiuk [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f8794d8 ] [AIRFLOW-5684] docker-compose-kubernetes still used (#6353) > Docker-compose-kubernetes still used by breeze > -- > > Key: AIRFLOW-5684 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5684 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > > After transitioning to KinD for kubernetes testing, we no longer need the > docker-compose-kubernetes.yaml. However it still looks for this > docker-compose in the breeze command. This PR removes these unneeded commands. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk merged pull request #6353: [AIRFLOW-5684] docker-compose-kubernetes still used
potiuk merged pull request #6353: [AIRFLOW-5684] docker-compose-kubernetes still used URL: https://github.com/apache/airflow/pull/6353 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5684) Docker-compose-kubernetes still used by breeze
[ https://issues.apache.org/jira/browse/AIRFLOW-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953712#comment-16953712 ] ASF GitHub Bot commented on AIRFLOW-5684: - potiuk commented on pull request #6353: [AIRFLOW-5684] docker-compose-kubernetes still used URL: https://github.com/apache/airflow/pull/6353 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Docker-compose-kubernetes still used by breeze > -- > > Key: AIRFLOW-5684 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5684 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > > After transitioning to KinD for kubernetes testing, we no longer need the > docker-compose-kubernetes.yaml. However it still looks for this > docker-compose in the breeze command. This PR removes these unneeded commands. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] feluelle commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
feluelle commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r335985718 ## File path: CHANGELOG.txt ## @@ -1,3 +1,134 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Features + +- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546) +- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407) +- [AIRFLOW-4939] Add default_task_retries config (#5570) +- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130) +- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) +- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743) + +Improvements + +- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738) +- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737) +- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736) +- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330) +- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034) +- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194) +- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334) +- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) +- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) +- [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of the aws hook (#6303) +- [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302) +- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304) +- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309) +- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153) +- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) +- [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) +- [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062) +- [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059) +- [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948) +- [AIRFLOW-3888] HA for Hive metastore connection (#4708) +- [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873) +- [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768) +- [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214) +- [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696) +- [AIRFLOW-2891] allow configurable docker_operator container name (#5689) +- [AIRFLOW-4285] Update task dependency context definition and usage (#5079) +- [AIRFLOW-5142] Fixed flaky Cassandra test (#5758) +- [AIRFLOW-5218] Less polling of AWS Batch job status (#5825) +- [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589) +- [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339) +- [AIRFLOW-5560] Allow no confirmation on reset dags in `airflow backfill` command (#6195) +- [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879) +- [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159) +- [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154) +- [AIRFLOW-4835] Refactor operator render_template (#5461) + +Bug Fixes +" +- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator (#6078) +- [Airflow 4923] Fix Databricks hook leaks API secret in logs (#5635) +- [AIRFLOW-5133] Keep original env state in provide_gcp_credential_file (#5747) +- [AIRFLOW-5497] Update docstring in ``airflow/utils/dag_processing.py`` (#6314) +- Revert/and then rework "[AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)" to improve performance (#5908) +- [AIRFLOW-5634] Don't allow editing of DagModelView (#6308) +- [AIRFLOW-4309] Remove Broken Dag error after Dag is deleted (#6102) +- [AIRFLOW-5387] Fix "show paused" pagination bug (#6100) +- [AIRFLOW-5489] Remove unneeded assignment of variable (#6106) +- [AIRFLOW-5491] mark_tasks pydoc is incorrect (#6108) +- [AIRFLOW-5492] added missing docstrings (#6107) +- [AIRFLOW-5503] Fix tree view layout on HDPI screen (#6125) +- [AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101) +- [AIRFLOW-3857] spark_submit_hook cannot kill driver pod in Kubernetes (#4678) +- [AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent Tasks' (#5909) +- [AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185) +- [AIRFLOW-5306] Fix the display of links when they contain special characters (#5904) +- [AIRFLOW-3705] Fix PostgresHook get_conn to use conn_name_attr (#5841) +- [AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler shutdown on SIGTERM (#6237) +- [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView (#6307) +- [AIRFLOW-4833] Allow to set Jinja env options in
[GitHub] [airflow] kaxil commented on a change in pull request #6182: [AIRFLOW-3877] -Changing full file path in scheduler
kaxil commented on a change in pull request #6182: [AIRFLOW-3877] -Changing full file path in scheduler URL: https://github.com/apache/airflow/pull/6182#discussion_r335985847 ## File path: airflow/jobs/scheduler_job.py ## @@ -1092,7 +1099,7 @@ def _enqueue_task_instances_with_queued_state(self, simple_dag_bag, ignore_task_deps=False, ignore_ti_state=False, pool=simple_task_instance.pool, -file_path=simple_dag.full_filepath, +file_path=change_file_path(simple_dag.full_filepath), Review comment: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (AIRFLOW-5684) Docker-compose-kubernetes still used by breeze
[ https://issues.apache.org/jira/browse/AIRFLOW-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Imberman updated AIRFLOW-5684: - Description: After transitioning to KinD for kubernetes testing, we no longer need the docker-compose-kubernetes.yaml. However it still looks for this docker-compose in the breeze command. This PR removes these unneeded commands. > Docker-compose-kubernetes still used by breeze > -- > > Key: AIRFLOW-5684 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5684 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Priority: Major > > After transitioning to KinD for kubernetes testing, we no longer need the > docker-compose-kubernetes.yaml. However it still looks for this > docker-compose in the breeze command. This PR removes these unneeded commands. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk edited a comment on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes
potiuk edited a comment on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes URL: https://github.com/apache/airflow/pull/6321#issuecomment-543160176 One side-effect of this change is that it will not run tests + pylint in branch pushes (it only runs tests when there is a PR). This however is a nice side effect - I am not porting it to v1-10-test branch but if there will be any other branch we want to push, it will not run the tests and will not block Travis. Only if someone makes PR, the full set of tests will be run. I think this is rather nice behaviour. We will have to adapt it once I port it to v1-10-test (but this will only happen after I port the PROD image there). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk edited a comment on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes
potiuk edited a comment on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes URL: https://github.com/apache/airflow/pull/6321#issuecomment-543160176 One side-effect of this change is that it will not run tests + pylint in branch pushes (it only runs tests when there is a PR). This however is a nice side effect - I am not porting it to v1-10-test branch but if there will be any other branch we want to push, it will not run the tests and will not block Travis. Only if someone makes PR the full set of tests will be run. I think this is rather nice behaviour. We will have to adapt it once I port it to v1-10-test (but this will only happen after I port the PROD image there). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk edited a comment on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes
potiuk edited a comment on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes URL: https://github.com/apache/airflow/pull/6321#issuecomment-543160176 One side-effect of this change is that it will not run tests + pylint in branch pushes (it only runs tests when there is a PR). This hover is a nice side effect - I am not porting it to v1-10-test branch but if there will be any other branch we want to push, it will not run the tests and will not block Travis. Only if someone makes PR the full set of tests will be run. I think this is rather nice behaviour. We will have to adapt it once I port it to v1-10-test (but this will only happen after I port the PROD image there). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk edited a comment on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes
potiuk edited a comment on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes URL: https://github.com/apache/airflow/pull/6321#issuecomment-543160176 One side-effect of this change is that it will not run tests + pylint in branch pushes (it only run tests when there is a PR). This hover is a nice side effect - I am not porting it to v1-10-test branch but if there will be any other branch we want to push, it will not run the tests and will not block Travis. Only if someone makes PR the full set of tests will be run. I think this is rather nice behaviour. We will have to adapt it once I port it to v1-10-test (but this will only happen after I port the PROD image there). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes
potiuk commented on issue #6321: [AIRFLOW-5649] Skips some tests for non relevant changes URL: https://github.com/apache/airflow/pull/6321#issuecomment-543160176 One drawback of this change is that it will not run tests + pylint in branch pushes (it only run tests when there is a PR). This hover is a nice side effect - I am not porting it to v1-10-test branch but if there will be any other branch we want to push, it will not run the tests and will not block Travis. Only if someone makes PR the full set of tests will be run. I think this is rather nice behaviour. We will have to adapt it once I port it to v1-10-test (but this will only happen after I port the PROD image there). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r335984477 ## File path: CHANGELOG.txt ## @@ -1,3 +1,134 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Features + +- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546) +- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407) +- [AIRFLOW-4939] Add default_task_retries config (#5570) +- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130) +- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) +- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743) + +Improvements + +- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738) +- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737) +- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736) +- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330) +- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034) +- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194) +- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334) +- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) +- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) +- [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of the aws hook (#6303) +- [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302) +- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304) +- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309) +- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153) +- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) +- [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) +- [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062) +- [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059) +- [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948) +- [AIRFLOW-3888] HA for Hive metastore connection (#4708) +- [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873) +- [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768) +- [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214) +- [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696) +- [AIRFLOW-2891] allow configurable docker_operator container name (#5689) +- [AIRFLOW-4285] Update task dependency context definition and usage (#5079) +- [AIRFLOW-5142] Fixed flaky Cassandra test (#5758) +- [AIRFLOW-5218] Less polling of AWS Batch job status (#5825) +- [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589) +- [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339) +- [AIRFLOW-5560] Allow no confirmation on reset dags in `airflow backfill` command (#6195) +- [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879) +- [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159) +- [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154) +- [AIRFLOW-4835] Refactor operator render_template (#5461) + +Bug Fixes +" +- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator Review comment: Yup it is This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
feluelle commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r335984291 ## File path: CHANGELOG.txt ## @@ -1,3 +1,134 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Features + +- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546) +- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407) +- [AIRFLOW-4939] Add default_task_retries config (#5570) +- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130) +- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) +- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743) + +Improvements + +- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738) +- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737) +- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736) +- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330) +- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034) +- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194) +- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334) +- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) +- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) +- [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of the aws hook (#6303) +- [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302) +- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304) +- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309) +- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153) +- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) +- [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) +- [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062) +- [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059) +- [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948) +- [AIRFLOW-3888] HA for Hive metastore connection (#4708) +- [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873) +- [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768) +- [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214) +- [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696) +- [AIRFLOW-2891] allow configurable docker_operator container name (#5689) +- [AIRFLOW-4285] Update task dependency context definition and usage (#5079) +- [AIRFLOW-5142] Fixed flaky Cassandra test (#5758) +- [AIRFLOW-5218] Less polling of AWS Batch job status (#5825) +- [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589) +- [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339) +- [AIRFLOW-5560] Allow no confirmation on reset dags in `airflow backfill` command (#6195) +- [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879) +- [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159) +- [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154) +- [AIRFLOW-4835] Refactor operator render_template (#5461) + +Bug Fixes +" +- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator +- [Airflow 4923] Fix Databricks hook leaks API secret in logs (#5635) +- [AIRFLOW-5133] Keep original env state in provide_gcp_credential_file (#5747) +- [AIRFLOW-5497] Update docstring in ``airflow/utils/dag_processing.py`` (#6314) +- Revert/and then rework "[AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)" to improve performance +- [AIRFLOW-5634] Don't allow editing of DagModelView (#6308) +- [AIRFLOW-4309] Remove Broken Dag error after Dag is deleted (#6102) +- [AIRFLOW-5387] Fix "show paused" pagination bug (#6100) +- [AIRFLOW-5489] Remove unneeded assignment of variable (#6106) +- [AIRFLOW-5491] mark_tasks pydoc is incorrect (#6108) +- [AIRFLOW-5492] added missing docstrings (#6107) +- [AIRFLOW-5503] Fix tree view layout on HDPI screen (#6125) +- [AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101) +- [AIRFLOW-3857] spark_submit_hook cannot kill driver pod in Kubernetes (#4678) +- [AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent Tasks' (#5909) +- [AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185) +- [AIRFLOW-5306] Fix the display of links when they contain special characters (#5904) +- [AIRFLOW-3705] Fix PostgresHook get_conn to use conn_name_attr (#5841) +- [AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler shutdown on SIGTERM (#6237) +- [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView (#6307) +- [AIRFLOW-4833] Allow to set Jinja env options in DAG declaration
[GitHub] [airflow] feluelle commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
feluelle commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r335983637 ## File path: CHANGELOG.txt ## @@ -1,3 +1,134 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Features + +- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546) +- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407) +- [AIRFLOW-4939] Add default_task_retries config (#5570) +- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130) +- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) +- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743) + +Improvements + +- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738) +- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737) +- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736) +- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330) +- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034) +- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194) +- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334) +- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) +- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) +- [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of the aws hook (#6303) +- [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302) +- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304) +- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309) +- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153) +- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) +- [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) +- [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062) +- [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059) +- [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948) +- [AIRFLOW-3888] HA for Hive metastore connection (#4708) +- [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873) +- [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768) +- [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214) +- [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696) +- [AIRFLOW-2891] allow configurable docker_operator container name (#5689) +- [AIRFLOW-4285] Update task dependency context definition and usage (#5079) +- [AIRFLOW-5142] Fixed flaky Cassandra test (#5758) +- [AIRFLOW-5218] Less polling of AWS Batch job status (#5825) +- [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589) +- [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339) +- [AIRFLOW-5560] Allow no confirmation on reset dags in `airflow backfill` command (#6195) +- [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879) +- [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159) +- [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154) +- [AIRFLOW-4835] Refactor operator render_template (#5461) + +Bug Fixes +" +- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator +- [Airflow 4923] Fix Databricks hook leaks API secret in logs (#5635) +- [AIRFLOW-5133] Keep original env state in provide_gcp_credential_file (#5747) +- [AIRFLOW-5497] Update docstring in ``airflow/utils/dag_processing.py`` (#6314) +- Revert/and then rework "[AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)" to improve performance +- [AIRFLOW-5634] Don't allow editing of DagModelView (#6308) +- [AIRFLOW-4309] Remove Broken Dag error after Dag is deleted (#6102) +- [AIRFLOW-5387] Fix "show paused" pagination bug (#6100) +- [AIRFLOW-5489] Remove unneeded assignment of variable (#6106) +- [AIRFLOW-5491] mark_tasks pydoc is incorrect (#6108) +- [AIRFLOW-5492] added missing docstrings (#6107) +- [AIRFLOW-5503] Fix tree view layout on HDPI screen (#6125) +- [AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101) +- [AIRFLOW-3857] spark_submit_hook cannot kill driver pod in Kubernetes (#4678) +- [AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent Tasks' (#5909) +- [AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185) +- [AIRFLOW-5306] Fix the display of links when they contain special characters (#5904) +- [AIRFLOW-3705] Fix PostgresHook get_conn to use conn_name_attr (#5841) +- [AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler shutdown on SIGTERM (#6237) +- [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView (#6307) +- [AIRFLOW-4833] Allow to set Jinja env options in DAG declaration
[GitHub] [airflow] feluelle commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
feluelle commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r335984077 ## File path: CHANGELOG.txt ## @@ -1,3 +1,134 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Features + +- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546) +- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407) +- [AIRFLOW-4939] Add default_task_retries config (#5570) +- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130) +- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) +- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743) + +Improvements + +- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738) +- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737) +- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736) +- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330) +- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034) +- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194) +- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334) +- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) +- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) +- [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of the aws hook (#6303) +- [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302) +- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304) +- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309) +- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153) +- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) +- [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) +- [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062) +- [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059) +- [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948) +- [AIRFLOW-3888] HA for Hive metastore connection (#4708) +- [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873) +- [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768) +- [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214) +- [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696) +- [AIRFLOW-2891] allow configurable docker_operator container name (#5689) +- [AIRFLOW-4285] Update task dependency context definition and usage (#5079) +- [AIRFLOW-5142] Fixed flaky Cassandra test (#5758) +- [AIRFLOW-5218] Less polling of AWS Batch job status (#5825) +- [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589) +- [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339) +- [AIRFLOW-5560] Allow no confirmation on reset dags in `airflow backfill` command (#6195) +- [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879) +- [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159) +- [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154) +- [AIRFLOW-4835] Refactor operator render_template (#5461) + +Bug Fixes +" +- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator +- [Airflow 4923] Fix Databricks hook leaks API secret in logs (#5635) +- [AIRFLOW-5133] Keep original env state in provide_gcp_credential_file (#5747) +- [AIRFLOW-5497] Update docstring in ``airflow/utils/dag_processing.py`` (#6314) +- Revert/and then rework "[AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)" to improve performance +- [AIRFLOW-5634] Don't allow editing of DagModelView (#6308) +- [AIRFLOW-4309] Remove Broken Dag error after Dag is deleted (#6102) +- [AIRFLOW-5387] Fix "show paused" pagination bug (#6100) +- [AIRFLOW-5489] Remove unneeded assignment of variable (#6106) +- [AIRFLOW-5491] mark_tasks pydoc is incorrect (#6108) +- [AIRFLOW-5492] added missing docstrings (#6107) +- [AIRFLOW-5503] Fix tree view layout on HDPI screen (#6125) +- [AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101) +- [AIRFLOW-3857] spark_submit_hook cannot kill driver pod in Kubernetes (#4678) +- [AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent Tasks' (#5909) +- [AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185) +- [AIRFLOW-5306] Fix the display of links when they contain special characters (#5904) +- [AIRFLOW-3705] Fix PostgresHook get_conn to use conn_name_attr (#5841) +- [AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler shutdown on SIGTERM (#6237) +- [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView (#6307) +- [AIRFLOW-4833] Allow to set Jinja env options in DAG declaration
[GitHub] [airflow] feluelle commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
feluelle commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r335982970 ## File path: CHANGELOG.txt ## @@ -1,3 +1,134 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Features + +- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546) +- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407) +- [AIRFLOW-4939] Add default_task_retries config (#5570) +- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130) +- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) +- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743) + +Improvements + +- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738) +- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737) +- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736) +- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330) +- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034) +- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194) +- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334) +- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) +- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) +- [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of the aws hook (#6303) +- [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302) +- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304) +- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309) +- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153) +- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) +- [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) +- [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062) +- [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059) +- [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948) +- [AIRFLOW-3888] HA for Hive metastore connection (#4708) +- [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873) +- [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768) +- [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214) +- [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696) +- [AIRFLOW-2891] allow configurable docker_operator container name (#5689) +- [AIRFLOW-4285] Update task dependency context definition and usage (#5079) +- [AIRFLOW-5142] Fixed flaky Cassandra test (#5758) +- [AIRFLOW-5218] Less polling of AWS Batch job status (#5825) +- [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589) +- [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339) +- [AIRFLOW-5560] Allow no confirmation on reset dags in `airflow backfill` command (#6195) +- [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879) +- [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159) +- [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154) +- [AIRFLOW-4835] Refactor operator render_template (#5461) + +Bug Fixes +" +- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator +- [Airflow 4923] Fix Databricks hook leaks API secret in logs (#5635) +- [AIRFLOW-5133] Keep original env state in provide_gcp_credential_file (#5747) +- [AIRFLOW-5497] Update docstring in ``airflow/utils/dag_processing.py`` (#6314) +- Revert/and then rework "[AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)" to improve performance +- [AIRFLOW-5634] Don't allow editing of DagModelView (#6308) +- [AIRFLOW-4309] Remove Broken Dag error after Dag is deleted (#6102) +- [AIRFLOW-5387] Fix "show paused" pagination bug (#6100) +- [AIRFLOW-5489] Remove unneeded assignment of variable (#6106) +- [AIRFLOW-5491] mark_tasks pydoc is incorrect (#6108) +- [AIRFLOW-5492] added missing docstrings (#6107) +- [AIRFLOW-5503] Fix tree view layout on HDPI screen (#6125) +- [AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101) +- [AIRFLOW-3857] spark_submit_hook cannot kill driver pod in Kubernetes (#4678) +- [AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent Tasks' (#5909) +- [AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185) +- [AIRFLOW-5306] Fix the display of links when they contain special characters (#5904) +- [AIRFLOW-3705] Fix PostgresHook get_conn to use conn_name_attr (#5841) +- [AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler shutdown on SIGTERM (#6237) +- [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView (#6307) +- [AIRFLOW-4833] Allow to set Jinja env options in DAG declaration
[GitHub] [airflow] ashb commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
ashb commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r335984007 ## File path: CHANGELOG.txt ## @@ -1,3 +1,134 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Features + +- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546) +- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407) +- [AIRFLOW-4939] Add default_task_retries config (#5570) +- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130) +- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) +- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743) + +Improvements + +- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738) +- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737) +- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736) +- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330) +- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034) +- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194) +- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334) +- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) +- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) +- [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of the aws hook (#6303) +- [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302) +- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304) +- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309) +- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153) +- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) +- [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) +- [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062) +- [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059) +- [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948) +- [AIRFLOW-3888] HA for Hive metastore connection (#4708) +- [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873) +- [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768) +- [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214) +- [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696) +- [AIRFLOW-2891] allow configurable docker_operator container name (#5689) +- [AIRFLOW-4285] Update task dependency context definition and usage (#5079) +- [AIRFLOW-5142] Fixed flaky Cassandra test (#5758) +- [AIRFLOW-5218] Less polling of AWS Batch job status (#5825) +- [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589) +- [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339) +- [AIRFLOW-5560] Allow no confirmation on reset dags in `airflow backfill` command (#6195) +- [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879) +- [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159) +- [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154) +- [AIRFLOW-4835] Refactor operator render_template (#5461) + +Bug Fixes +" +- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator Review comment: I guess this is to do with how it got merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r335983532 ## File path: CHANGELOG.txt ## @@ -1,3 +1,134 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Features + +- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546) +- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407) +- [AIRFLOW-4939] Add default_task_retries config (#5570) +- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130) +- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) +- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743) + +Improvements + +- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738) +- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737) +- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736) +- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330) +- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034) +- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194) +- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334) +- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) +- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) +- [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of the aws hook (#6303) +- [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302) +- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304) +- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309) +- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153) +- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) +- [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) +- [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062) +- [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059) +- [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948) +- [AIRFLOW-3888] HA for Hive metastore connection (#4708) +- [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873) +- [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768) +- [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214) +- [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696) +- [AIRFLOW-2891] allow configurable docker_operator container name (#5689) +- [AIRFLOW-4285] Update task dependency context definition and usage (#5079) +- [AIRFLOW-5142] Fixed flaky Cassandra test (#5758) +- [AIRFLOW-5218] Less polling of AWS Batch job status (#5825) +- [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589) +- [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339) +- [AIRFLOW-5560] Allow no confirmation on reset dags in `airflow backfill` command (#6195) +- [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879) +- [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159) +- [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154) +- [AIRFLOW-4835] Refactor operator render_template (#5461) + +Bug Fixes +" +- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator +- [Airflow 4923] Fix Databricks hook leaks API secret in logs (#5635) +- [AIRFLOW-5133] Keep original env state in provide_gcp_credential_file (#5747) +- [AIRFLOW-5497] Update docstring in ``airflow/utils/dag_processing.py`` (#6314) +- Revert/and then rework "[AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)" to improve performance +- [AIRFLOW-5634] Don't allow editing of DagModelView (#6308) +- [AIRFLOW-4309] Remove Broken Dag error after Dag is deleted (#6102) +- [AIRFLOW-5387] Fix "show paused" pagination bug (#6100) +- [AIRFLOW-5489] Remove unneeded assignment of variable (#6106) +- [AIRFLOW-5491] mark_tasks pydoc is incorrect (#6108) +- [AIRFLOW-5492] added missing docstrings (#6107) +- [AIRFLOW-5503] Fix tree view layout on HDPI screen (#6125) +- [AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101) +- [AIRFLOW-3857] spark_submit_hook cannot kill driver pod in Kubernetes (#4678) +- [AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent Tasks' (#5909) +- [AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185) +- [AIRFLOW-5306] Fix the display of links when they contain special characters (#5904) +- [AIRFLOW-3705] Fix PostgresHook get_conn to use conn_name_attr (#5841) +- [AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler shutdown on SIGTERM (#6237) +- [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView (#6307) +- [AIRFLOW-4833] Allow to set Jinja env options in DAG declaration
[GitHub] [airflow] kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r335983177 ## File path: CHANGELOG.txt ## @@ -1,3 +1,134 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Features + +- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546) +- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407) +- [AIRFLOW-4939] Add default_task_retries config (#5570) +- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130) +- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) +- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743) + +Improvements + +- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738) +- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737) +- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736) +- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330) +- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034) +- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194) +- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334) +- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) +- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) +- [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of the aws hook (#6303) +- [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302) +- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304) +- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309) +- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153) +- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) +- [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) +- [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062) +- [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059) +- [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948) +- [AIRFLOW-3888] HA for Hive metastore connection (#4708) +- [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873) +- [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768) +- [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214) +- [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696) +- [AIRFLOW-2891] allow configurable docker_operator container name (#5689) +- [AIRFLOW-4285] Update task dependency context definition and usage (#5079) +- [AIRFLOW-5142] Fixed flaky Cassandra test (#5758) +- [AIRFLOW-5218] Less polling of AWS Batch job status (#5825) +- [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589) +- [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339) +- [AIRFLOW-5560] Allow no confirmation on reset dags in `airflow backfill` command (#6195) +- [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879) +- [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159) +- [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154) +- [AIRFLOW-4835] Refactor operator render_template (#5461) + +Bug Fixes +" +- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator +- [Airflow 4923] Fix Databricks hook leaks API secret in logs (#5635) +- [AIRFLOW-5133] Keep original env state in provide_gcp_credential_file (#5747) +- [AIRFLOW-5497] Update docstring in ``airflow/utils/dag_processing.py`` (#6314) +- Revert/and then rework "[AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)" to improve performance +- [AIRFLOW-5634] Don't allow editing of DagModelView (#6308) +- [AIRFLOW-4309] Remove Broken Dag error after Dag is deleted (#6102) +- [AIRFLOW-5387] Fix "show paused" pagination bug (#6100) +- [AIRFLOW-5489] Remove unneeded assignment of variable (#6106) +- [AIRFLOW-5491] mark_tasks pydoc is incorrect (#6108) +- [AIRFLOW-5492] added missing docstrings (#6107) +- [AIRFLOW-5503] Fix tree view layout on HDPI screen (#6125) +- [AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101) +- [AIRFLOW-3857] spark_submit_hook cannot kill driver pod in Kubernetes (#4678) +- [AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent Tasks' (#5909) +- [AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185) +- [AIRFLOW-5306] Fix the display of links when they contain special characters (#5904) +- [AIRFLOW-3705] Fix PostgresHook get_conn to use conn_name_attr (#5841) +- [AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler shutdown on SIGTERM (#6237) +- [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView (#6307) +- [AIRFLOW-4833] Allow to set Jinja env options in DAG declaration
[GitHub] [airflow] kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r335982966 ## File path: CHANGELOG.txt ## @@ -1,3 +1,134 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Features + +- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546) +- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407) +- [AIRFLOW-4939] Add default_task_retries config (#5570) +- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130) +- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) +- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743) + +Improvements + +- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738) +- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737) +- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736) +- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330) +- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034) +- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194) +- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334) +- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) +- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) +- [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of the aws hook (#6303) +- [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302) +- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304) +- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309) +- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153) +- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) +- [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) +- [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062) +- [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059) +- [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948) +- [AIRFLOW-3888] HA for Hive metastore connection (#4708) +- [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873) +- [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768) +- [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214) +- [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696) +- [AIRFLOW-2891] allow configurable docker_operator container name (#5689) +- [AIRFLOW-4285] Update task dependency context definition and usage (#5079) +- [AIRFLOW-5142] Fixed flaky Cassandra test (#5758) +- [AIRFLOW-5218] Less polling of AWS Batch job status (#5825) +- [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589) +- [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339) +- [AIRFLOW-5560] Allow no confirmation on reset dags in `airflow backfill` command (#6195) +- [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879) +- [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159) +- [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154) +- [AIRFLOW-4835] Refactor operator render_template (#5461) + +Bug Fixes +" +- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator +- [Airflow 4923] Fix Databricks hook leaks API secret in logs (#5635) +- [AIRFLOW-5133] Keep original env state in provide_gcp_credential_file (#5747) +- [AIRFLOW-5497] Update docstring in ``airflow/utils/dag_processing.py`` (#6314) +- Revert/and then rework "[AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)" to improve performance +- [AIRFLOW-5634] Don't allow editing of DagModelView (#6308) +- [AIRFLOW-4309] Remove Broken Dag error after Dag is deleted (#6102) +- [AIRFLOW-5387] Fix "show paused" pagination bug (#6100) +- [AIRFLOW-5489] Remove unneeded assignment of variable (#6106) +- [AIRFLOW-5491] mark_tasks pydoc is incorrect (#6108) +- [AIRFLOW-5492] added missing docstrings (#6107) +- [AIRFLOW-5503] Fix tree view layout on HDPI screen (#6125) +- [AIRFLOW-5481] Allow Deleting Renamed DAGs (#6101) +- [AIRFLOW-3857] spark_submit_hook cannot kill driver pod in Kubernetes (#4678) +- [AIRFLOW-4391] Fix tooltip for None-State Tasks in 'Recent Tasks' (#5909) +- [AIRFLOW-5554] Require statsd 3.3.0 minimum (#6185) +- [AIRFLOW-5306] Fix the display of links when they contain special characters (#5904) +- [AIRFLOW-3705] Fix PostgresHook get_conn to use conn_name_attr (#5841) +- [AIRFLOW-5581] Cleanly shutdown KubernetesJobWatcher for safe Scheduler shutdown on SIGTERM (#6237) +- [AIRFLOW-5634] Don't allow disabled fields to be edited in DagModelView (#6307) +- [AIRFLOW-4833] Allow to set Jinja env options in DAG declaration
[GitHub] [airflow] kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r335982762 ## File path: CHANGELOG.txt ## @@ -1,3 +1,134 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Features + +- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546) +- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407) +- [AIRFLOW-4939] Add default_task_retries config (#5570) +- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130) +- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) +- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743) + +Improvements + +- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738) +- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737) +- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736) +- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330) +- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034) +- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194) +- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334) +- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) +- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) +- [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of the aws hook (#6303) +- [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302) +- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304) +- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309) +- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153) +- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) +- [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) +- [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062) +- [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059) +- [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948) +- [AIRFLOW-3888] HA for Hive metastore connection (#4708) +- [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873) +- [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768) +- [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214) +- [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696) +- [AIRFLOW-2891] allow configurable docker_operator container name (#5689) +- [AIRFLOW-4285] Update task dependency context definition and usage (#5079) +- [AIRFLOW-5142] Fixed flaky Cassandra test (#5758) +- [AIRFLOW-5218] Less polling of AWS Batch job status (#5825) +- [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589) +- [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339) +- [AIRFLOW-5560] Allow no confirmation on reset dags in `airflow backfill` command (#6195) +- [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879) +- [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159) +- [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154) +- [AIRFLOW-4835] Refactor operator render_template (#5461) + +Bug Fixes +" +- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator Review comment: tbh we shouldn't have to do this manually !! So I don't mind if we don't have it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r335982474 ## File path: CHANGELOG.txt ## @@ -1,3 +1,134 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Features + +- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546) +- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407) +- [AIRFLOW-4939] Add default_task_retries config (#5570) +- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130) +- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) +- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743) + +Improvements + +- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738) +- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737) +- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736) +- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330) +- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034) +- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194) +- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334) +- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) +- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) +- [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of the aws hook (#6303) +- [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302) +- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304) +- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309) +- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153) +- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) +- [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) +- [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062) +- [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059) +- [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948) +- [AIRFLOW-3888] HA for Hive metastore connection (#4708) +- [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873) +- [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768) +- [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214) +- [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696) +- [AIRFLOW-2891] allow configurable docker_operator container name (#5689) +- [AIRFLOW-4285] Update task dependency context definition and usage (#5079) +- [AIRFLOW-5142] Fixed flaky Cassandra test (#5758) +- [AIRFLOW-5218] Less polling of AWS Batch job status (#5825) +- [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589) +- [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339) +- [AIRFLOW-5560] Allow no confirmation on reset dags in `airflow backfill` command (#6195) +- [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879) +- [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159) +- [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154) +- [AIRFLOW-4835] Refactor operator render_template (#5461) + +Bug Fixes +" +- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator Review comment: ```suggestion - [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator (#6078) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r335982190 ## File path: CHANGELOG.txt ## @@ -1,3 +1,134 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Features + +- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546) +- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407) +- [AIRFLOW-4939] Add default_task_retries config (#5570) +- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130) +- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) +- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743) + +Improvements + +- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738) +- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737) +- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736) +- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330) +- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034) +- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194) +- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334) +- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) +- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) +- [AIRFLOW-5126] Read ``aws_session_token`` in extra_config of the aws hook (#6303) +- [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302) +- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304) +- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309) +- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153) +- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) +- [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) +- [AIRFLOW-5445] Reduce the required resources for the Kubernetes's sidecar (#6062) +- [AIRFLOW-5443] Use alpine image in Kubernetes's sidecar (#6059) +- [AIRFLOW-5344] Add --proxy-user parameter to SparkSubmitOperator (#5948) +- [AIRFLOW-3888] HA for Hive metastore connection (#4708) +- [AIRFLOW-5269] Reuse session in Scheduler Job from health endpoint (#5873) +- [AIRFLOW-5153] Option to force delete non-empty BQ datasets (#5768) +- [AIRFLOW-4443] Document LatestOnly behavior for external trigger (#5214) +- [AIRFLOW-2891] Make DockerOperator container_name be templateable (#5696) +- [AIRFLOW-2891] allow configurable docker_operator container name (#5689) +- [AIRFLOW-4285] Update task dependency context definition and usage (#5079) +- [AIRFLOW-5142] Fixed flaky Cassandra test (#5758) +- [AIRFLOW-5218] Less polling of AWS Batch job status (#5825) +- [AIRFLOW-4956] Fix LocalTaskJob heartbeat log spamming (#5589) +- [AIRFLOW-3160] Load latest_dagruns asynchronously on home page (#5339) +- [AIRFLOW-5560] Allow no confirmation on reset dags in `airflow backfill` command (#6195) +- [AIRFLOW-5280] conn: Remove aws_default's default region name (#5879) +- [AIRFLOW-5528] end_of_log_mark should not be a log record (#6159) +- [AIRFLOW-5526] Update docs configuration due to migration of GCP docs (#6154) +- [AIRFLOW-4835] Refactor operator render_template (#5461) + +Bug Fixes +" +- [AIRFLOW-5459] Use a dynamic tmp location in Dataflow operator +- [Airflow 4923] Fix Databricks hook leaks API secret in logs (#5635) +- [AIRFLOW-5133] Keep original env state in provide_gcp_credential_file (#5747) +- [AIRFLOW-5497] Update docstring in ``airflow/utils/dag_processing.py`` (#6314) +- Revert/and then rework "[AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)" to improve performance Review comment: Might be worth adding PR Number for this? ```suggestion - Revert/and then rework "[AIRFLOW-4797] Improve performance and behaviour of zombie detection (#5511)" to improve performance (#5908) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-5649) Skipping tests for non relevant changes
[ https://issues.apache.org/jira/browse/AIRFLOW-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953707#comment-16953707 ] ASF GitHub Bot commented on AIRFLOW-5649: - potiuk commented on pull request #6322: [DO NOT MERGE] - Test if py change triggers tests after [AIRFLOW-5649] URL: https://github.com/apache/airflow/pull/6322 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Skipping tests for non relevant changes > --- > > Key: AIRFLOW-5649 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5649 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > > We can skip running tests for some changes that are not relevant: > > 1) Skip pylint checks if there are no .py files changed > 2) Skip running regular tests if there are no .py files changed > 3) Skip running kubernetes tests if there are no changes to > airflow/kubernetes or tests/airflow/kubernetes/ *.py files. > > We still run all test in CRON on master so in case of any false negatives we > will catch them anyway. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] potiuk closed pull request #6322: [DO NOT MERGE] - Test if py change triggers tests after [AIRFLOW-5649]
potiuk closed pull request #6322: [DO NOT MERGE] - Test if py change triggers tests after [AIRFLOW-5649] URL: https://github.com/apache/airflow/pull/6322 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6327: [AIRFLOW-4675] Make airflow/lineage Pylint compatible
feluelle commented on a change in pull request #6327: [AIRFLOW-4675] Make airflow/lineage Pylint compatible URL: https://github.com/apache/airflow/pull/6327#discussion_r335976709 ## File path: airflow/lineage/backend/__init__.py ## @@ -16,13 +16,12 @@ # specific language governing permissions and limitations # under the License. # -"""Sends lineage metadata to a backend""" +"""Send lineage metadata to a backend""" class LineageBackend: -"""Sends lineage metadata to a backend""" -def send_lineage(self, - operator=None, inlets=None, outlets=None, context=None): +"""Sends lineage metadata to a backend.""" +def send_lineage(self, operator, inlets, outlets, context): Review comment: Sorry @enricapq what Ash said is probably the right change not what I suggested first. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6327: [AIRFLOW-4675] Make airflow/lineage Pylint compatible
feluelle commented on a change in pull request #6327: [AIRFLOW-4675] Make airflow/lineage Pylint compatible URL: https://github.com/apache/airflow/pull/6327#discussion_r335973293 ## File path: airflow/lineage/backend/__init__.py ## @@ -16,13 +16,12 @@ # specific language governing permissions and limitations # under the License. # -"""Sends lineage metadata to a backend""" +"""Send lineage metadata to a backend""" class LineageBackend: -"""Sends lineage metadata to a backend""" -def send_lineage(self, - operator=None, inlets=None, outlets=None, context=None): +"""Sends lineage metadata to a backend.""" +def send_lineage(self, operator, inlets, outlets, context): Review comment: > If so the change should be to the subclass to make them optional, not removing the optional here. True - that would also not break backwards compatibility. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] feluelle commented on a change in pull request #6327: [AIRFLOW-4675] Make airflow/lineage Pylint compatible
feluelle commented on a change in pull request #6327: [AIRFLOW-4675] Make airflow/lineage Pylint compatible URL: https://github.com/apache/airflow/pull/6327#discussion_r335973293 ## File path: airflow/lineage/backend/__init__.py ## @@ -16,13 +16,12 @@ # specific language governing permissions and limitations # under the License. # -"""Sends lineage metadata to a backend""" +"""Send lineage metadata to a backend""" class LineageBackend: -"""Sends lineage metadata to a backend""" -def send_lineage(self, - operator=None, inlets=None, outlets=None, context=None): +"""Sends lineage metadata to a backend.""" +def send_lineage(self, operator, inlets, outlets, context): Review comment: > I imagine Pylint was complaining because the subclass had a different signature? Yes. > If so the change should be to the subclass to make them optional, not removing the optional here. True - that would also not break backwards compatibility. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
ashb commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r335969885 ## File path: CHANGELOG.txt ## @@ -1,3 +1,134 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Feature +""" +- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546) +- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407) +- [AIRFLOW-4939] Add default_task_retries config (#5570) +- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130) +- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) +- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743) + +Improvement Review comment: Need to update my automated generator :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r335969129 ## File path: CHANGELOG.txt ## @@ -1,3 +1,134 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Feature Review comment: ```suggestion New Features ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
kaxil commented on a change in pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356#discussion_r335967145 ## File path: CHANGELOG.txt ## @@ -1,3 +1,134 @@ +Airflow 1.10.6, 2019-10-21 +-- + +New Feature +""" +- [AIRFLOW-4908] Implement BigQuery Hooks/Operators for update_dataset, patch_dataset and get_dataset (#5546) +- [AIRFLOW-4741] Optionally report task errors to Sentry (#5407) +- [AIRFLOW-4939] Add default_task_retries config (#5570) +- [AIRFLOW-5508] Add config setting to limit which StatsD metrics are emitted (#6130) +- [AIRFLOW-4222] Add cli autocomplete for bash & zsh (#5789) +- [AIRFLOW-3871] Operators template fields can now render fields inside objects (#4743) + +Improvement +""" +- [AIRFLOW-5127] Gzip support for CassandraToGoogleCloudStorageOperator (#5738) +- [AIRFLOW-5125] Add gzip support for AdlsToGoogleCloudStorageOperator (#5737) +- [AIRFLOW-5124] Add gzip support for S3ToGoogleCloudStorageOperator (#5736) +- [AIRFLOW-5653] Log AirflowSkipException in task instance log to make it clearer why tasks might be skipped (#6330) +- [AIRFLOW-5343] Remove legacy SQLAlchmey pessimistic pool disconnect handling (#6034) +- [AIRFLOW-5561] Relax httplib2 version required for gcp extra (#6194) +- [AIRFLOW-5657] Update the upper bound for dill dependency (#6334) +- [AIRFLOW-5292] Allow ECSOperator to tag tasks (#5891) +- [AIRFLOW-4939] Simplify Code for Default Task Retries (#6233) +- [AIRFLOW-5126] Read aws_session_token in extra_config of the aws hook (#6303) +- [AIRFLOW-5636] Allow adding or overriding existing Operator Links (#6302) +- [AIRFLOW-4965] Handle quote exceptions in GCP AI operators (v1.10) (#6304) +- [AIRFLOW-3783] Speed up Redshift to S3 UNload with HEADERs (#6309) +- [AIRFLOW-3388] Add support to Array Jobs for AWS Batch Operator (#6153) +- [AIRFLOW-4574] add option to provide private_key in SSHHook (#6104) +- [AIRFLOW-5530] Fiix typo in AWS SQS sensors (#6012) Review comment: Maybe this is just a typo in PR, but probably we can update it in the Changelog ```suggestion - [AIRFLOW-5530] Fix typo in AWS SQS sensors (#6012) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb opened a new pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1)
ashb opened a new pull request #6356: [AIRFLOW-XXX] Changelog for 1.10.6 (rc1) URL: https://github.com/apache/airflow/pull/6356 I have picked a date that is when I plan to end the vote and release it if the vote passes. If it doesn't then we will need to update the Changelog again with whatever fixes we need anyway. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #6327: [AIRFLOW-4675] Make airflow/lineage Pylint compatible
ashb commented on a change in pull request #6327: [AIRFLOW-4675] Make airflow/lineage Pylint compatible URL: https://github.com/apache/airflow/pull/6327#discussion_r335964889 ## File path: airflow/lineage/datasets.py ## @@ -104,13 +118,15 @@ def map_type(name): class DataBase(DataSet): +"""DataBase class, subclass of DataSet.""" Review comment: This comment is almost worse than no comment at all. It doesn't tell me anything more about the class. If we aren't going to have a descriptibe doccomment that adds more info then I'd rather we just disabled the warning for these class This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
