[GitHub] [airflow] dstandish commented on a change in pull request #14521: Add Asana Provider

[GitBox]

dstandish commented on a change in pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#discussion_r584487356



##
File path: docs/apache-airflow-providers-asana/operators/asana.rst
##
@@ -0,0 +1,87 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+
+
+
+.. _howto/operator:AsanaCreateTaskOperator:
+
+AsanaCreateTaskOperator
+===
+
+Use the :class:`~airflow.providers.asana.operators.AsanaCreateTaskOperator` to
+create an Asana task.
+
+
+Using the Operator
+^^
+
+The AsanaCreateTaskOperator minimally requires the new task's name and
+the Asana connection to use to connect to your account (``asana_conn_id``). 
There are many other
+`task attributes you can specify 
`_
+through the ``optional_task_parameters``. You must specify at least one of 
'workspace',
+'parent', or 'projects' in the ``optional_task_parameters``.
+
+
+.. _howto/operator:AsanaDeleteTaskOperator:
+
+AsanaDeleteTaskOperator
+===
+
+Use the :class:`~airflow.providers.asana.operators.AsanaDeleteTaskOperator` to
+update an existing Asana task.
+
+
+Using the Operator
+^^
+
+The AsanaDeleteTaskOperator requires a task id to delete. Use the 
``asana_conn_id``
+parameter to specify the Asana connection to use to connect to your account.
+
+
+.. _howto/operator:AsanaFindTaskOperator:
+
+AsanaFindTaskOperator
+===
+
+Use the :class:`~airflow.providers.asana.operators.AsanaFindTaskOperator` to
+search for Asana tasks that fit some criteria.
+
+
+Using the Operator
+^^
+
+The AsanaFindTaskOperator requires a dict of search parameters following the 
description
+`here 
`_.
+Use the ``asana_conn_id`` parameter to specify the Asana connection to use to 
connect
+to your account.
+
+.. _howto/operator:AsanaUpdateTaskOperator:
+
+AsanaUpdateTaskOperator
+===
+
+Use the :class:`~airflow.providers.asana.operators.AsanaUpdateTaskOperator` to
+update an existing Asana task.
+
+
+Using the Operator
+^^
+
+The AsanaUpdateTaskOperator minimally requires the task id to update and
+the Asana connection to use to connect to your account (``asana_conn_id``). 
There are many other
+`task attributes you can overwrite 
`_
+through the ``optional_task_parameters``.

Review comment:
   naming

##
File path: docs/apache-airflow-providers-asana/operators/asana.rst
##
@@ -0,0 +1,87 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+
+
+
+.. _howto/operator:AsanaCreateTaskOperator:
+
+AsanaCreateTaskOperator
+===
+
+Use the :class:`~airflow.providers.asana.operators.AsanaCreateTaskOperator` to
+create an Asana task.
+
+
+Using the Operator
+^^
+
+The AsanaCreateTaskOperator minimally requires the new task's name and
+the Asana connection to use to connect to your account (``asana_conn_id``). 
There are many other
+`task attributes you can specify 
`_
+through the ``optional_task_parameters``. You must specify at least one of 
'workspace',
+'parent', or 'projects' in the ``optional_task_parameters``.

Review comment:
   here too




---

[GitHub] [airflow] dstandish commented on a change in pull request #14521: Add Asana Provider

[GitBox]

dstandish commented on a change in pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#discussion_r584487125



##
File path: docs/apache-airflow-providers-asana/operators/asana.rst
##
@@ -0,0 +1,87 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+
+
+
+.. _howto/operator:AsanaCreateTaskOperator:
+
+AsanaCreateTaskOperator
+===
+
+Use the :class:`~airflow.providers.asana.operators.AsanaCreateTaskOperator` to
+create an Asana task.
+
+
+Using the Operator
+^^
+
+The AsanaCreateTaskOperator minimally requires the new task's name and
+the Asana connection to use to connect to your account (``asana_conn_id``). 
There are many other
+`task attributes you can specify 
`_
+through the ``optional_task_parameters``. You must specify at least one of 
'workspace',

Review comment:
   needs to be updated to reflect param name change





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dstandish commented on a change in pull request #14521: Add Asana Provider

[GitBox]

dstandish commented on a change in pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#discussion_r584484280



##
File path: airflow/providers/asana/hooks/asana.py
##
@@ -0,0 +1,72 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Connect to Asana."""
+
+from asana import Client
+from cached_property import cached_property
+
+from airflow.hooks.base import BaseHook
+
+
+class AsanaHook(BaseHook):
+"""Wrapper around Asana Python client library."""
+
+conn_name_attr = "asana_conn_id"
+default_conn_name = "asana_default"
+conn_type = "asana"
+hook_name = "Asana"
+
+def __init__(self, conn_id: str = default_conn_name, *args, **kwargs) -> 
None:
+super().__init__(*args, **kwargs)
+self.asana_conn_id = conn_id
+self.uri = None

Review comment:
   looks like this does not do anything?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on a change in pull request #14490: Bugfix: Fix wrong output of tags and owners in dag detail API endpoint

[GitBox]

ephraimbuddy commented on a change in pull request #14490:
URL: https://github.com/apache/airflow/pull/14490#discussion_r584478774



##
File path: tests/api_connexion/schemas/test_dag_schema.py
##
@@ -122,11 +123,11 @@ def test_serialize(self):
 'is_paused': None,
 'is_subdag': False,
 'orientation': 'LR',
-'owners': [],
+'owner': '',
 'params': {'foo': 1},
 'schedule_interval': {'__type': 'TimeDelta', 'days': 1, 'seconds': 
0, 'microseconds': 0},
 'start_date': '2020-06-19T00:00:00+00:00',
-'tags': None,
+'tags': ["example1", "example2"],

Review comment:
   Previously, the endpoint returns tags object and that made it to output 
empty objects {} because in the DAG object, tags is a list of strings unlike 
DagModel where tags is an object.
   
   I think, since this endpoint only outputs data and doesn't add, we will not 
worry about adding anything else. I have removed the connection between the 
schema with DagSchema where it inherited DagSchema which defined tags. 
   
   Now the schema is standing alone and not connected with other schemas, and 
only for the Dagdetails endpoint which only outputs data.
   
   What do you think?
   





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jmelot commented on a change in pull request #14521: Add Asana Provider

[GitBox]

jmelot commented on a change in pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#discussion_r584476559



##
File path: airflow/providers/asana/operators/asana_tasks.py
##
@@ -0,0 +1,198 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Dict, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.asana.hooks.asana import AsanaHook
+from airflow.utils.decorators import apply_defaults
+
+
+class AsanaCreateTaskOperator(BaseOperator):
+"""
+This operator can be used to create Asana tasks. For more information on
+Asana optional task parameters, see 
https://developers.asana.com/docs/create-a-task
+
+.. seealso::
+For more information on how to use this operator, take a look at the 
guide:
+:ref:`howto/operator:AsanaCreateTaskOperator`
+
+:param asana_conn_id: The Asana connection to use.
+:type asana_conn_id: str
+:param name: Name of the task.
+:type name: str
+:param optional_task_parameters: Any of the optional task creation 
parameters.
+See https://developers.asana.com/docs/create-a-task for a complete 
list.
+You must specify at least one of 'workspace', 'parent', or 'projects'.
+:type optional_task_parameters: dict
+"""
+
+@apply_defaults
+def __init__(
+self,
+*,
+asana_conn_id: str,
+name: str,
+optional_task_parameters: Optional[dict] = None,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+self.asana_conn_id = asana_conn_id
+self.name = name
+self.optional_task_parameters = optional_task_parameters
+self.hook = AsanaHook(conn_id=self.asana_conn_id)

Review comment:
   Done, this makes sense, I've moved it to `execute`





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jmelot commented on a change in pull request #14521: Add Asana Provider

[GitBox]

jmelot commented on a change in pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#discussion_r584475888



##
File path: airflow/providers/asana/operators/asana_tasks.py
##
@@ -0,0 +1,195 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Dict, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.asana.hooks.asana import AsanaHook
+from airflow.utils.decorators import apply_defaults
+
+
+class AsanaCreateTaskOperator(BaseOperator):
+"""
+This operator can be used to create Asana tasks. For more information on
+Asana optional task parameters, see 
https://developers.asana.com/docs/create-a-task
+
+.. seealso::
+For more information on how to use this operator, take a look at the 
guide:
+:ref:`howto/operator:AsanaCreateTaskOperator`
+
+:param asana_conn_id: The Asana connection to use.
+:type asana_conn_id: str
+:param name: Name of the task.
+:type name: str
+:param optional_task_parameters: Any of the optional task creation 
parameters.
+See https://developers.asana.com/docs/create-a-task for a complete 
list.
+You must specify at least one of 'workspace', 'parent', or 'projects'.
+:type optional_task_parameters: dict
+"""
+
+@apply_defaults
+def __init__(
+self,
+*,
+asana_conn_id: str,
+name: str,
+optional_task_parameters: Optional[dict] = None,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+self.asana_conn_id = asana_conn_id
+self.name = name
+self.optional_task_parameters = optional_task_parameters
+self.hook = AsanaHook(conn_id=self.asana_conn_id)
+
+def execute(self, context: Dict) -> str:
+asana_client = self.hook.get_conn()
+
+params = {"name": self.name}
+if self.optional_task_parameters is not None:
+params.update(self.optional_task_parameters)
+response = asana_client.tasks.create(params=params)

Review comment:
   Makes sense, done!





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on pull request #14381: BugFix: Fix taskInstance API call fails if a task is removed from running DAG

[GitBox]

ephraimbuddy commented on pull request #14381:
URL: https://github.com/apache/airflow/pull/14381#issuecomment-787689694


   > Yeah, not sure what happened there. Likely a timeout or some other glitch.
   
   Have pushed severally and MySQL 8 still failing. I can't pin what it is.  🙂



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on pull request #14493: Bugfix: Fix False alarm in import changes rule

[GitBox]

ephraimbuddy commented on pull request #14493:
URL: https://github.com/apache/airflow/pull/14493#issuecomment-787688153


   @kaxil, Any idea why this is failing on CI but don't fail locally? Also the 
failing parts are not the ones I worked on in this PR



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] 2011211242 opened a new issue #14538: Task can not exit normally when I use map in billiard

[GitBox]

2011211242 opened a new issue #14538:
URL: https://github.com/apache/airflow/issues/14538


   When error occured in my subprocess, sometime task cannot terminnal 
normally. The task status on the web was running.
   
   
   
   
   **Apache Airflow version**:
   
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
   
   **Environment**:
   
   - **Cloud provider or hardware configuration**:
   - **OS** (e.g. from /etc/os-release):
   - **Kernel** (e.g. `uname -a`):
   - **Install tools**:
   - **Others**:
   
   **What happened**:
   
   
   
   **What you expected to happen**:
   
   
   
   **How to reproduce it**:
   
   
   
   **Anything else we need to know**:
   
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #14538: Task can not exit normally when I use map in billiard

[GitBox]

boring-cyborg[bot] commented on issue #14538:
URL: https://github.com/apache/airflow/issues/14538#issuecomment-787686774


   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #14521: Add Asana Provider

[GitBox]

github-actions[bot] commented on pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#issuecomment-787679428


   [The Workflow run](https://github.com/apache/airflow/actions/runs/609557577) 
is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static 
checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider 
packages,^Checks: Helm tests$,^Test OpenAPI*.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #14537: Add more tips about health checks

[GitBox]

mik-laj commented on a change in pull request #14537:
URL: https://github.com/apache/airflow/pull/14537#discussion_r584468771



##
File path: docs/apache-airflow/production-deployment.rst
##
@@ -111,11 +111,7 @@ Airflow users occasionally report instances of the 
scheduler hanging without a t
 * `Scheduler gets stuck without a trace 
`_
 * `Scheduler stopping frequently 
`_
 
-Strategies for mitigation:
-
-* When running on kubernetes, use a ``livenessProbe`` on the scheduler 
deployment to fail if the scheduler
-  has not heartbeat in a while.
-  `Example: 
`_.
+To mitigate these issues, make sure you have a :doc:`health check 
` set up that will detect when your 
scheduler has not heartbeat in a while.

Review comment:
   ```suggestion
   To mitigate these issues, make sure you have a :doc:`health check 
` set up that will detect when your scheduler 
has not heartbeat in a while.
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #14537: Add more tips about health checks

[GitBox]

mik-laj commented on a change in pull request #14537:
URL: https://github.com/apache/airflow/pull/14537#discussion_r584468647



##
File path: docs/apache-airflow/logging-monitoring/check-health.rst
##
@@ -20,7 +20,13 @@
 Checking Airflow Health Status
 ==
 
-Airflow has s two methods to check the health of components - HTTP checks and 
CLI checks. Their choice depends on the role of the component as well as what 
tools it uses to monitor the deployment.
+Airflow has two methods to check the health of components - HTTP checks and 
CLI checks. Their choice depends on the role of the component as well as what 
tools it uses to monitor the deployment.
+
+For example, when running on Kubernetes, use `a Liveness probes 
`__
 (``livenessProbe`` property) with :ref:`CLI checks 
`__ on the scheduler deployment to 
restart it when it fail. For the webserver, you can configure the readiness 
probe (``readinessProbe`` property) using :ref:`check-health/http-endpoint`.
+
+For an example for a Docker Compose environment, see the 
``docker-compose.yaml`` file available in the :doc:`/start/docker.rst`.

Review comment:
   ```suggestion
   For an example for a Docker Compose environment, see the 
``docker-compose.yaml`` file available in the :doc:`/start/docker`.
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #14537: Add more tips about health checks

[GitBox]

mik-laj commented on a change in pull request #14537:
URL: https://github.com/apache/airflow/pull/14537#discussion_r584468418



##
File path: docs/apache-airflow/logging-monitoring/check-health.rst
##
@@ -20,7 +20,13 @@
 Checking Airflow Health Status
 ==
 
-Airflow has s two methods to check the health of components - HTTP checks and 
CLI checks. Their choice depends on the role of the component as well as what 
tools it uses to monitor the deployment.
+Airflow has two methods to check the health of components - HTTP checks and 
CLI checks. Their choice depends on the role of the component as well as what 
tools it uses to monitor the deployment.
+
+For example, when running on Kubernetes, use `a Liveness probes 
`__
 (``livenessProbe`` property) with :ref:`CLI checks 
`__ on the scheduler deployment to 
restart it when it fail. For the webserver, you can configure the readiness 
probe (``readinessProbe`` property) using :ref:`check-health/http-endpoint`.

Review comment:
   ```suggestion
   For example, when running on Kubernetes, use `a Liveness probes 
`__
 (``livenessProbe`` property) with :ref:`CLI checks 
` on the scheduler deployment to restart 
it when it fail. For the webserver, you can configure the readiness probe 
(``readinessProbe`` property) using :ref:`check-health/http-endpoint`.
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mrkm4ntr commented on pull request #12637: Remove unsupported arguments

[GitBox]

mrkm4ntr commented on pull request #12637:
URL: https://github.com/apache/airflow/pull/12637#issuecomment-787670978


   I think this is still valid. Fixed tests.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jmelot commented on a change in pull request #14521: Add Asana Provider

[GitBox]

jmelot commented on a change in pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#discussion_r584459468



##
File path: airflow/providers/asana/hooks/asana.py
##
@@ -0,0 +1,60 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Connect to Asana."""
+
+from asana import Client
+
+from airflow.hooks.base import BaseHook
+
+
+class AsanaHook(BaseHook):
+"""
+Wrapper around Asana Python client library.
+"""
+
+conn_name_attr = "asana_conn_id"
+default_conn_name = "asana_default"
+conn_type = "asana"
+hook_name = "Asana"
+
+def __init__(self, conn_id: str = default_conn_name, *args, **kwargs) -> 
None:
+super().__init__(*args, **kwargs)
+self.asana_conn_id = conn_id
+self.connection = kwargs.pop("connection", None)
+self.client = None
+self.extras = None
+self.uri = None
+
+def get_conn(self) -> Client:
+"""
+Creates Asana Client
+"""
+self.connection = self.get_connection(self.asana_conn_id)
+self.extras = self.connection.extra_dejson.copy()
+
+if self.client is not None:

Review comment:
   Great idea. I've made a commit with several changes in response to your 
comments, including this one, will resolve those now and address the rest soon. 
Thanks again!





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jmelot commented on a change in pull request #14521: Add Asana Provider

[GitBox]

jmelot commented on a change in pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#discussion_r584454603



##
File path: airflow/providers/asana/operators/asana_tasks.py
##
@@ -0,0 +1,198 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Dict, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.asana.hooks.asana import AsanaHook
+from airflow.utils.decorators import apply_defaults
+
+
+class AsanaCreateTaskOperator(BaseOperator):
+"""
+This operator can be used to create Asana tasks. For more information on
+Asana optional task parameters, see 
https://developers.asana.com/docs/create-a-task
+
+.. seealso::
+For more information on how to use this operator, take a look at the 
guide:
+:ref:`howto/operator:AsanaCreateTaskOperator`
+
+:param asana_conn_id: The Asana connection to use.
+:type asana_conn_id: str
+:param name: Name of the task.
+:type name: str
+:param optional_task_parameters: Any of the optional task creation 
parameters.
+See https://developers.asana.com/docs/create-a-task for a complete 
list.
+You must specify at least one of 'workspace', 'parent', or 'projects'.
+:type optional_task_parameters: dict
+"""
+
+@apply_defaults
+def __init__(
+self,
+*,
+asana_conn_id: str,
+name: str,
+optional_task_parameters: Optional[dict] = None,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+self.asana_conn_id = asana_conn_id
+self.name = name
+self.optional_task_parameters = optional_task_parameters
+self.hook = AsanaHook(conn_id=self.asana_conn_id)
+
+def execute(self, context: Dict) -> str:
+asana_client = self.hook.get_conn()
+
+params = {"name": self.name}
+if self.optional_task_parameters is not None:
+params.update(self.optional_task_parameters)
+response = asana_client.tasks.create(params=params)
+
+self.log.info(response)
+return response["gid"]
+
+
+class AsanaUpdateTaskOperator(BaseOperator):
+"""
+This operator can be used to update Asana tasks.
+For more information on Asana optional task parameters, see
+https://developers.asana.com/docs/update-a-task
+
+.. seealso::
+For more information on how to use this operator, take a look at the 
guide:
+:ref:`howto/operator:AsanaUpdateTaskOperator`
+
+:param asana_conn_id: The Asana connection to use.
+:type asana_conn_id: str
+:param asana_task_gid: Asana task ID to update
+:type asana_task_gid: str
+:param task_update_parameters: Any task parameters that should be updated.
+See https://developers.asana.com/docs/update-a-task for a complete 
list.
+:type task_update_parameters: dict
+"""
+
+@apply_defaults
+def __init__(
+self,
+*,
+asana_conn_id: str,
+asana_task_gid: str,
+task_update_parameters: dict,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+
+self.asana_conn_id = asana_conn_id
+self.asana_task_gid = asana_task_gid
+self.task_update_parameters = task_update_parameters
+self.hook = AsanaHook(conn_id=self.asana_conn_id)
+
+def execute(self, context: Dict) -> None:
+asana_client = self.hook.get_conn()
+
+response = asana_client.tasks.update(task=self.asana_task_gid, 
params=self.task_update_parameters)
+self.log.info(response)
+
+
+class AsanaDeleteTaskOperator(BaseOperator):
+"""
+This operator can be used to delete Asana tasks.
+
+.. seealso::
+For more information on how to use this operator, take a look at the 
guide:
+:ref:`howto/operator:AsanaDeleteTaskOperator`
+
+:param asana_conn_id: The Asana connection to use.
+:type asana_conn_id: str
+:param asana_task_gid: Asana Task ID to delete.
+:type asana_task_gid: str
+"""
+
+@apply_defaults
+def __init__(
+self,
+*,
+asana_conn_id: str,
+asana_task_gid: str,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+
+

[GitHub] [airflow] dstandish commented on a change in pull request #14521: Add Asana Provider

[GitBox]

dstandish commented on a change in pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#discussion_r58963



##
File path: airflow/providers/asana/operators/asana_tasks.py
##
@@ -0,0 +1,198 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Dict, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.asana.hooks.asana import AsanaHook
+from airflow.utils.decorators import apply_defaults
+
+
+class AsanaCreateTaskOperator(BaseOperator):
+"""
+This operator can be used to create Asana tasks. For more information on
+Asana optional task parameters, see 
https://developers.asana.com/docs/create-a-task
+
+.. seealso::
+For more information on how to use this operator, take a look at the 
guide:
+:ref:`howto/operator:AsanaCreateTaskOperator`
+
+:param asana_conn_id: The Asana connection to use.
+:type asana_conn_id: str
+:param name: Name of the task.
+:type name: str
+:param optional_task_parameters: Any of the optional task creation 
parameters.
+See https://developers.asana.com/docs/create-a-task for a complete 
list.
+You must specify at least one of 'workspace', 'parent', or 'projects'.
+:type optional_task_parameters: dict
+"""
+
+@apply_defaults
+def __init__(
+self,
+*,
+asana_conn_id: str,
+name: str,
+optional_task_parameters: Optional[dict] = None,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+self.asana_conn_id = asana_conn_id
+self.name = name
+self.optional_task_parameters = optional_task_parameters
+self.hook = AsanaHook(conn_id=self.asana_conn_id)

Review comment:
   ok ... 
   
   after reviewing your hook i see you do not call get_connection until 
get_conn, so in this case what you're doing here is perfectly safe
   
   however, i still recommend you do not initialize hook in operator init 
because this is something we all watch out for so you should just avoid it 
anyway.  it is assumed in general that hooks may call get_connection in hook 
init.
   





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Rishabhraghwendra18 commented on issue #14054: SambaHook using old unmaintained library

[GitBox]

Rishabhraghwendra18 commented on issue #14054:
URL: https://github.com/apache/airflow/issues/14054#issuecomment-787647380


   ya..sure



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dstandish commented on a change in pull request #14521: Add Asana Provider

[GitBox]

dstandish commented on a change in pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#discussion_r58963



##
File path: airflow/providers/asana/operators/asana_tasks.py
##
@@ -0,0 +1,198 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Dict, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.asana.hooks.asana import AsanaHook
+from airflow.utils.decorators import apply_defaults
+
+
+class AsanaCreateTaskOperator(BaseOperator):
+"""
+This operator can be used to create Asana tasks. For more information on
+Asana optional task parameters, see 
https://developers.asana.com/docs/create-a-task
+
+.. seealso::
+For more information on how to use this operator, take a look at the 
guide:
+:ref:`howto/operator:AsanaCreateTaskOperator`
+
+:param asana_conn_id: The Asana connection to use.
+:type asana_conn_id: str
+:param name: Name of the task.
+:type name: str
+:param optional_task_parameters: Any of the optional task creation 
parameters.
+See https://developers.asana.com/docs/create-a-task for a complete 
list.
+You must specify at least one of 'workspace', 'parent', or 'projects'.
+:type optional_task_parameters: dict
+"""
+
+@apply_defaults
+def __init__(
+self,
+*,
+asana_conn_id: str,
+name: str,
+optional_task_parameters: Optional[dict] = None,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+self.asana_conn_id = asana_conn_id
+self.name = name
+self.optional_task_parameters = optional_task_parameters
+self.hook = AsanaHook(conn_id=self.asana_conn_id)

Review comment:
   ok ... 
   
   after reviewing your hook i see you do not call get_connection until 
get_conn, so in this case what you're doing here perfectly safe
   
   however, i still recommend you do not initialize hook in operator init 
because this is something we all watch out for so you should just avoid it 
anyway.  it is assumed in general that hooks may call get_connection in hook 
init.
   





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dstandish commented on a change in pull request #14521: Add Asana Provider

[GitBox]

dstandish commented on a change in pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#discussion_r58963



##
File path: airflow/providers/asana/operators/asana_tasks.py
##
@@ -0,0 +1,198 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Dict, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.asana.hooks.asana import AsanaHook
+from airflow.utils.decorators import apply_defaults
+
+
+class AsanaCreateTaskOperator(BaseOperator):
+"""
+This operator can be used to create Asana tasks. For more information on
+Asana optional task parameters, see 
https://developers.asana.com/docs/create-a-task
+
+.. seealso::
+For more information on how to use this operator, take a look at the 
guide:
+:ref:`howto/operator:AsanaCreateTaskOperator`
+
+:param asana_conn_id: The Asana connection to use.
+:type asana_conn_id: str
+:param name: Name of the task.
+:type name: str
+:param optional_task_parameters: Any of the optional task creation 
parameters.
+See https://developers.asana.com/docs/create-a-task for a complete 
list.
+You must specify at least one of 'workspace', 'parent', or 'projects'.
+:type optional_task_parameters: dict
+"""
+
+@apply_defaults
+def __init__(
+self,
+*,
+asana_conn_id: str,
+name: str,
+optional_task_parameters: Optional[dict] = None,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+self.asana_conn_id = asana_conn_id
+self.name = name
+self.optional_task_parameters = optional_task_parameters
+self.hook = AsanaHook(conn_id=self.asana_conn_id)

Review comment:
   ok ... 
   
   after reviewing your hook i see you do not call get_connection until 
get_conn, so in this case what you're doing here **perfectly safe**
   
   **however**, i still recommend you do not initialize hook in operator init 
because this is something we all watch out for so you should just avoid it 
anyway.  it is assumed in general that hooks may call get_connection in hook 
init.
   





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dstandish commented on a change in pull request #14521: Add Asana Provider

[GitBox]

dstandish commented on a change in pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#discussion_r584443649



##
File path: airflow/providers/asana/hooks/asana.py
##
@@ -0,0 +1,58 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Connect to Asana."""
+
+from asana import Client
+
+from airflow.hooks.base import BaseHook
+
+
+class AsanaHook(BaseHook):
+"""Wrapper around Asana Python client library."""
+
+conn_name_attr = "asana_conn_id"
+default_conn_name = "asana_default"
+conn_type = "asana"
+hook_name = "Asana"
+
+def __init__(self, conn_id: str = default_conn_name, *args, **kwargs) -> 
None:
+super().__init__(*args, **kwargs)
+self.asana_conn_id = conn_id
+self.connection = kwargs.pop("connection", None)
+self.client = None
+self.extras = None
+self.uri = None
+
+def get_conn(self) -> Client:
+"""Creates Asana Client"""
+self.connection = self.get_connection(self.asana_conn_id)

Review comment:
   if connection was supplied at init, this will overwrite it

##
File path: airflow/providers/asana/hooks/asana.py
##
@@ -0,0 +1,58 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Connect to Asana."""
+
+from asana import Client
+
+from airflow.hooks.base import BaseHook
+
+
+class AsanaHook(BaseHook):
+"""Wrapper around Asana Python client library."""
+
+conn_name_attr = "asana_conn_id"
+default_conn_name = "asana_default"
+conn_type = "asana"
+hook_name = "Asana"
+
+def __init__(self, conn_id: str = default_conn_name, *args, **kwargs) -> 
None:
+super().__init__(*args, **kwargs)
+self.asana_conn_id = conn_id
+self.connection = kwargs.pop("connection", None)

Review comment:
   why not make connection a param if you intend for it to be supplied 
optionally and

##
File path: airflow/providers/asana/operators/asana_tasks.py
##
@@ -0,0 +1,198 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Dict, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.asana.hooks.asana import AsanaHook
+from airflow.utils.decorators import apply_defaults
+
+
+class AsanaCreateTaskOperator(BaseOperator):
+"""
+This operator can be used to create Asana tasks. For more information on
+Asana optional task parameters, see 
https://developers.asana.com/docs/create-a-task
+
+.. seealso::
+For more information on how to use this operator, take a look at the 
guide:
+:ref:`howto/operator:AsanaCreateTaskOperator`
+
+:param asana_conn_id: The Asana connection to use.
+:type asana_conn_id: str
+ 

[GitHub] [airflow] mik-laj opened a new pull request #14537: Health checks tips

[GitBox]

mik-laj opened a new pull request #14537:
URL: https://github.com/apache/airflow/pull/14537


   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dstandish commented on a change in pull request #14521: Add Asana Provider

[GitBox]

dstandish commented on a change in pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#discussion_r584443236



##
File path: airflow/providers/asana/hooks/asana.py
##
@@ -0,0 +1,58 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Connect to Asana."""
+
+from asana import Client
+
+from airflow.hooks.base import BaseHook
+
+
+class AsanaHook(BaseHook):
+"""Wrapper around Asana Python client library."""
+
+conn_name_attr = "asana_conn_id"
+default_conn_name = "asana_default"
+conn_type = "asana"
+hook_name = "Asana"
+
+def __init__(self, conn_id: str = default_conn_name, *args, **kwargs) -> 
None:
+super().__init__(*args, **kwargs)
+self.asana_conn_id = conn_id
+self.connection = kwargs.pop("connection", None)
+self.client = None
+self.extras = None
+self.uri = None
+
+def get_conn(self) -> Client:
+"""Creates Asana Client"""
+self.connection = self.get_connection(self.asana_conn_id)
+self.extras = self.connection.extra_dejson.copy()
+
+if self.client is not None:
+return self.client
+
+if (self.connection.password is None) or 
(len(self.connection.password.strip()) == 0):

Review comment:
   ```suggestion
   if not self.connection.password:
   ```
   suggesting the simpler approach here for readability ...





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #14105: Add Google leveldb hook (#13109)

[GitBox]

github-actions[bot] commented on pull request #14105:
URL: https://github.com/apache/airflow/pull/14105#issuecomment-787631897


   [The Workflow run](https://github.com/apache/airflow/actions/runs/609350045) 
is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static 
checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider 
packages,^Checks: Helm tests$,^Test OpenAPI*.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mrkm4ntr closed pull request #12637: Remove unsupported arguments

[GitBox]

mrkm4ntr closed pull request #12637:
URL: https://github.com/apache/airflow/pull/12637


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mrkm4ntr opened a new pull request #12637: Remove unsupported arguments

[GitBox]

mrkm4ntr opened a new pull request #12637:
URL: https://github.com/apache/airflow/pull/12637


   We cannot override channel, username, and icon now. Plseas refer  
https://api.slack.com/messaging/webhooks#advanced_message_formatting
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #14490: Bugfix: Fix wrong output of tags and owners in dag detail API endpoint

[GitBox]

mik-laj commented on a change in pull request #14490:
URL: https://github.com/apache/airflow/pull/14490#discussion_r584433963



##
File path: tests/api_connexion/schemas/test_dag_schema.py
##
@@ -122,11 +123,11 @@ def test_serialize(self):
 'is_paused': None,
 'is_subdag': False,
 'orientation': 'LR',
-'owners': [],
+'owner': '',
 'params': {'foo': 1},
 'schedule_interval': {'__type': 'TimeDelta', 'days': 1, 'seconds': 
0, 'microseconds': 0},
 'start_date': '2020-06-19T00:00:00+00:00',
-'tags': None,
+'tags': ["example1", "example2"],

Review comment:
   We should return objects to make it more future-proof and extensible. 
for example, if we want to add a color, we can do so without breaking changes.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #14489: Add a retry with wait interval for SSH operator

[GitBox]

mik-laj commented on issue #14489:
URL: https://github.com/apache/airflow/issues/14489#issuecomment-787618674


   SGTM. GoogleCompute retry authorizations, so generic 
   hook can also do it. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj opened a new pull request #14536: Disable health checks for ad-hoc containers

[GitBox]

mik-laj opened a new pull request #14536:
URL: https://github.com/apache/airflow/pull/14536


   This slows things down a bit if we're trying to write scripts that can 
sometimes run multiple containers. The health of the containers is already 
checked by Docker, so we don't have to check it on our side every time.
   
   For legacy health checks (modern health checks are a bit slower):
   ```
   $ time docker-compose run --rm -e CONNECTION_CHECK_MAX_COUNT=0 
airflow-worker airflow celery
   0.66s user 0.14s system 23% cpu 3.396 total
   $ time docker-compose run --rm -e CONNECTION_CHECK_MAX_COUNT=0 
airflow-worker airflow celery
   0.66s user 0.14s system 22% cpu 3.576 total
   $ time docker-compose run --rm -e CONNECTION_CHECK_MAX_COUNT=0 
airflow-worker airflow celery
   0.61s user 0.13s system 22% cpu 3.210 total
   ```
   ```
   $ time docker-compose run --rm -e CONNECTION_CHECK_MAX_COUNT=1 
airflow-worker airflow celery
   0.67s user 0.18s system 19% cpu 4.456 total
   $ time docker-compose run --rm -e CONNECTION_CHECK_MAX_COUNT=1 
airflow-worker airflow celery
   0.63s user 0.14s system 18% cpu 4.182 total
   $ time docker-compose run --rm -e CONNECTION_CHECK_MAX_COUNT=1 
airflow-worker airflow celery
   0.65s user 0.15s system 18% cpu 4.373 total
   ```
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #14535: Fix: Mysql 5.7 id utf8mb3

[GitBox]

boring-cyborg[bot] commented on pull request #14535:
URL: https://github.com/apache/airflow/pull/14535#issuecomment-787603756


   Congratulations on your first Pull Request and welcome to the Apache Airflow 
community! If you have any issues or are unsure about any anything please check 
our Contribution Guide 
(https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
   Here are some useful points:
   - Pay attention to the quality of your code (flake8, pylint and type 
annotations). Our [pre-commits]( 
https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks)
 will help you with that.
   - In case of a new feature add useful documentation (in docstrings or in 
`docs/` directory). Adding a new operator? Check this short 
[guide](https://github.com/apache/airflow/blob/master/docs/apache-airflow/howto/custom-operator.rst)
 Consider adding an example DAG that shows how users should use it.
   - Consider using [Breeze 
environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for 
testing locally, it’s a heavy docker but it ships with a working Airflow and a 
lot of integrations.
   - Be patient and persistent. It might take some time to get a review or get 
the final approval from Committers.
   - Please follow [ASF Code of 
Conduct](https://www.apache.org/foundation/policies/conduct) for all 
communication including (but not limited to) comments on Pull Requests, Mailing 
list and Slack.
   - Be sure to read the [Airflow Coding style]( 
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
   Apache Airflow is a community-driven project and together we are making it 
better 🚀.
   In case of doubts contact the developers at:
   Mailing List: d...@airflow.apache.org
   Slack: https://s.apache.org/airflow-slack
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] hbrls opened a new pull request #14535: Fix: Mysql 5.7 id utf8mb3

[GitBox]

hbrls opened a new pull request #14535:
URL: https://github.com/apache/airflow/pull/14535


   This may not be a ready PR. The code diff best expresses my ideas.
   
   I'm initializing a mysql 5.7 db from empty using `hub.docker.com, 
apache/airflow:2.0.1-python3.8`. The migration scripts throw errors of `(1071, 
'Specified key was too long; max key length is 767 bytes')`. 
`AIRFLOW__CORE__SQL_ENGINE_COLLATION_FOR_IDS: utf8mb3_general_ci` does not 
work. (ref #7570).
   
   After go through the issues and code, I had an assumption that `utf8mb3` 
config was not applied to all places that it should. I applied those migration 
scripts manually, then it works.
   
   I'm not confident about how the migration scripts work.
   
   1. Should it be generated by some tools and not modified manually?
   2. Some scripts modify existed columns. Can I directly fix the `create 
table` scripts and skip the `modify table` ones? How does this affect the old 
users wants to upgrade?
   
   Another question.
   
   I cannot fix the line 'bop.create_primary_key('pk_xcom', ['dag_id', 
'task_id', 'key', 'execution_date'])` and skip the column `execution_date`. My 
DBA always told me not to use too many columns for keys. Why in this case 
`dag_id` and `task_id` is not enough?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Abdur-rahmaanJ commented on issue #14283: To capitalise or not to capitalise

[GitBox]

Abdur-rahmaanJ commented on issue #14283:
URL: https://github.com/apache/airflow/issues/14283#issuecomment-787600869


   Companion PR: #14534 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] YangMuye commented on issue #14265: successful DAG run fails to be scheduled after being manually cleared if Dag.dagrun_timeout is set

[GitBox]

YangMuye commented on issue #14265:
URL: https://github.com/apache/airflow/issues/14265#issuecomment-787598399


   It seems that the dag run does not need to be successful first to hit this 
issue.
   A dag run that fails due to timeout can never be rerun. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Abdur-rahmaanJ opened a new pull request #14534: docs: capitalise & fixes (#14283)

[GitBox]

Abdur-rahmaanJ opened a new pull request #14534:
URL: https://github.com/apache/airflow/pull/14534


   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (8ced652 -> e273366)

[kamilbregula]

This is an automated email from the ASF dual-hosted git repository.

kamilbregula pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 8ced652  Implemented S3 Bucket Tagging (#14402)
 add e273366  Use airflow db check command in entrypoint_prod.sh (#14530)

No new revisions were added by this update.

Summary of changes:
 scripts/in_container/prod/entrypoint_prod.sh | 184 ++-
 1 file changed, 99 insertions(+), 85 deletions(-)

[GitHub] [airflow] mik-laj merged pull request #14530: Use airflow db check command in entrypoint_prod.sh

[GitBox]

mik-laj merged pull request #14530:
URL: https://github.com/apache/airflow/pull/14530


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dstandish commented on a change in pull request #14521: Add Asana Provider

[GitBox]

dstandish commented on a change in pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#discussion_r584355256



##
File path: airflow/providers/asana/operators/asana_tasks.py
##
@@ -0,0 +1,195 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Dict, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.asana.hooks.asana import AsanaHook
+from airflow.utils.decorators import apply_defaults
+
+
+class AsanaCreateTaskOperator(BaseOperator):
+"""
+This operator can be used to create Asana tasks. For more information on
+Asana optional task parameters, see 
https://developers.asana.com/docs/create-a-task
+
+.. seealso::
+For more information on how to use this operator, take a look at the 
guide:
+:ref:`howto/operator:AsanaCreateTaskOperator`
+
+:param asana_conn_id: The Asana connection to use.
+:type asana_conn_id: str
+:param name: Name of the task.
+:type name: str
+:param optional_task_parameters: Any of the optional task creation 
parameters.
+See https://developers.asana.com/docs/create-a-task for a complete 
list.
+You must specify at least one of 'workspace', 'parent', or 'projects'.
+:type optional_task_parameters: dict
+"""
+
+@apply_defaults
+def __init__(
+self,
+*,
+asana_conn_id: str,
+name: str,
+optional_task_parameters: Optional[dict] = None,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+self.asana_conn_id = asana_conn_id
+self.name = name
+self.optional_task_parameters = optional_task_parameters
+self.hook = AsanaHook(conn_id=self.asana_conn_id)
+
+def execute(self, context: Dict) -> str:
+asana_client = self.hook.get_conn()
+
+params = {"name": self.name}
+if self.optional_task_parameters is not None:
+params.update(self.optional_task_parameters)
+response = asana_client.tasks.create(params=params)
+
+self.log.info(response)
+return response["gid"]
+
+
+class AsanaUpdateTaskOperator(BaseOperator):
+"""
+This operator can be used to update Asana tasks.
+For more information on Asana optional task parameters, see
+https://developers.asana.com/docs/update-a-task
+
+.. seealso::
+For more information on how to use this operator, take a look at the 
guide:
+:ref:`howto/operator:AsanaUpdateTaskOperator`
+
+:param asana_conn_id: The Asana connection to use.
+:type asana_conn_id: str
+:param asana_task_gid: Asana task ID to update
+:type asana_task_gid: str
+:param task_update_parameters: Any task parameters that should be updated.
+See https://developers.asana.com/docs/update-a-task for a complete 
list.
+:type task_update_parameters: dict
+"""
+
+@apply_defaults
+def __init__(
+self,
+*,
+asana_conn_id: str,
+asana_task_gid: str,
+task_update_parameters: dict,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+
+self.asana_conn_id = asana_conn_id
+self.asana_task_gid = asana_task_gid
+self.task_update_parameters = task_update_parameters
+self.hook = AsanaHook(conn_id=self.asana_conn_id)
+
+def execute(self, context: Dict) -> None:
+asana_client = self.hook.get_conn()
+
+response = asana_client.tasks.update(task=self.asana_task_gid, 
params=self.task_update_parameters)
+self.log.info(response)
+
+
+class AsanaDeleteTaskOperator(BaseOperator):
+"""
+This operator can be used to delete Asana tasks.
+
+.. seealso::
+For more information on how to use this operator, take a look at the 
guide:
+:ref:`howto/operator:AsanaDeleteTaskOperator`
+
+:param asana_conn_id: The Asana connection to use.
+:type asana_conn_id: str
+:param asana_task_gid: Asana Task ID to delete.
+:type asana_task_gid: str
+"""
+
+@apply_defaults
+def __init__(
+self,
+*,
+asana_conn_id: str,
+asana_task_gid: str,
+**kwargs,
+) -> None:
+super().__init__(**kwargs)
+
+ 

[GitHub] [airflow] github-actions[bot] commented on pull request #14521: Add Asana Provider

[GitBox]

github-actions[bot] commented on pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#issuecomment-787572458


   [The Workflow run](https://github.com/apache/airflow/actions/runs/608974448) 
is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static 
checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider 
packages,^Checks: Helm tests$,^Test OpenAPI*.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (AIRFLOW-6786) Adding KafkaConsumerHook, KafkaProducerHook, and KafkaSensor

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292554#comment-17292554
 ] 

ASF GitHub Bot commented on AIRFLOW-6786:
-

github-actions[bot] closed pull request #11520:
URL: https://github.com/apache/airflow/pull/11520


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Adding KafkaConsumerHook, KafkaProducerHook, and KafkaSensor
> 
>
> Key: AIRFLOW-6786
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6786
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: contrib, hooks
>Affects Versions: 1.10.9
>Reporter: Daniel Ferguson
>Assignee: Daniel Ferguson
>Priority: Minor
>
> Add the KafkaProducerHook.
>  Add the KafkaConsumerHook.
>  Add the KafkaSensor which listens to messages with a specific topic.
>  Related Issue:
>  #1311 (Pre-dates Jira Migration)
> Reminder to contributors:
> You must add an Apache License header to all new files
>  Please squash your commits when possible and follow the 7 rules of good Git 
> commits
>  I am new to the community, I am not sure the files are at the right place or 
> missing anything.
> The sensor could be used as the first node of a dag where the second node can 
> be a TriggerDagRunOperator. The messages are polled in a batch and the dag 
> runs are dynamically generated.
> Thanks!
> Note, as per denied PR [#1415|https://github.com/apache/airflow/pull/1415], 
> it is important to mention these integrations are not suitable for 
> low-latency/high-throughput/streaming. For reference, [#1415 
> (comment)|https://github.com/apache/airflow/pull/1415#issuecomment-484429806].
> Co-authored-by: Dan Ferguson 
> [dferguson...@gmail.com|mailto:dferguson...@gmail.com]
>  Co-authored-by: YuanfΞi Zhu



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] github-actions[bot] commented on pull request #11699: Enforce ESLint (JavaScript linting) with pre-commit hooks

[GitBox]

github-actions[bot] commented on pull request #11699:
URL: https://github.com/apache/airflow/pull/11699#issuecomment-787558826


   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #11652: Support for sorting DAGs in the web UI

[GitBox]

github-actions[bot] commented on pull request #11652:
URL: https://github.com/apache/airflow/pull/11652#issuecomment-787558832


   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed pull request #11520: [AIRFLOW-6786] Added Kafka components differently

[GitBox]

github-actions[bot] closed pull request #11520:
URL: https://github.com/apache/airflow/pull/11520


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed pull request #11750: update variables names in _in_container_script_init.sh

[GitBox]

github-actions[bot] closed pull request #11750:
URL: https://github.com/apache/airflow/pull/11750


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #11540: Support DATE and DATETIME in MSSQLToGCSOperator

[GitBox]

github-actions[bot] commented on pull request #11540:
URL: https://github.com/apache/airflow/pull/11540#issuecomment-787558855


   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #11843: Fix issue with moving end_date in elastic dag

[GitBox]

github-actions[bot] commented on pull request #11843:
URL: https://github.com/apache/airflow/pull/11843#issuecomment-787558810


   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed pull request #12226: feat(CI): create markdown-link-checker.yml

[GitBox]

github-actions[bot] closed pull request #12226:
URL: https://github.com/apache/airflow/pull/12226


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #11627: Add D205 pydocstyle check

[GitBox]

github-actions[bot] commented on pull request #11627:
URL: https://github.com/apache/airflow/pull/11627#issuecomment-787558842


   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #12158: StatsD metrics have tags + generate reference docs

[GitBox]

github-actions[bot] commented on pull request #12158:
URL: https://github.com/apache/airflow/pull/12158#issuecomment-787558774


   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed pull request #12242: Enable markdownlint rule - MD012/no-multiple-blanks

[GitBox]

github-actions[bot] closed pull request #12242:
URL: https://github.com/apache/airflow/pull/12242


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #12087: Modified ajax call

[GitBox]

github-actions[bot] commented on pull request #12087:
URL: https://github.com/apache/airflow/pull/12087#issuecomment-787558783


   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #12271: Update example to use dag decorator tag

[GitBox]

github-actions[bot] commented on pull request #12271:
URL: https://github.com/apache/airflow/pull/12271#issuecomment-787558748


   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #11960: Log more types of errors when loading a DAG file to DB (POC)

[GitBox]

github-actions[bot] commented on pull request #11960:
URL: https://github.com/apache/airflow/pull/11960#issuecomment-787558796


   This pull request has been automatically marked as stale because it has not 
had recent activity. It will be closed in 5 days if no further activity occurs. 
Thank you for your contributions.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dstandish commented on a change in pull request #14492: Added new community provider Airbyte

[GitBox]

dstandish commented on a change in pull request #14492:
URL: https://github.com/apache/airflow/pull/14492#discussion_r584379136



##
File path: tests/providers/airbyte/hooks/test_airbyte.py
##
@@ -0,0 +1,107 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+import unittest
+from unittest import mock
+
+import pytest
+import requests
+import requests_mock
+
+from airflow.exceptions import AirflowException
+from airflow.models import Connection
+from airflow.providers.airbyte.hooks.airbyte import AirbyteHook
+
+AIRBYTE_CONN_ID = 'test'
+CONNECTION_ID = {"connectionId": "test"}
+JOB_ID = 1
+
+
+def get_airbyte_connection(unused_conn_id=None):
+return Connection(conn_id='test', conn_type='http', host='test:8001/')
+
+
+def mock_init(*args, **kwargs):
+pass
+
+
+class TestAirbyteHook(unittest.TestCase):
+"""Test get, post and raise_for_status"""
+
+def setUp(self):
+session = requests.Session()
+adapter = requests_mock.Adapter()
+session.mount('mock', adapter)
+get_airbyte_connection()

Review comment:
   does this have any effect?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dstandish commented on a change in pull request #14492: Added new community provider Airbyte

[GitBox]

dstandish commented on a change in pull request #14492:
URL: https://github.com/apache/airflow/pull/14492#discussion_r584377006



##
File path: airflow/providers/airbyte/hooks/airbyte.py
##
@@ -0,0 +1,92 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import time
+from typing import Optional
+
+from airflow.exceptions import AirflowException
+from airflow.providers.http.hooks.http import HttpHook
+
+
+class AirbyteJobController:
+"""Airbyte job status"""
+
+RUNNING = "running"
+SUCCEEDED = "succeeded"
+CANCELLED = "canceled"
+PENDING = "pending"
+FAILED = "failed"
+ERROR = "error"
+
+
+class AirbyteHook(HttpHook, AirbyteJobController):
+"""Hook for Airbyte API"""
+
+def __init__(self, airbyte_conn_id: str) -> None:
+super().__init__(http_conn_id=airbyte_conn_id)
+
+def wait_for_job(self, job_id: str, wait_time: int = 3, timeout: 
Optional[int] = None) -> None:

Review comment:
   > I worry a little about the async calls. Airbyte is a data 
integration/synchronization tool. It is strange that I want to trigger 
subsequent tasks before getting any results from AirbyteOperator. I agree that 
using the `op >> sensor` is more performative in terms of allocating Airflow 
resources, but I think it is good to leave these two options and create 
examples about them.
   
   right it's largely about about freeing up airflow resources.  but also by 
separating, there might be a case where you'd want to clear the sensor and not 
retrigger the submit.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] RNHTTR commented on a change in pull request #14321: Fix bug allowing task instances to survive when dagrun_timeout is exceeded

[GitBox]

RNHTTR commented on a change in pull request #14321:
URL: https://github.com/apache/airflow/pull/14321#discussion_r584217795



##
File path: airflow/jobs/scheduler_job.py
##
@@ -1696,10 +1696,18 @@ def _schedule_dag_run(
 and dag.dagrun_timeout
 and dag_run.start_date < timezone.utcnow() - dag.dagrun_timeout
 ):
-dag_run.state = State.FAILED
-dag_run.end_date = timezone.utcnow()
-self.log.info("Run %s of %s has timed-out", dag_run.run_id, 
dag_run.dag_id)
+dag_run.set_state(State.FAILED)
+unfinished_task_instances = (
+session.query(TI)
+.filter(TI.dag_id == dag_run.dag_id)
+.filter(TI.execution_date == dag_run.execution_date)
+.filter(TI.state.in_(State.unfinished))
+)
+for task_instance in unfinished_task_instances:
+task_instance.state = State.SKIPPED

Review comment:
   E.g:
   
   ``` python
   def test_do_schedule_max_active_runs_dag_timed_out(self):
   """Test that tasks are set to a finished state when their DAG times 
out"""
   
   dag = DAG(
   dag_id='test_max_active_run_with_dag_timed_out',
   start_date=DEFAULT_DATE,
   schedule_interval='@once',
   max_active_runs=1,
   catchup=True,
   )
   dag.dagrun_timeout = datetime.timedelta(seconds=1)
   
   with dag:
   task1 = BashOperator(
   task_id='task1',
   bash_command=' for((i=1;i<=600;i+=1)); do sleep "$i";  done',
   )
   
   session = settings.Session()
   dagbag = DagBag(
   dag_folder=os.devnull,
   include_examples=False,
   read_dags_from_db=True,
   )
   
   dagbag.bag_dag(dag=dag, root_dag=dag)
   dagbag.sync_to_db(session=session)
   
   run1 = dag.create_dagrun(
   run_type=DagRunType.SCHEDULED,
   execution_date=DEFAULT_DATE,
   state=State.RUNNING,
   session=session,
   )
   run1_ti = run1.get_task_instance(task1.task_id, session)
   run1_ti.state = State.RUNNING
   
   sleep(1)
   
   run2 = dag.create_dagrun(
   run_type=DagRunType.SCHEDULED,
   execution_date=DEFAULT_DATE + timedelta(seconds=10),
   state=State.RUNNING,
   session=session,
   )
   
   dag.sync_to_db(session=session)
   
   job = SchedulerJob(subdir=os.devnull)
   job.executor = MockExecutor()
   job.processor_agent = mock.MagicMock(spec=DagFileProcessorAgent)
   
   num_queued = job._do_scheduling(session)
   
   assert run1.state == State.FAILED
   assert run1_ti.state == State.SKIPPED
   assert run2.state == State.RUNNING
   
   num_queued = job._do_scheduling(session)
   run2_ti = run2.get_task_instance(task1.task_id, session)
   assert run2_ti.state == State.QUEUED





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] RNHTTR commented on a change in pull request #14321: Fix bug allowing task instances to survive when dagrun_timeout is exceeded

[GitBox]

RNHTTR commented on a change in pull request #14321:
URL: https://github.com/apache/airflow/pull/14321#discussion_r584373939



##
File path: airflow/jobs/scheduler_job.py
##
@@ -1696,10 +1696,18 @@ def _schedule_dag_run(
 and dag.dagrun_timeout
 and dag_run.start_date < timezone.utcnow() - dag.dagrun_timeout
 ):
-dag_run.state = State.FAILED
-dag_run.end_date = timezone.utcnow()
-self.log.info("Run %s of %s has timed-out", dag_run.run_id, 
dag_run.dag_id)
+dag_run.set_state(State.FAILED)
+unfinished_task_instances = (
+session.query(TI)
+.filter(TI.dag_id == dag_run.dag_id)
+.filter(TI.execution_date == dag_run.execution_date)
+.filter(TI.state.in_(State.unfinished))
+)
+for task_instance in unfinished_task_instances:
+task_instance.state = State.SKIPPED

Review comment:
   Test seems to be successful with a 1 second sleep. Just pushed a commit 
including a test for this scenario





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] RNHTTR commented on a change in pull request #14321: Fix bug allowing task instances to survive when dagrun_timeout is exceeded

[GitBox]

RNHTTR commented on a change in pull request #14321:
URL: https://github.com/apache/airflow/pull/14321#discussion_r584373939



##
File path: airflow/jobs/scheduler_job.py
##
@@ -1696,10 +1696,18 @@ def _schedule_dag_run(
 and dag.dagrun_timeout
 and dag_run.start_date < timezone.utcnow() - dag.dagrun_timeout
 ):
-dag_run.state = State.FAILED
-dag_run.end_date = timezone.utcnow()
-self.log.info("Run %s of %s has timed-out", dag_run.run_id, 
dag_run.dag_id)
+dag_run.set_state(State.FAILED)
+unfinished_task_instances = (
+session.query(TI)
+.filter(TI.dag_id == dag_run.dag_id)
+.filter(TI.execution_date == dag_run.execution_date)
+.filter(TI.state.in_(State.unfinished))
+)
+for task_instance in unfinished_task_instances:
+task_instance.state = State.SKIPPED

Review comment:
   Test seems to be successful with a 1 second sleep. Pushing the update now





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] marcosmarxm commented on a change in pull request #14492: Added new community provider Airbyte

[GitBox]

marcosmarxm commented on a change in pull request #14492:
URL: https://github.com/apache/airflow/pull/14492#discussion_r584370491



##
File path: airflow/providers/airbyte/hooks/airbyte.py
##
@@ -0,0 +1,91 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import time
+from typing import Optional
+
+from airflow.exceptions import AirflowException
+from airflow.providers.http.hooks.http import HttpHook
+
+
+class AirbyteHook(HttpHook):
+"""Hook for Airbyte API"""
+
+RUNNING = "running"
+SUCCEEDED = "succeeded"
+CANCELLED = "cancelled"
+PENDING = "pending"
+FAILED = "failed"
+ERROR = "error"
+
+def __init__(self, airbyte_conn_id: str, api_version: str = "v1") -> None:
+super().__init__(http_conn_id=airbyte_conn_id)
+self.api_version: str = api_version
+
+def wait_for_job(self, job_id: str, wait_seconds: int = 3, timeout: 
Optional[float] = None) -> None:
+"""
+Helper method which polls a job to check if it finishes.
+
+:param job_id: Id of the Airbyte job
+:type job_id: str
+:param wait_seconds: Number of seconds between checks
+:type wait_seconds: int
+:param timeout: How many seconds wait for job to be ready. Used only 
if ``asynchronous`` is False
+:type timeout: float
+"""
+state = None
+start = time.monotonic()
+while state not in (self.ERROR, self.SUCCEEDED, self.CANCELLED):
+if timeout and start + timeout < time.monotonic():
+raise AirflowException(f"Timeout: Airbyte job {job_id} is not 
ready after {timeout}s")
+time.sleep(wait_seconds)
+try:
+job = self.get_job(job_id=job_id)
+state = job.json().get("job").get("status")
+except AirflowException as err:
+self.log.info("Retrying. Airbyte API returned server error 
when waiting for job: %s", err)
+
+if state == self.ERROR:
+raise AirflowException(f"Job failed:\n{job}")
+if state == self.CANCELLED:
+raise AirflowException(f"Job was cancelled:\n{job}")
+
+def submit_sync_connection(self, connection_id: str) -> dict:
+"""
+Submits a job to a Airbyte server.
+
+:param connection_id: Required. The ConnectionId of the Airbyte 
Connection.
+:type connectiond_id: str
+"""
+return self.run(
+endpoint=f"api/{self.api_version}/connections/sync",
+json={"connectionId": connection_id},
+headers={"accept": "application/json"},
+)
+
+def get_job(self, job_id: str) -> dict:
+"""
+Gets the resource representation for a job in Airbyte.
+
+:param job_id: Id of the Airbyte job
+:type job_id: str
+"""
+return self.run(
+endpoint=f"api/{self.api_version}/jobs/get",

Review comment:
   @turbaszek created the `api_version` this way. Users can inform from the 
**AirbyteOperator** the version they like to use.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] marcosmarxm commented on a change in pull request #14492: Added new community provider Airbyte

[GitBox]

marcosmarxm commented on a change in pull request #14492:
URL: https://github.com/apache/airflow/pull/14492#discussion_r584370130



##
File path: airflow/providers/airbyte/hooks/airbyte.py
##
@@ -0,0 +1,92 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import time
+from typing import Optional
+
+from airflow.exceptions import AirflowException
+from airflow.providers.http.hooks.http import HttpHook
+
+
+class AirbyteJobController:
+"""Airbyte job status"""
+
+RUNNING = "running"
+SUCCEEDED = "succeeded"
+CANCELLED = "canceled"
+PENDING = "pending"
+FAILED = "failed"
+ERROR = "error"
+
+
+class AirbyteHook(HttpHook, AirbyteJobController):
+"""Hook for Airbyte API"""
+
+def __init__(self, airbyte_conn_id: str) -> None:
+super().__init__(http_conn_id=airbyte_conn_id)
+
+def wait_for_job(self, job_id: str, wait_time: int = 3, timeout: 
Optional[int] = None) -> None:

Review comment:
   @turbaszek and @dstandish I created the **AirbyteSensor** class. At that 
moment I think it is working like **DataprocOperator**. I can send a parameter 
`asynchronous = True`. In this way, **AirbyteOperator** returns the `job_id` 
right after the job creation call. That way, the user can follow the job using 
AirbyteSensor.
   The DAG would look something: `op >> sensor`.
   
   For cases where performance and works slot aren't a problem, it's possible 
to use the `asynchronous = False` (default) parameter the operator will make 
calls to the `wait_for_job` function waiting for some result: error, canceled, 
completed.
   
   I worry a little about the async calls. Airbyte is a data 
integration/synchronization tool. It is strange that I want to trigger 
subsequent tasks before getting any results from AirbyteOperator. I agree that 
using the `op >> sensor` is more performative in terms of allocating Airflow 
resources, but I think it is good to leave these two options and create 
examples about them.
   
   Example Airbyte API returns from the `/connections/sync` responsible to 
trigger a job.
   ```json
   {
 "job": {
   "id": 0,
   "configType": "check_connection_source",
   "configId": "string",
   "createdAt": 0,
   "updatedAt": 0,
   "status": "pending"
 },
 "attempts": [
   {
 "attempt": {
   "id": 0,
   "status": "running",
   "createdAt": 0,
   "updatedAt": 0,
   "endedAt": 0,
   "bytesSynced": 0,
   "recordsSynced": 0
 },
 "logs": {
   "logLines": [
 "string"
   ]
 }
   }
 ]
   }
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #14105: Add Google leveldb hook (#13109)

[GitBox]

github-actions[bot] commented on pull request #14105:
URL: https://github.com/apache/airflow/pull/14105#issuecomment-787533483


   The PR most likely needs to run full matrix of tests because it modifies 
parts of the core of Airflow. However, committers might decide to merge it 
quickly and take the risk. If they don't merge it quickly - please rebase it to 
the latest master at your convenience, or amend the last commit of the PR, and 
push it with --force-with-lease.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on pull request #14105: Add Google leveldb hook (#13109)

[GitBox]

ephraimbuddy commented on pull request #14105:
URL: https://github.com/apache/airflow/pull/14105#issuecomment-787533490


   Please rebase



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] marcosmarxm commented on a change in pull request #14492: Added new community provider Airbyte

[GitBox]

marcosmarxm commented on a change in pull request #14492:
URL: https://github.com/apache/airflow/pull/14492#discussion_r584368457



##
File path: airflow/providers/airbyte/hooks/airbyte.py
##
@@ -0,0 +1,92 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import time
+from typing import Optional
+
+from airflow.exceptions import AirflowException
+from airflow.providers.http.hooks.http import HttpHook
+
+
+class AirbyteJobController:
+"""Airbyte job status"""
+
+RUNNING = "running"
+SUCCEEDED = "succeeded"
+CANCELLED = "canceled"
+PENDING = "pending"
+FAILED = "failed"
+ERROR = "error"
+
+
+class AirbyteHook(HttpHook, AirbyteJobController):
+"""Hook for Airbyte API"""
+
+def __init__(self, airbyte_conn_id: str) -> None:
+super().__init__(http_conn_id=airbyte_conn_id)
+
+def wait_for_job(self, job_id: str, wait_time: int = 3, timeout: 
Optional[int] = None) -> None:
+"""
+Helper method which polls a job to check if it finishes.
+
+:param job_id: Id of the Airbyte job
+:type job_id: str
+:param wait_time: Number of seconds between checks
+:type wait_time: int
+:param timeout: How many seconds wait for job to be ready. Used only 
if ``asynchronous`` is False
+:type timeout: int
+"""
+state = None
+start = time.monotonic()
+while state not in (self.ERROR, self.SUCCEEDED, self.CANCELLED):
+if timeout and start + timeout < time.monotonic():
+raise AirflowException(f"Timeout: Airbyte job {job_id} is not 
ready after {timeout}s")
+time.sleep(wait_time)
+try:
+job = self.get_job(job_id=job_id)
+state = job.json().get("job").get("status")
+except AirflowException as err:
+self.log.info("Retrying. Airbyte API returned server error 
when waiting for job: %s", err)
+
+if state == self.ERROR:
+raise AirflowException(f"Job failed:\n{job}")
+if state == self.CANCELLED:
+raise AirflowException(f"Job was cancelled:\n{job}")
+
+def submit_job(self, connection_id: str) -> dict:

Review comment:
   You are correct! Already changed it





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on pull request #14254: Features/sftp to wasb: Transfer to scan files from sftp source path and upload them to Azure Blob Storage

[GitBox]

ephraimbuddy commented on pull request #14254:
URL: https://github.com/apache/airflow/pull/14254#issuecomment-787529838


   LGTM, just a nit..Can you use pure assert in tests(Because of this PR 
https://github.com/apache/airflow/pull/12951)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on issue #14505: Build failed likely due to unclean state of repo for self-hosted runner

[GitBox]

ashb commented on issue #14505:
URL: https://github.com/apache/airflow/issues/14505#issuecomment-787526083


   > Is the chown run always before ?
   
   Yes. That is in the pre-start script, and the runner is configured to run a 
single job then stop (at which point systemd restarts it)
   
   > And with root user ?
   
   Yes otherwise it wouldn't do anything



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil opened a new pull request #12599: Restrict "build images" workflow to certain branches

[GitBox]

kaxil opened a new pull request #12599:
URL: https://github.com/apache/airflow/pull/12599


   Currently, since we don't restrict it, on forks this runs the build step on 
all branches
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] RosterIn commented on a change in pull request #12688: attempt to upgrade mysqlclient

[GitBox]

RosterIn commented on a change in pull request #12688:
URL: https://github.com/apache/airflow/pull/12688#discussion_r584360961



##
File path: setup.py
##
@@ -363,8 +363,8 @@ def get_sphinx_theme_version() -> str:
 'pymssql~=2.1,>=2.1.5',
 ]
 mysql = [
-'mysql-connector-python>=8.0.11, <=8.0.22',
-'mysqlclient>=1.3.6,<1.4',
+'mysql-connector-python>=8.0.11, <=8.0.18',

Review comment:
   why do you want to use older version of `mysql-connector-python`?
   see https://github.com/apache/airflow/pull/13370





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil closed pull request #12599: Restrict "build images" workflow to certain branches

[GitBox]

kaxil closed pull request #12599:
URL: https://github.com/apache/airflow/pull/12599


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] RosterIn commented on a change in pull request #14028: Reuse private_key_passphrase from SSHHook super class

[GitBox]

RosterIn commented on a change in pull request #14028:
URL: https://github.com/apache/airflow/pull/14028#discussion_r584360235



##
File path: airflow/providers/ssh/hooks/ssh.py
##
@@ -131,9 +132,9 @@ def __init__(  # pylint: disable=too-many-statements
 self.key_file = extra_options.get("key_file")
 
 private_key = extra_options.get('private_key')
-private_key_passphrase = 
extra_options.get('private_key_passphrase')
+self.private_key_passphrase = 
extra_options.get('private_key_passphrase')
 if private_key:
-self.pkey = self._pkey_from_private_key(private_key, 
passphrase=private_key_passphrase)
+self.pkey = self._pkey_from_private_key(private_key, 
passphrase=self.private_key_passphrase)

Review comment:
   ```suggestion
   self.pkey = self._pkey_from_private_key(
   private_key, passphrase=self.private_key_passphrase
   )
   ```
   to make pylint happy





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #13153: WinRM Operator: Fix stout decoding issue & add option to run command

[GitBox]

kaxil commented on pull request #13153:
URL: https://github.com/apache/airflow/pull/13153#issuecomment-787522719


   Updated PR with fixes to static checks and rebased to master



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #13153: fix stout decoding issue and add option to run command as powershell script

[GitBox]

kaxil commented on a change in pull request #13153:
URL: https://github.com/apache/airflow/pull/13153#discussion_r584359471



##
File path: airflow/providers/microsoft/winrm/operators/winrm.py
##
@@ -88,10 +97,17 @@ def execute(self, context: dict) -> Union[list, str]:
 
 # pylint: disable=too-many-nested-blocks
 try:
-self.log.info("Running command: '%s'...", self.command)
-command_id = self.winrm_hook.winrm_protocol.run_command(  # type: 
ignore[attr-defined]
-winrm_client, self.command
-)
+if self.ps_path is not None:
+self.log.info("Running command as powershell script: '%s'...", 
self.command)
+encoded_ps = 
b64encode(self.command.encode('utf_16_le')).decode('ascii')
+command_id = self.winrm_hook.winrm_protocol.run_command(  # 
type: ignore[attr-defined]
+winrm_client, '{0} -encodedcommand 
{1}'.format(self.ps_path, encoded_ps)

Review comment:
   ```suggestion
   winrm_client, f'{self.ps_path} -encodedcommand 
{encoded_ps}'
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #13153: fix stout decoding issue and add option to run command as powershell script

[GitBox]

github-actions[bot] commented on pull request #13153:
URL: https://github.com/apache/airflow/pull/13153#issuecomment-787521932


   The PR is likely OK to be merged with just subset of tests for default 
Python and Database versions without running the full matrix of tests, because 
it does not modify the core of Airflow. If the committers decide that the full 
tests matrix is needed, they will add the label 'full tests needed'. Then you 
should rebase to the latest master or amend the last commit of the PR, and push 
it with --force-with-lease.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #14533: Add docs about celery monitoring

[GitBox]

github-actions[bot] commented on pull request #14533:
URL: https://github.com/apache/airflow/pull/14533#issuecomment-787520941


   The PR is likely ready to be merged. No tests are needed as no important 
environment files, nor python files were modified by it. However, committers 
might decide that full test matrix is needed and add the 'full tests needed' 
label. Then you should rebase it to the latest master or amend the last commit 
of the PR, and push it with --force-with-lease.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on pull request #14402: Implemented S3 Bucket Tagging

[GitBox]

github-actions[bot] commented on pull request #14402:
URL: https://github.com/apache/airflow/pull/14402#issuecomment-787520731


   The PR is likely OK to be merged with just subset of tests for default 
Python and Database versions without running the full matrix of tests, because 
it does not modify the core of Airflow. If the committers decide that the full 
tests matrix is needed, they will add the label 'full tests needed'. Then you 
should rebase to the latest master or amend the last commit of the PR, and push 
it with --force-with-lease.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (f25ec33 -> 8ced652)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from f25ec33  Add CLI check for scheduler (#14519)
 add 8ced652  Implemented S3 Bucket Tagging (#14402)

No new revisions were added by this update.

Summary of changes:
 .../amazon/aws/example_dags/example_s3_bucket.py   |   2 +
 ...e_s3_bucket.py => example_s3_bucket_tagging.py} |  49 +++
 airflow/providers/amazon/aws/hooks/s3.py   |  73 +-
 .../providers/amazon/aws/operators/s3_bucket.py|   8 ++
 .../amazon/aws/operators/s3_bucket_tagging.py  | 152 +
 airflow/providers/amazon/provider.yaml |   3 +
 .../operators/s3.rst   | 105 ++
 tests/providers/amazon/aws/hooks/test_s3.py| 122 +
 .../amazon/aws/operators/test_s3_bucket_tagging.py | 120 
 ..._bucket.py => test_s3_bucket_tagging_system.py} |  23 +++-
 10 files changed, 627 insertions(+), 30 deletions(-)
 copy airflow/providers/amazon/aws/example_dags/{example_s3_bucket.py => 
example_s3_bucket_tagging.py} (54%)
 create mode 100644 airflow/providers/amazon/aws/operators/s3_bucket_tagging.py
 create mode 100644 docs/apache-airflow-providers-amazon/operators/s3.rst
 create mode 100644 
tests/providers/amazon/aws/operators/test_s3_bucket_tagging.py
 copy tests/providers/amazon/aws/operators/{test_example_s3_bucket.py => 
test_s3_bucket_tagging_system.py} (67%)

[GitHub] [airflow] potiuk merged pull request #14402: Implemented S3 Bucket Tagging

[GitBox]

potiuk merged pull request #14402:
URL: https://github.com/apache/airflow/pull/14402


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #14402: Implemented S3 Bucket Tagging

[GitBox]

boring-cyborg[bot] commented on pull request #14402:
URL: https://github.com/apache/airflow/pull/14402#issuecomment-787520698


   Awesome work, congrats on your first merged pull request!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk closed issue #14279: Add AWS S3 Bucket Tagging Operator

[GitBox]

potiuk closed issue #14279:
URL: https://github.com/apache/airflow/issues/14279


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on a change in pull request #14402: Implemented S3 Bucket Tagging

[GitBox]

potiuk commented on a change in pull request #14402:
URL: https://github.com/apache/airflow/pull/14402#discussion_r584358322



##
File path: airflow/providers/amazon/aws/hooks/s3.py
##
@@ -848,3 +848,74 @@ def generate_presigned_url(
 except ClientError as e:
 self.log.error(e.response["Error"]["Message"])
 return None
+
+@provide_bucket_name
+def get_bucket_tagging(self, bucket_name: Optional[str] = None) -> 
Optional[List[Dict[str, str]]]:
+"""
+Gets a List of tags from a bucket.
+
+:param bucket_name: The name of the bucket.
+:type bucket_name: str
+:return: A List containing the key/value pairs for the tags
+:rtype: Optional[List[Dict[str, str]]]

Review comment:
   Yep. This is what we usually prefer for the integrations/providers - to 
transparently move what the API we integrate with rather than fix it - this is 
usually more future-proof.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dstandish commented on a change in pull request #14521: Add Asana Provider

[GitBox]

dstandish commented on a change in pull request #14521:
URL: https://github.com/apache/airflow/pull/14521#discussion_r584350856



##
File path: airflow/providers/asana/operators/asana_tasks.py
##
@@ -0,0 +1,195 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Dict, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.asana.hooks.asana import AsanaHook
+from airflow.utils.decorators import apply_defaults
+
+
+class AsanaCreateTaskOperator(BaseOperator):
+"""
+This operator can be used to create Asana tasks. For more information on
+Asana optional task parameters, see 
https://developers.asana.com/docs/create-a-task
+
+.. seealso::
+For more information on how to use this operator, take a look at the 
guide:
+:ref:`howto/operator:AsanaCreateTaskOperator`
+
+:param asana_conn_id: The Asana connection to use.
+:type asana_conn_id: str
+:param name: Name of the task.
+:type name: str
+:param optional_task_parameters: Any of the optional task creation 
parameters.
+See https://developers.asana.com/docs/create-a-task for a complete 
list.
+You must specify at least one of 'workspace', 'parent', or 'projects'.
+:type optional_task_parameters: dict
+"""
+
+@apply_defaults
+def __init__(
+self,
+*,
+asana_conn_id: str,
+name: str,
+optional_task_parameters: Optional[dict] = None,

Review comment:
   ```suggestion
   task_parameters: Optional[dict] = None,
   ```
   
   i'd suggest calling it `task_parameters`
   
   the fact that they are optional is clear enough from the type annotation and 
default
   

##
File path: airflow/providers/asana/hooks/asana.py
##
@@ -0,0 +1,60 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Connect to Asana."""
+
+from asana import Client
+
+from airflow.hooks.base import BaseHook
+
+
+class AsanaHook(BaseHook):
+"""
+Wrapper around Asana Python client library.
+"""
+
+conn_name_attr = "asana_conn_id"
+default_conn_name = "asana_default"
+conn_type = "asana"
+hook_name = "Asana"
+
+def __init__(self, conn_id: str = default_conn_name, *args, **kwargs) -> 
None:
+super().__init__(*args, **kwargs)
+self.asana_conn_id = conn_id
+self.connection = kwargs.pop("connection", None)
+self.client = None
+self.extras = None
+self.uri = None
+
+def get_conn(self) -> Client:
+"""
+Creates Asana Client
+"""
+self.connection = self.get_connection(self.asana_conn_id)
+self.extras = self.connection.extra_dejson.copy()
+
+if self.client is not None:

Review comment:
   one way this can be made a little cleaner is to make `client` a cached 
property and let get_conn just return `self.client`
   
   this let's you clean up a few things... e.g. you don't need to inititalize 
`self.client = None`, don't need to check if it is none, don't need to assign 
it when creating ... you just return the client object

##
File path: airflow/providers/asana/operators/asana_tasks.py
##
@@ -0,0 +1,195 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under 

[GitHub] [airflow] potiuk commented on issue #14505: Build failed likely due to unclean state of repo for self-hosted runner

[GitBox]

potiuk commented on issue #14505:
URL: https://github.com/apache/airflow/issues/14505#issuecomment-787515344


   My hypothesis is that the previous runs docker containers are still working 
in the background (killing a docker container running in the background might 
not happen immeddiately).  Those running docker containers could continue 
creating files while the next "init" is happening. That's why "chown" would not 
work - because the files get created . 
   
   That's where re-creating the whole tmpfs from scratch could help (if that 
hypothesis is right) - first of all deleting the tmpfs volume could fail (or 
wait until the docker container finish), secondly recreating the whole volume 
is a sure way to always be 'clean' - even if there is anything writing to the 
old tmpfs, the new volume will not share anything with the new one.
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #14505: Build failed likely due to unclean state of repo for self-hosted runner

[GitBox]

potiuk edited a comment on issue #14505:
URL: https://github.com/apache/airflow/issues/14505#issuecomment-787512598


   > Will take a look in detail tomorrow, but permissions are being reset by 
https://github.com/apache/airflow-ci-infra/blob/user-data-WIP/cloud-init.yml#L129.
   > 
   > Wonder what went on here
   
   Is the chown run always before ? And with root user ?
   
   I think this is one of those cases where I prefer to have a completely 
"pristine"  empty directory rather trying to reuse last checkout.
   
   GitHub checkout by default only fetches one single commit, it is super-fast 
so we save basically nothing by caching the result of previous checkout. And we 
avoid any kind of errors like this one - where some edge case we missed 
happens. 
   
   I had in the past numerous similar issues on pretty much every single CI 
system I built. And starting from clean state was always final state I got to.  
This saved a lot of time for me and for the users trying to understand what's 
going on. 
   
   I propose we just hard-wipe workdir always. We will save a lot of time for 
debugging - this is what pretty much every single CI out there does, so there 
is no point in doing it differently.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on issue #14505: Build failed likely due to unclean state of repo for self-hosted runner

[GitBox]

potiuk edited a comment on issue #14505:
URL: https://github.com/apache/airflow/issues/14505#issuecomment-787512598


   > Will take a look in detail tomorrow, but permissions are being reset by 
https://github.com/apache/airflow-ci-infra/blob/user-data-WIP/cloud-init.yml#L129.
   > 
   > Wonder what went on here
   
   Is the chown run always before ? And with root user ?
   
   I think this is one of those cases where I prefer to have a completely 
"pristine"  empty directory rather trying to reuse last checkout.
   
   GitHub checkout by default only fetches one single commit, it is super-fast 
so we save basically nothing by caching the result of previous checkout. And we 
avoid any kind of errors like this one - where some edge case we missed 
happens. 
   
   I had in the past numerous similar issues on pretty much every single CI 
system I built. And starting from clean state was always final state I got to.  
This saved a lot of time for me and for the users trying to understand what's 
going on. 
   
   I propose we just hard-wipe workdir always. We will save a lot of time for 
debugging.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #14505: Build failed likely due to unclean state of repo for self-hosted runner

[GitBox]

potiuk commented on issue #14505:
URL: https://github.com/apache/airflow/issues/14505#issuecomment-787512598


   > Will take a look in detail tomorrow, but permissions are being reset by 
https://github.com/apache/airflow-ci-infra/blob/user-data-WIP/cloud-init.yml#L129.
   > 
   > Wonder what went on here
   
   Is the chown run always before ? And with root user ?
   
   I think this is one of those cases where I prefer to have a completely 
"pristine"  empty directory rather trying to reuse last checkout.
   
   GitHub checkout by default only fetches one single commit, it is super-fast 
so we save basically nothing by caching the result of previous checkout. And we 
avoid any kind of errors like this one - where some edge case is handled. 
   
   I had in the past numerous similar issues on pretty much every single CI 
system I built. And starting from clean state was always final state I got to.  
This saved a lot of time for me and for the users trying to understand what's 
going on. 
   
   I propose we just hard-wipe workdir always. We will save a lot of time for 
debugging.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj opened a new pull request #14533: Add docs about celery monitoring

[GitBox]

mik-laj opened a new pull request #14533:
URL: https://github.com/apache/airflow/pull/14533


   Part of:  https://github.com/apache/airflow/issues/11161
   
   To have a full description of the monitoring of all Airflow components, I 
add information about HTTP and CLI checks for Celery. Thanks to this, we will 
not have to search for information one by one, but everything will be in one 
place. 
   
   Documentation for Celery does not describe the inspect ping command, but 
hopefully, this will be added soon.
   
   CC: @auvipy 
   
   
   
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on pull request #14531: Running tests in parallel for self-hosted runners

[GitBox]

ashb commented on pull request #14531:
URL: https://github.com/apache/airflow/pull/14531#issuecomment-787510745


   > but I am not sure if the auto-scaling solution we have will handle two 
types
   
   Not yet, but it could. We'd need two ASGs, and the logic for which one to 
shake up would need encoding in the lambda.
   
   Some of the upload steps can be reduced by moving everything to us-east-* 
regions too - closer to Azure where GitHub runs, and closer to docker.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #14531: Running tests in parallel for self-hosted runners

[GitBox]

potiuk edited a comment on pull request #14531:
URL: https://github.com/apache/airflow/pull/14531#issuecomment-787506971


   Hey @ashb - we need bigger machines as I suspected :) . 
   
   The good news is that it will be much cheaper in the long run as we will 
need them for far less time.
   
   The tests are failing but mainly because of memory problems and timeouts (so 
I guess we are simply using too much of RAM , if we up the machine to 64 GB I 
think this should go rather smoothly. The good news is that even with not 
enough memory (and with failures/timeouts) the tests took ~26 m (!) for 
`sqlite` - rather than > 1 h, so when we have enough memory we can achieve the 
15 minutes I was hoping for. Those 64 GB machines are only a bit more expensive 
than the 32 GB ones, so we will save a lot of credits when it works. 
   
   We can even optimize it away a bit and have two self-hosted types:
   
   1) Big 64 GB ones for the tests
   2) Smaller 32 GB ones  for everything else
   
   It should be rather easy to configure in the `CI.yml`, but I am not sure if 
the auto-scaling solution we have will handle two types?
   
   Here is the job that we have partial successes/failures and it shows how 
those tests will look like. This is actually a good  one to show how the tests 
will look like.  You can see that the output is nicely grouped and you can see 
very clearly the monitoring and progress (it will be much nicer when we have 
more memory because each test will progress much faster). Also I print summary 
of the failed tests at the end - only "failure" outputs are fully printed to 
the logs at the end with "Red" groups - this will make it far easier to analyze 
problems (the same kind of output improvement is in the sequential version of 
the tests run on GitHub runners).
   
   In this  case three test types succeeded (`Heisentests Core Providers`) and 
remaining 5 had some failures (most of them from what I see is due to timeouts, 
which is perfectly understandable if we run out of memory and started to swap 
out to remote SSD in the cloud): 
https://github.com/apache/airflow/pull/14531/checks?check_run_id=1999175947. 
   
   ## Rationale for bigger machines
   
   From what I see we have machines with 32 GB and since half of it will easily 
be eaten by `tmpfs` when we start writing logs and the like, we only have ~16 
GB which is not enough. During my tests: 
https://twitter.com/higrys/status/1366037359461101569/photo/1 all the tests 
running in parallel took ~35 GB of memory on my 64 GB machine. I had just local 
SSD not `tmpfs` for those tests, but i do not think we need 30 GB tmpfs for all 
logs, docker, tmp etc (and we can fine tune that if we do).  
   
   Also it is more important than before to clean-up the `tmpfs` volumes before 
each run and make them "pristine" for every run - because we will be using 
nearly all of it. I think that will also help with cases like #14505 where some 
left-overs from previous runs are causing the jobs to fail.
   
   This is the mchine state before the tests are run: 
   
   ```
   totalusedfree  shared  buff/cache   
available
 Mem:   30Gi   696Mi24Gi   3.4Gi   5.5Gi
26Gi
 Swap:0B  0B  0B
   
 Filesystem  Size  Used Avail Use% Mounted on
 /dev/root   7.7G  2.6G  5.2G  34% /
 devtmpfs 16G 0   16G   0% /dev
 tmpfs16G 0   16G   0% /dev/shm
 tmpfs   3.1G  804K  3.1G   1% /run
 tmpfs   5.0M 0  5.0M   0% /run/lock
 tmpfs16G 0   16G   0% /sys/fs/cgroup
 tmpfs   3.1G  168K  3.1G   1% /tmp
 tmpfs21G  2.9G   18G  15% /var/lib/docker
 tmpfs16G  534M   15G   4% /home/runner/actions-runner/_work
 /dev/loop0   98M   98M 0 100% /snap/core/10185
 /dev/loop1   56M   56M 0 100% /snap/core18/1885
 /dev/loop2   71M   71M 0 100% /snap/lxd/16922
 /dev/loop3   29M   29M 0 100% /snap/amazon-ssm-agent/2012
   
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #14531: Running tests in parallel for self-hosted runners

[GitBox]

potiuk edited a comment on pull request #14531:
URL: https://github.com/apache/airflow/pull/14531#issuecomment-787506971


   Hey @ashb - we need bigger machines as I suspected :) . 
   
   The good news is that it will be much cheaper in the long run as we will 
need them for far less time.
   
   The tests are failing but mainly because of memory problems and timeouts (so 
I guess we are simply using too much of RAM , if we up the machine to 64 GB I 
think this should go rather smoothly. The good news is that even with not 
enough memory (and with failures/timeouts) the tests took ~26 m for `sqlite` - 
rather than > 1 h, so when we have enough memory we can achieve the 15 minutes 
I was hoping for. Those 64 GB machines are only a bit more expensive than the 
32 GB ones, so we will save a lot of credits when it works. 
   
   We can even optimize it away a bit and have two self-hosted types:
   
   1) Big 64 GB ones for the tests
   2) Smaller 32 GB ones  for everything else
   
   It should be rather easy to configure in the `CI.yml`, but I am not sure if 
the auto-scaling solution we have will handle two types?
   
   Here is the job that we have partial successes/failures and it shows how 
those tests will look like. This is actually a good  one to show how the tests 
will look like.  You can see that the output is nicely grouped and you can see 
very clearly the monitoring and progress (it will be much nicer when we have 
more memory because each test will progress much faster). Also I print summary 
of the failed tests at the end - only "failure" outputs are fully printed to 
the logs at the end with "Red" groups - this will make it far easier to analyze 
problems (the same kind of output improvement is in the sequential version of 
the tests run on GitHub runners).
   
   In this  case three test types succeeded (`Heisentests Core Providers`) and 
remaining 5 had some failures (most of them from what I see is due to timeouts, 
which is perfectly understandable if we run out of memory and started to swap 
out to remote SSD in the cloud): 
https://github.com/apache/airflow/pull/14531/checks?check_run_id=1999175947. 
   
   ## Rationale for bigger machines
   
   From what I see we have machines with 32 GB and since half of it will easily 
be eaten by `tmpfs` when we start writing logs and the like, we only have ~16 
GB which is not enough. During my tests: 
https://twitter.com/higrys/status/1366037359461101569/photo/1 all the tests 
running in parallel took ~35 GB of memory on my 64 GB machine. I had just local 
SSD not `tmpfs` for those tests, but i do not think we need 30 GB tmpfs for all 
logs, docker, tmp etc (and we can fine tune that if we do).  
   
   Also it is more important than before to clean-up the `tmpfs` volumes before 
each run and make them "pristine" for every run - because we will be using 
nearly all of it. I think that will also help with cases like #14505 where some 
left-overs from previous runs are causing the jobs to fail.
   
   This is the mchine state before the tests are run: 
   
   ```
   totalusedfree  shared  buff/cache   
available
 Mem:   30Gi   696Mi24Gi   3.4Gi   5.5Gi
26Gi
 Swap:0B  0B  0B
   
 Filesystem  Size  Used Avail Use% Mounted on
 /dev/root   7.7G  2.6G  5.2G  34% /
 devtmpfs 16G 0   16G   0% /dev
 tmpfs16G 0   16G   0% /dev/shm
 tmpfs   3.1G  804K  3.1G   1% /run
 tmpfs   5.0M 0  5.0M   0% /run/lock
 tmpfs16G 0   16G   0% /sys/fs/cgroup
 tmpfs   3.1G  168K  3.1G   1% /tmp
 tmpfs21G  2.9G   18G  15% /var/lib/docker
 tmpfs16G  534M   15G   4% /home/runner/actions-runner/_work
 /dev/loop0   98M   98M 0 100% /snap/core/10185
 /dev/loop1   56M   56M 0 100% /snap/core18/1885
 /dev/loop2   71M   71M 0 100% /snap/lxd/16922
 /dev/loop3   29M   29M 0 100% /snap/amazon-ssm-agent/2012
   
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #14531: Running tests in parallel for self-hosted runners

[GitBox]

potiuk edited a comment on pull request #14531:
URL: https://github.com/apache/airflow/pull/14531#issuecomment-787506971


   Hey @ashb - we need bigger machines as I suspected :) . 
   
   The good news is that it will be much cheaper in the long run as we will 
need them for far less time.
   
   The tests are failing but mainly because of memory problems and timeouts (so 
I guess we are simply using too much of RAM , if we up the machine to 64 GB I 
think this should go rather smoothly. The good news is that even with not 
enough memory (and with failures/timeouts) the tests took ~26 m for `sqlite` - 
rather than > 1 h, so when we have enough memory we can achieve the 15 minutes 
I was hoping for. Those 64 GB machines are only a bit more expensive than the 
32 GB ones, so we will save a lot of credits when it works. 
   
   We can even optimize it away a bit and have two self-hosted types:
   
   1) Big 64 GB ones for the tests
   2) Smaller 32 GB ones  for everything else
   
   It should be rather easy to configure in the `CI.yml`, but I am not sure if 
the auto-scaling solution we have will handle two types?
   
   Here is the job that we have partial successes/failures and it shows how 
those tests will look like. This is actually a good  one to show how the tests 
will look like.  You can see that the output is nicely grouped and you can see 
very clearly the monitoring and progress (it will be much nicer when we have 
more memory because each test will progress much faster). Also I print summary 
of the failed tests at the end - only "failure" outputs are fully printed to 
the logs at the end with "Red" groups - this will make it far easier to analyze 
problems (the same kind of output improvement is in the sequential version of 
the tests run on GitHub runners).
   
   In this  case three test types succeeded (`Heisentests Core Providers`) and 
remaining 5 had some failures (most of them from what I see is due to timeouts, 
which is perfectly understandable if we run out of memory and started to swap 
out to remote SSD in the cloud): 
https://github.com/apache/airflow/pull/14531/checks?check_run_id=1999175947. 
   
   Rationale for bigger machines:
   
   From what I see we have machines with 32 GB and since half of it will easily 
be eaten by `tmpfs` when we start writing logs and the like, we only have ~16 
GB which is not enough. During my tests: 
https://twitter.com/higrys/status/1366037359461101569/photo/1 all the tests 
running in parallel took ~35 GB of memory on my 64 GB machine. I had just local 
SSD not `tmpfs` for those tests, but i do not think we need 30 GB tmpfs for all 
logs, docker, tmp etc (and we can fine tune that if we do).  
   
   Also it is more important than before to clean-up the `tmpfs` volumes before 
each run and make them "pristine" for every run - because we will be using 
nearly all of it. I think that will also help with cases like #14505 where some 
left-overs from previous runs are causing the jobs to fail.
   
   This is the mchine state before the tests are run: 
   
   ```
   totalusedfree  shared  buff/cache   
available
 Mem:   30Gi   696Mi24Gi   3.4Gi   5.5Gi
26Gi
 Swap:0B  0B  0B
   
 Filesystem  Size  Used Avail Use% Mounted on
 /dev/root   7.7G  2.6G  5.2G  34% /
 devtmpfs 16G 0   16G   0% /dev
 tmpfs16G 0   16G   0% /dev/shm
 tmpfs   3.1G  804K  3.1G   1% /run
 tmpfs   5.0M 0  5.0M   0% /run/lock
 tmpfs16G 0   16G   0% /sys/fs/cgroup
 tmpfs   3.1G  168K  3.1G   1% /tmp
 tmpfs21G  2.9G   18G  15% /var/lib/docker
 tmpfs16G  534M   15G   4% /home/runner/actions-runner/_work
 /dev/loop0   98M   98M 0 100% /snap/core/10185
 /dev/loop1   56M   56M 0 100% /snap/core18/1885
 /dev/loop2   71M   71M 0 100% /snap/lxd/16922
 /dev/loop3   29M   29M 0 100% /snap/amazon-ssm-agent/2012
   
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #14531: Running tests in parallel for self-hosted runners

[GitBox]

potiuk edited a comment on pull request #14531:
URL: https://github.com/apache/airflow/pull/14531#issuecomment-787506971


   Hey @ashb - we need bigger machines as I suspected :) . 
   
   The good news is that it will be much cheaper in the long run as we will 
need them for far less time.
   
   The tests are failing but mainly because of memory problems and timeouts (so 
I guess we are simply using too much of RAM , if we up the machine to 64 GB I 
think this should go rather smoothly. The good news is that even with not 
enough memory (and with failures/timeouts) the tests took ~26 m for `sqlite` - 
rather than > 1 h, so when we have enough memory we can achieve the 15 minutes 
I was hoping for. Those 64 GB machines are only a bit more expensive than the 
32 GB ones, so we will save a lot of credits when it works. 
   
   We can even optimize it away a bit and have two self-hosted types:
   
   1) Big 64 GB ones for the tests
   2) Smaller 32 GB ones  for everything else
   
   It should be rather easy to configure in the `CI.yml`, but I am not sure if 
the auto-scaling solution we have will handle two types?
   
   Here is the job that we have partial successes/failures and it shows how 
those tests will look like. This is actually a good  one to show how the tests 
will look like.  You can see that the output is nicely grouped and you can see 
very clearly the monitoring and progress (it will be much nicer when we have 
more memory because each test will progress much faster). Also I print summary 
of the failed tests at the end - only "failure" outputs are fully printed to 
the logs at the end with "Red" groups - this will make it far easier to analyze 
problems (the same kind of output improvement is in the sequential version of 
the tests run on GitHub runners).
   
   In this  case three test types succeeded (`Heisentests Core Providers`) and 
remaining 5 had some failures (most of them from what I see is due to timeouts, 
which is perfectly understandable if we run out of memory and started to swap 
out to remote SSD in the cloud): 
https://github.com/apache/airflow/pull/14531/checks?check_run_id=1999175947. 
   
   Rationale for bigger machines:
   
   From what I see we have machines with 32 GB and since half of it will easily 
be eaten by `tmpfs` when we start writing logs and the like, we only have ~16 
GB which is not enough. During my tests: 
https://twitter.com/higrys/status/1366037359461101569/photo/1 all the tests 
running in parallel took ~35 GB of memory on my 64 GB machine. I had just local 
SSD not `tmpfs` for those tests, but i do not think we need 30 GB tmpfs for all 
logs, docker, tmp etc (and we can fine tune that if we do).  
   
   Also it is more important than before to clean-up the `tmpfs` volumes before 
each run and make them "pristine" for every run - because we will be using 
nearly all of it. I think that will also help with cases like #14505 where some 
left-overs from previous runs are causing the jobs to fail.
   
   This is the mchine state before the tests are run: 
   
   ```
 totalusedfree  shared  buff/cache  
 available
 Mem:   30Gi   696Mi24Gi   3.4Gi   5.5Gi
26Gi
 Swap:0B  0B  0B
   
 Filesystem  Size  Used Avail Use% Mounted on
 /dev/root   7.7G  2.6G  5.2G  34% /
 devtmpfs 16G 0   16G   0% /dev
 tmpfs16G 0   16G   0% /dev/shm
 tmpfs   3.1G  804K  3.1G   1% /run
 tmpfs   5.0M 0  5.0M   0% /run/lock
 tmpfs16G 0   16G   0% /sys/fs/cgroup
 tmpfs   3.1G  168K  3.1G   1% /tmp
 tmpfs21G  2.9G   18G  15% /var/lib/docker
 tmpfs16G  534M   15G   4% /home/runner/actions-runner/_work
 /dev/loop0   98M   98M 0 100% /snap/core/10185
 /dev/loop1   56M   56M 0 100% /snap/core18/1885
 /dev/loop2   71M   71M 0 100% /snap/lxd/16922
 /dev/loop3   29M   29M 0 100% /snap/amazon-ssm-agent/2012
   
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #14531: Running tests in parallel for self-hosted runners

[GitBox]

potiuk edited a comment on pull request #14531:
URL: https://github.com/apache/airflow/pull/14531#issuecomment-787506971


   Hey @ashb - we need bigger machines as I suspected :) . 
   
   The good news is that it will be much cheaper in the long run as we will 
need them for far less time.
   
   The tests are failing but mainly because of memory problems and timeouts (so 
I guess we are simply using too much of RAM , if we up the machine to 64 GB I 
think this should go rather smoothly. The good news is that even with not 
enough memory (and with failures/timeouts) the tests took ~26 m for `sqlite` - 
rather than > 1 h, so when we have enough memory we can achieve the 15 minutes 
I was hoping for. Those 64 GB machines are only a bit more expensive than the 
32 GB ones, so we will save a lot of credits when it works. 
   
   We can even optimize it away a bit and have two self-hosted types:
   
   1) Big 64 GB ones for the tests
   2) Smaller 32 GB ones  for everything else
   
   It should be rather easy to configure in the `CI.yml`, but I am not sure if 
the auto-scaling solution we have will handle two types?
   
   Here is the job that we have partial successes/failures and it shows how 
those tests will look like. This is actually a good  one to show how the tests 
will look like.  You can see that the output is nicely grouped and you can see 
very clearly the monitoring and progress (it will be much nicer when we have 
more memory because each test will progress much faster). Also I print summary 
of the failed tests at the end - only "failure" outputs are fully printed to 
the logs at the end with "Red" groups - this will make it far easier to analyze 
problems (the same kind of output improvement is in the sequential version of 
the tests run on GitHub runners).
   
   In this  case three test types succeeded (`Heisentests Core Providers`) and 
remaining 5 had some failures (most of them from what I see is due to timeouts, 
which is perfectly understandable if we run out of memory and started to swap 
out to remote SSD in the cloud): 
https://github.com/apache/airflow/pull/14531/checks?check_run_id=1999175947. 
   
   Rationale for bigger machines:
   
   From what I see we have machines with 32 GB and since half of it will easily 
be eaten by `tmpfs` when we start writing logs and the like, we only have ~16 
GB which is not enough. During my tests: 
https://twitter.com/higrys/status/1366037359461101569/photo/1 all the tests 
running in parallel took ~35 GB of memory on my 64 GB machine. I had just local 
SSD not `tmpfs` for those tests, but i do not think we need 30 GB tmpfs for all 
logs, docker, tmp etc (and we can fine tune that if we do).  
   
   Also it is more important than before to clean-up the `tmpfs` volumes before 
each run and make them "pristine" for every run - because we will be using 
nearly all of it. I think that will also help with cases like #14505 where some 
left-overs from previous runs are causing the jobs to fail.
   
   ```
 totalusedfree  shared  buff/cache  
 available
 Mem:   30Gi   696Mi24Gi   3.4Gi   5.5Gi
26Gi
 Swap:0B  0B  0B
   
 Filesystem  Size  Used Avail Use% Mounted on
 /dev/root   7.7G  2.6G  5.2G  34% /
 devtmpfs 16G 0   16G   0% /dev
 tmpfs16G 0   16G   0% /dev/shm
 tmpfs   3.1G  804K  3.1G   1% /run
 tmpfs   5.0M 0  5.0M   0% /run/lock
 tmpfs16G 0   16G   0% /sys/fs/cgroup
 tmpfs   3.1G  168K  3.1G   1% /tmp
 tmpfs21G  2.9G   18G  15% /var/lib/docker
 tmpfs16G  534M   15G   4% /home/runner/actions-runner/_work
 /dev/loop0   98M   98M 0 100% /snap/core/10185
 /dev/loop1   56M   56M 0 100% /snap/core18/1885
 /dev/loop2   71M   71M 0 100% /snap/lxd/16922
 /dev/loop3   29M   29M 0 100% /snap/amazon-ssm-agent/2012
   
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj merged pull request #14519: Add CLI check for scheduler(s)

[GitBox]

mik-laj merged pull request #14519:
URL: https://github.com/apache/airflow/pull/14519


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (9f35cff -> f25ec33)

[kamilbregula]

This is an automated email from the ASF dual-hosted git repository.

kamilbregula pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 9f35cff  Fix asset recompilation message (#14532)
 add f25ec33  Add CLI check for scheduler (#14519)

No new revisions were added by this update.

Summary of changes:
 airflow/cli/cli_parser.py  |  76 ++--
 airflow/cli/commands/jobs_command.py   |  53 +
 .../logging-monitoring/check-health.rst|  41 +++
 docs/apache-airflow/start/docker-compose.yaml  |   5 +
 docs/apache-airflow/start/docker.rst   |   2 +-
 tests/cli/commands/test_jobs_command.py| 128 +
 tests/cli/test_cli_parser.py   |   7 +-
 7 files changed, 299 insertions(+), 13 deletions(-)
 create mode 100644 airflow/cli/commands/jobs_command.py
 create mode 100644 tests/cli/commands/test_jobs_command.py

[GitHub] [airflow] potiuk commented on pull request #14531: Running tests in parallel for self-hosted runners

[GitBox]

potiuk commented on pull request #14531:
URL: https://github.com/apache/airflow/pull/14531#issuecomment-787506971


   Hey @ashb - we need bigger machines as I suspected :) . 
   
   The good news is that it will be much cheaper in the long run as we will 
need them for far less time.
   
   The tests are failing but mainly because of memory problems and timeouts (so 
I guess we are simply using too muh , if we up the machine to 64 GB I think 
this should go rather smoothly. The good news is that even with not enough 
memory (and with failures/timeouts) the tests took ~26 m for sqlite - rather 
than > 1h, so when we have enough memory we can achieve the 15 minutes I was 
hoping for. Those 64GB machines are only a bit more expensive than the 32GB 
ones, so we will save a lot of credits when it works. 
   
   We can even optimize it away a bit and have two self-hosted types:
   
   1) Big 64GB ones for the tests
   2) Smaller 32GB ones  for everything else
   
   It should be rather easy to configure in the CI.yml , but I am not sure if 
the auto-scaling solution we have will handle two types?
   
   Here is the job that we have partial successes/failures and it shows how 
those tests will look like. This is actually a good  one to show how the tests 
will look like.  You can see that the output is nicely grouped and you can see 
very clearly the monitoring and progress (it will be much nicer when we have 
more memory because each test will progress much faster). Also I print summary 
of the failed tests at the end - only "failure" outputs are fully printed to 
the logs at the end with "Red" groups - this will make it far easier to analyze 
problems (the same kind of output improvement is in the sequential version of 
the tests run on GitHub runners).
   
   In this  case three test types succeeded (`Heisentests Core Providers`) and 
remaining 5 had some failures (most of them from what I see is due to timeouts, 
which is perfectly understandable if we run out of memory and started to swap 
out to remote SSD in the cloud): 
https://github.com/apache/airflow/pull/14531/checks?check_run_id=1999175947. 
   
   Rationale for bigger machines:
   
   From what I see we have machines with 32GB and since half of it will easily 
be eaten by `tmpfs` when we start writing logs and the like, we only have ~16GB 
which is not enough. During my tests: 
https://twitter.com/higrys/status/1366037359461101569/photo/1 all the tests 
running in parallell took ~35GB of memory on my 64 GB machine (I had just local 
SSD not tmpfs for that but i do not think we need 35GB tmpfs for all logs, 
docker, tmp etc (and we can fine tune that if we do).  
   
   Also it is more important than before to clean-up the tmpfs volumes before 
each run and make them "pristine" for every run - because we will be using 
nearly all of it. I think that will also help with cases like #14505 where some 
left-overs from previous runs are causing the jobs to fail.
   
   ```
 totalusedfree  shared  buff/cache  
 available
 Mem:   30Gi   696Mi24Gi   3.4Gi   5.5Gi
26Gi
 Swap:0B  0B  0B
   
 Filesystem  Size  Used Avail Use% Mounted on
 /dev/root   7.7G  2.6G  5.2G  34% /
 devtmpfs 16G 0   16G   0% /dev
 tmpfs16G 0   16G   0% /dev/shm
 tmpfs   3.1G  804K  3.1G   1% /run
 tmpfs   5.0M 0  5.0M   0% /run/lock
 tmpfs16G 0   16G   0% /sys/fs/cgroup
 tmpfs   3.1G  168K  3.1G   1% /tmp
 tmpfs21G  2.9G   18G  15% /var/lib/docker
 tmpfs16G  534M   15G   4% /home/runner/actions-runner/_work
 /dev/loop0   98M   98M 0 100% /snap/core/10185
 /dev/loop1   56M   56M 0 100% /snap/core18/1885
 /dev/loop2   71M   71M 0 100% /snap/lxd/16922
 /dev/loop3   29M   29M 0 100% /snap/amazon-ssm-agent/2012
   
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] gelinger777 commented on issue #14520: docker-compose install error

[GitBox]

gelinger777 commented on issue #14520:
URL: https://github.com/apache/airflow/issues/14520#issuecomment-787506699


   Thank you @mik-laj 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj closed issue #14520: docker-compose install error

[GitBox]

mik-laj closed issue #14520:
URL: https://github.com/apache/airflow/issues/14520


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on issue #14505: Build failed likely due to unclean state of repo for self-hosted runner

[GitBox]

ashb commented on issue #14505:
URL: https://github.com/apache/airflow/issues/14505#issuecomment-787504549


   Will take a look in detail tomorrow, but permissions are being reset by 
https://github.com/apache/airflow-ci-infra/blob/user-data-WIP/cloud-init.yml#L129.
   
   Wonder what went on here



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (53b9062 -> 9f35cff)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 53b9062  Add pre-commit check to sort and remove duplicates from the 
spelling wordlist (#13170)
 add 9f35cff  Fix asset recompilation message (#14532)

No new revisions were added by this update.

Summary of changes:
 airflow/www/ask_for_recompile_assets_if_needed.sh | 5 -
 scripts/in_container/entrypoint_ci.sh | 4 +++-
 2 files changed, 7 insertions(+), 2 deletions(-)

[GitHub] [airflow] potiuk merged pull request #14532: Fix asset recompilation message

[GitBox]

potiuk merged pull request #14532:
URL: https://github.com/apache/airflow/pull/14532


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #14074: Fixes Issue #14073 - GoogleDisplayVideo360CreateSDFDownloadTaskOperator does not push necessary xcom

[GitBox]

potiuk commented on pull request #14074:
URL: https://github.com/apache/airflow/pull/14074#issuecomment-787501381


   Hey @sarahamilton -> can you please rebase this one? 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org