Re: [PR] Suppress jaydebeapi.Error when setAutoCommit or getAutoCommit is unsupported by JDBC driver [airflow]
dabla commented on PR #38707: URL: https://github.com/apache/airflow/pull/38707#issuecomment-2048968598 > > Yes I saw that and I have same issue I think with MSGraphOperator but I don't fully understand how I can fix this :( > > Simply - You can't use code from `airfliow` in providers until the provider has `apache-airflow>=NEXT_MINOR` - until then the provider must have a "polyfill" - i.e. catch import error and have the same code that is in the provider that providers same functionality when provider is installed on airflow < NEXT_MINOR. With a note to remove it when min-airflow version is >= NEXT_MINOR (we have a policy for bumping - in two weeks we increase min-airflow-version to 2.7.0, so after that any code that was there fore < 2.7 can be removed from providers. > > For SQL providers, a simpler way around it is to add a code to common.sql as a new feature and use `apache-airfow-provider-common-sql >= in `provider.yaml` - but then it should be visibly marked as added in common.sql x.y.z (via since flag in the docuemntation). > > There was a discussion to have `common.util` provider at some point of time but it stalled a bit. Maybe we should come back to it. Ok got it, thank you for the explantion, indeed I saw that discussion of the common.util. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] FIX: Applying airflow queue with spark submit operator is ignored [airflow]
pateash commented on PR #38852: URL: https://github.com/apache/airflow/pull/38852#issuecomment-2048946813 > Tests are failing https://github.com/apache/airflow/actions/runs/8612286825/job/23601405072?pr=38852#step:7:1943 Fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix error when setting try_number from TaskInstancePydantic [airflow]
dstandish commented on code in PR #38907: URL: https://github.com/apache/airflow/pull/38907#discussion_r1560444913 ## airflow/models/taskinstance.py: ## @@ -539,7 +539,11 @@ def _refresh_from_db( task_instance.end_date = ti.end_date task_instance.duration = ti.duration task_instance.state = ti.state -task_instance.try_number = ti._try_number # private attr to get value unaltered by accessor +try: +task_instance.try_number = ti._try_number # private attr to get value unaltered by accessor +except AttributeError: +task_instance.try_number = ti.try_number # the TaskInstancePydantic case Review Comment: this might not be right, gonna mark draft again -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix check of correct dag when remote call for _get_ti [airflow]
dstandish merged PR #38909: URL: https://github.com/apache/airflow/pull/38909 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(airflow) branch main updated (afe8e82531 -> b90fa78e88)
This is an automated email from the ASF dual-hosted git repository. dstandish pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from afe8e82531 Don't wait for DagRun lock in mini scheduler (#38914) add b90fa78e88 Fix check of correct dag when remote call for _get_ti (#38909) No new revisions were added by this update. Summary of changes: airflow/cli/commands/task_command.py | 7 +-- 1 file changed, 5 insertions(+), 2 deletions(-)
Re: [PR] Remove unused attr _try_number on TaskInstancePydantic [airflow]
dstandish closed pull request #38908: Remove unused attr _try_number on TaskInstancePydantic URL: https://github.com/apache/airflow/pull/38908 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Remove unused attr _try_number on TaskInstancePydantic [airflow]
dstandish commented on PR #38908: URL: https://github.com/apache/airflow/pull/38908#issuecomment-2048930761 closing in favor of #38907 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix error when setting try_number from TaskInstancePydantic [airflow]
dstandish commented on code in PR #38907: URL: https://github.com/apache/airflow/pull/38907#discussion_r1560437167 ## airflow/models/taskinstance.py: ## @@ -539,7 +539,11 @@ def _refresh_from_db( task_instance.end_date = ti.end_date task_instance.duration = ti.duration task_instance.state = ti.state -task_instance.try_number = ti._try_number # private attr to get value unaltered by accessor +try: +task_instance.try_number = ti._try_number # private attr to get value unaltered by accessor +except AttributeError: +task_instance.try_number = ti.try_number # the TaskInstancePydantic case Review Comment: ok @uranusjr I updated this PR to make pydantic try_number behavior more like orm model PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix error when setting try_number from TaskInstancePydantic [airflow]
dstandish commented on code in PR #38907: URL: https://github.com/apache/airflow/pull/38907#discussion_r1560430449 ## airflow/models/taskinstance.py: ## @@ -539,7 +539,11 @@ def _refresh_from_db( task_instance.end_date = ti.end_date task_instance.duration = ti.duration task_instance.state = ti.state -task_instance.try_number = ti._try_number # private attr to get value unaltered by accessor +try: +task_instance.try_number = ti._try_number # private attr to get value unaltered by accessor +except AttributeError: +task_instance.try_number = ti.try_number # the TaskInstancePydantic case Review Comment: Yeah i'm actually working on that right now. But I'm on a detour trying to get try number working properly from a logging perspective with db isolation enabled. Try number immutability cannot come soon enough. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix error when setting try_number from TaskInstancePydantic [airflow]
uranusjr commented on code in PR #38907: URL: https://github.com/apache/airflow/pull/38907#discussion_r1560429399 ## airflow/models/taskinstance.py: ## @@ -539,7 +539,11 @@ def _refresh_from_db( task_instance.end_date = ti.end_date task_instance.duration = ti.duration task_instance.state = ti.state -task_instance.try_number = ti._try_number # private attr to get value unaltered by accessor +try: +task_instance.try_number = ti._try_number # private attr to get value unaltered by accessor +except AttributeError: +task_instance.try_number = ti.try_number # the TaskInstancePydantic case Review Comment: Maybe we should make `_try_number` an alias (property) on TaskInstancePydantic instead? This seems too problematic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]
uranusjr commented on code in PR #37948:
URL: https://github.com/apache/airflow/pull/37948#discussion_r1560428293
##
airflow/dag_processing/manager.py:
##
@@ -1029,6 +1051,25 @@ def _collect_results_from_processor(self, processor) ->
None:
)
self._file_stats[processor.file_path] = stat
file_name = Path(processor.file_path).stem
+
+"""crude exposure of instrumentation code which may need to be
furnished"""
+span = Trace.get_tracer("DagFileProcessorManager").start_span(
+"dag_processing", start_time=datetime_to_nano(processor.start_time)
Review Comment:
Is it possible for tracers to just accept datetime objects instead? From
past experience with StatsD, using a raw number as value is prone to user
errors, especially since Airflow does not otherwise use nanoseconds, but either
milliseconds or seconds instead. Having two time units is already one too many,
adding a third one isn’t a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]
uranusjr commented on code in PR #37948:
URL: https://github.com/apache/airflow/pull/37948#discussion_r1560427149
##
airflow/traces/tracer.py:
##
@@ -0,0 +1,280 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import inspect
+import logging
+import socket
+from typing import TYPE_CHECKING, Any, Callable
+
+from airflow.configuration import conf
+from airflow.typing_compat import Protocol
+
+log = logging.getLogger(__name__)
+
+
+def gen_context(trace_id, span_id):
+"""Generate span context from trace_id and span_id."""
+from airflow.traces.otel_tracer import gen_context as otel_gen_context
+
+return otel_gen_context(trace_id, span_id)
+
+
+def gen_links_from_kv_list(list):
+"""Generate links from kv list of {trace_id:int, span_id:int}."""
+from airflow.traces.otel_tracer import gen_links_from_kv_list
+
+return gen_links_from_kv_list(list)
+
+
+def span(func):
+"""Decorate a function with span."""
+
+def wrapper(*args, **kwargs):
+func_name = func.__name__
+qual_name = func.__qualname__
+module_name = func.__module__
+if "." in qual_name:
+component = f"{qual_name.rsplit('.', 1)[0]}"
+else:
+component = module_name
+with Trace.start_span(span_name=func_name, component=component):
+if len(inspect.signature(func).parameters) > 0:
+return func(*args, **kwargs)
+else:
+return func()
+
+return wrapper
+
+
+class EmptyContext:
+"""If no Tracer is configured, EmptyContext is used as a fallback."""
+
+def __init__(self):
+self.trace_id = 1
+
+
+class EmptySpan:
+"""If no Tracer is configured, EmptySpan is used as a fallback."""
+
+def __enter__(self):
+"""Enter."""
+return self
+
+def __exit__(self, *args, **kwargs):
+"""Exit."""
+pass
+
+def __call__(self, obj):
+"""Call."""
+return obj
+
+def get_span_context(self):
+"""Get span context."""
+return EMPTY_CTX
+
+def set_attribute(self, key, value) -> None:
+"""Set an attribute to the span."""
+pass
+
+def set_attributes(self, attributes) -> None:
+"""Set multiple attributes at once."""
+pass
+
+def add_event(
+self,
+name: str,
+attributes: Any | None = None,
+timestamp: int | None = None,
+) -> None:
+"""Add event to span."""
+pass
+
+def add_link(
+self,
+context: Any,
+attributes: Any | None = None,
+) -> None:
+"""Add link to the span."""
+pass
+
+def end(self, end_time=None, *args, **kwargs) -> None:
+"""End."""
+pass
+
+
+EMPTY_SPAN = EmptySpan()
+EMPTY_CTX = EmptyContext()
+
+
+class Tracer(Protocol):
+"""This class is only used for TypeChecking (for IDEs, mypy, etc)."""
+
+instance: Tracer | EmptyTrace | None = None
+
+@classmethod
+def get_tracer(cls, component):
+"""Get a tracer."""
+raise NotImplementedError()
+
+@classmethod
+def start_span(
+cls,
+span_name: str,
+component: str | None = None,
+parent_sc=None,
+span_id=None,
+links=None,
+start_time=None,
+):
+"""Start a span."""
+raise NotImplementedError()
+
+@classmethod
+def use_span(cls, span):
+"""Use a span as current."""
+raise NotImplementedError()
+
+@classmethod
+def get_current_span(self):
+raise NotImplementedError()
+
+@classmethod
+def start_span_from_dagrun(
+cls,
+dagrun,
+span_name=None,
+service_name=None,
+component=None,
+links=None,
+):
+"""Start a span from dagrun."""
+raise NotImplementedError()
+
+@classmethod
+def start_span_from_taskinstance(
+cls,
+ti,
+span_name=None,
+component=None,
+child=False,
+links=None,
+):
+"""Start a span from taskinstance."""
+raise NotImplementedError()
+
+
+class EmptyTrace:
+"""If no Tracer is configured,
Re: [PR] Fix check of correct dag when remote call for _get_ti [airflow]
uranusjr commented on PR #38909: URL: https://github.com/apache/airflow/pull/38909#issuecomment-2048902078 The new check is not complete since another DAG can have a task with the same name, but as an internal interface this is probably good enough. We could add `dag_id` as an attribute, but I’d prefer we just work on making `dag.get_task()` (or something similar) for a serialised DAG instead so we don’t need to pass in the task separately in the first place, but just the DAG and the `task_id` to get the task inside the function. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix built query with implicit cartessian product in `AirflowSecurityManagerV2` [airflow]
uranusjr commented on code in PR #38913: URL: https://github.com/apache/airflow/pull/38913#discussion_r1560404962 ## airflow/www/security_manager.py: ## @@ -200,7 +200,7 @@ def get_dag_id_from_task_instance(resource_pk): return None composite_pk = json.loads(resource_pk) ti = session.scalar( -select(DagRun) +select(TaskInstance) Review Comment: Maybe we should just do `select(TaskInstance.dag_id)` here since we really onlt use the field below? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Allow sort by on all fields in MappedInstances.tsx [airflow]
uranusjr commented on code in PR #38090:
URL: https://github.com/apache/airflow/pull/38090#discussion_r1560397252
##
airflow/api_connexion/endpoints/task_instance_endpoint.py:
##
@@ -239,16 +239,32 @@ def get_mapped_task_instances(
.options(joinedload(TI.rendered_task_instance_fields))
)
-if order_by:
+if order_by is not None:
if order_by == "state":
entry_query = entry_query.order_by(TI.state.asc(),
TI.map_index.asc())
elif order_by == "-state":
entry_query = entry_query.order_by(TI.state.desc(),
TI.map_index.asc())
+elif order_by == "duration":
+print("duration")
+entry_query = entry_query.order_by(TI.duration.asc(),
TI.map_index.asc())
+elif order_by == "-duration":
+print("-duration")
+entry_query = entry_query.order_by(TI.duration.desc(),
TI.map_index.asc())
+elif order_by == "start_date":
+entry_query = entry_query.order_by(TI.start_date.asc(),
TI.map_index.asc())
+elif order_by == "-start_date":
+entry_query = entry_query.order_by(TI.start_date.desc(),
TI.map_index.asc())
+elif order_by == "end_date":
+entry_query = entry_query.order_by(TI.end_date.asc(),
TI.map_index.asc())
+elif order_by == "-end_date":
+entry_query = entry_query.order_by(TI.end_date.desc(),
TI.map_index.asc())
elif order_by == "-map_index":
+print("map index!!")
entry_query = entry_query.order_by(TI.map_index.desc())
else:
raise BadRequest(detail=f"Ordering with '{order_by}' is not
supported")
else:
+print("map inde ascx!!")
entry_query = entry_query.order_by(TI.map_index.asc())
Review Comment:
We can flatten this to
```python
if order is None:
entry_query = entry_query.order_by(TI.map_index.asc())
elif order_by == "state":
entry_query = entry_query.order_by(TI.state.asc(), TI.map_index.asc())
... # More...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Bugfix: Move rendering of `map_index_template` so it renders for failed tasks as long as it was defined before the point of failure [airflow]
uranusjr commented on code in PR #38902: URL: https://github.com/apache/airflow/pull/38902#discussion_r1560385357 ## airflow/models/taskinstance.py: ## @@ -2715,29 +2727,26 @@ def signal_handler(signum, frame): # Execute the task with set_current_context(context): -result = self._execute_task(context, task_orig) +result, rendered_map_index = self._execute_task(context, task_orig, jinja_env=jinja_env) Review Comment: I wonder if we should just try to render the template _regardless_ of the task at all. Something like ```python def _render_map_index(...): ... try: result = self._execute_task(context, task_orig) except Exception: # If the task failed, swallow rendering error so it doesn't mask the main error. with contextlib.suppress(jinja2.TemplateSyntaxError, jinja2.UndefinedError): _render_map_index(...) raise else: # If the task succeeded, render normally to let rendering error bubble up. _render_map_index(...) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(airflow) branch main updated: Don't wait for DagRun lock in mini scheduler (#38914)
This is an automated email from the ASF dual-hosted git repository. dstandish pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new afe8e82531 Don't wait for DagRun lock in mini scheduler (#38914) afe8e82531 is described below commit afe8e82531cece4f3aeaea465b985ebd7257e71d Author: Daniel Standish <15932138+dstand...@users.noreply.github.com> AuthorDate: Wed Apr 10 20:06:37 2024 -0700 Don't wait for DagRun lock in mini scheduler (#38914) We should just bail if something else is already "minischeduling". We already catch OperationalError so there's nothing else we need to change for this. Sometimes (e.g. with certain task mapping scenarios) many tasks from same dag run are trying to "minischedule" at the same time. If something else is already locking, it's probably better to just move on and not wait. --- airflow/models/taskinstance.py | 1 + 1 file changed, 1 insertion(+) diff --git a/airflow/models/taskinstance.py b/airflow/models/taskinstance.py index a55ea0fe77..c3bf25e343 100644 --- a/airflow/models/taskinstance.py +++ b/airflow/models/taskinstance.py @@ -3492,6 +3492,7 @@ class TaskInstance(Base, LoggingMixin): run_id=ti.run_id, ), session=session, +nowait=True, ).one() task = ti.task
Re: [PR] Don't wait for lock on DagRun in mini scheduler [airflow]
dstandish merged PR #38914: URL: https://github.com/apache/airflow/pull/38914 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] FIX bug duplicated add base_folder to oss_remote_log_location in oss_wri… [airflow]
boring-cyborg[bot] commented on PR #38917: URL: https://github.com/apache/airflow/pull/38917#issuecomment-2048851415 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst) Here are some useful points: - Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices). - Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits. Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://s.apache.org/airflow-slack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] FIX bug duplicated add base_folder to oss_remote_log_location in oss_wri… [airflow]
hh-cn opened a new pull request, #38917: URL: https://github.com/apache/airflow/pull/38917 As title said -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Don't wait for lock on DagRun in mini scheduler [airflow]
ashb commented on PR #38914: URL: https://github.com/apache/airflow/pull/38914#issuecomment-2048845639 This might have the inverse effect of putting more load on the scheduler as it might end up doing more task expansion -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(airflow) branch main updated: Add retry logic for RPC calls (#38910)
This is an automated email from the ASF dual-hosted git repository.
dstandish pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new cdc7f19b57 Add retry logic for RPC calls (#38910)
cdc7f19b57 is described below
commit cdc7f19b571a99cbbad5091dcb11e2d4f1439fb3
Author: Daniel Standish <15932138+dstand...@users.noreply.github.com>
AuthorDate: Wed Apr 10 19:47:20 2024 -0700
Add retry logic for RPC calls (#38910)
I have found that when RPC server restarts it can take 30-60s for the
server to be able to respond to RPC calls. This implements exponential wait for
that case. 10 might seem excessive but I found that 3 or 5 didn't always do the
trick.
---
airflow/api_internal/internal_api_call.py | 14 +-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/airflow/api_internal/internal_api_call.py
b/airflow/api_internal/internal_api_call.py
index 8dcd5dba30..c3a67d03ee 100644
--- a/airflow/api_internal/internal_api_call.py
+++ b/airflow/api_internal/internal_api_call.py
@@ -19,10 +19,13 @@ from __future__ import annotations
import inspect
import json
+import logging
from functools import wraps
from typing import Callable, TypeVar
import requests
+import tenacity
+from urllib3.exceptions import NewConnectionError
from airflow.configuration import conf
from airflow.exceptions import AirflowConfigException, AirflowException
@@ -32,6 +35,8 @@ from airflow.typing_compat import ParamSpec
PS = ParamSpec("PS")
RT = TypeVar("RT")
+logger = logging.getLogger(__name__)
+
class InternalApiConfig:
"""Stores and caches configuration for Internal API."""
@@ -96,7 +101,14 @@ def internal_api_call(func: Callable[PS, RT]) ->
Callable[PS, RT]:
headers = {
"Content-Type": "application/json",
}
-
+from requests.exceptions import ConnectionError
+
+@tenacity.retry(
+stop=tenacity.stop_after_attempt(10),
+wait=tenacity.wait_exponential(min=1),
+retry=tenacity.retry_if_exception_type((NewConnectionError,
ConnectionError)),
+before_sleep=tenacity.before_log(logger, logging.WARNING),
+)
def make_jsonrpc_request(method_name: str, params_json: str) -> bytes:
data = {"jsonrpc": "2.0", "method": method_name, "params": params_json}
internal_api_endpoint = InternalApiConfig.get_internal_api_endpoint()
Re: [PR] Add retry logic for RPC calls [airflow]
dstandish merged PR #38910: URL: https://github.com/apache/airflow/pull/38910 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Retry if failed from queued should be separate from try_number [airflow]
Bowrna commented on issue #38304: URL: https://github.com/apache/airflow/issues/38304#issuecomment-2048824149 The retry logic handled here in taskinstance.py. It is failed and therefore checks if its eligible for retry, if yes its queued again. But having another logic like FAILED_IN_QUEUE and TRY_NUMBER_FOR_QUEUE to handle the queue failed task makes sense to me for now. If you see other way, please let me know. https://github.com/apache/airflow/blob/b6ff085679c283cd3ccc3edf20dd3e6b0eaec967/airflow/models/taskinstance.py#L2992-L3015 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Fix bug in LivyOperator when its trigger times out [airflow]
mateuslatrova opened a new pull request, #38916: URL: https://github.com/apache/airflow/pull/38916 ### Related issue closes: https://github.com/apache/airflow/issues/37898 ### Explain When a LivyOperator was instantiated with deferrable=True and its batch job ran for more time than the set execution_timeout, airflow would detect this timeout and would cancel the trigger and then try to kill the task with the 'on_kill' method. But that would fail raising an AttributeError because the batch_id attribute wouldn't be defined in the constructor method. From now on, the LivyTrigger will timeout itself before airflow does it, and it will send an event to the LivyOperator signaling that a timeout happened. This way, the operator can stop the running Livy batch job, and can fail the task instance gracefully. --- -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix bug in LivyOperator when its trigger times out [airflow]
boring-cyborg[bot] commented on PR #38916: URL: https://github.com/apache/airflow/pull/38916#issuecomment-2048741411 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst) Here are some useful points: - Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices). - Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits. Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://s.apache.org/airflow-slack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] 401 UNAUTHORIZED when using AUTH_ROLE_PUBLIC = "Admin" [airflow]
Lee-W commented on issue #38900: URL: https://github.com/apache/airflow/issues/38900#issuecomment-2048640534 Sure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add dataset event dataset dag run queue association [airflow]
github-actions[bot] closed pull request #37424: Add dataset event dataset dag run queue association URL: https://github.com/apache/airflow/pull/37424 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Updated logic to allow AWS Batch Hook `get_job_description` retries to be more effective [airflow]
github-actions[bot] closed pull request #37552: Updated logic to allow AWS Batch Hook `get_job_description` retries to be more effective URL: https://github.com/apache/airflow/pull/37552 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] WIP Feature/add bq dataset export operator [airflow]
github-actions[bot] commented on PR #37695: URL: https://github.com/apache/airflow/pull/37695#issuecomment-2048630173 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add four unit tests for aws/utils [airflow]
slycyberguy commented on PR #38820: URL: https://github.com/apache/airflow/pull/38820#issuecomment-2048617564 > You will need to remove these lines from the project structure tests: > > https://github.com/apache/airflow/blob/main/tests/always/test_project_structure.py#L85-L86 I took care of it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] add templated field support for extra containers [airflow]
mg-aik commented on PR #38510: URL: https://github.com/apache/airflow/pull/38510#issuecomment-2048575212 Finally, I have been waiting for this one! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Status of testing Providers that were prepared on April 10, 2024 [airflow]
stiak commented on issue #38904: URL: https://github.com/apache/airflow/issues/38904#issuecomment-2048558292 #37736 is good.  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Fix `update-common-sql-api-stubs` pre-commit check [airflow]
Taragolis opened a new pull request, #38915:
URL: https://github.com/apache/airflow/pull/38915
This pre-commit check doesn't run due to wrong path and seems like it
doesn't run for a long time.
I'm not sure that do we need this stub files, since provider drop support of
Airflow 2.4 for a long time, and there is no any subclassing resolution
required but this more for the future discussion.
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] Status of testing Providers that were prepared on April 10, 2024 [airflow]
jliu0812 commented on issue #38904: URL: https://github.com/apache/airflow/issues/38904#issuecomment-2048542154 [#38022](https://github.com/apache/airflow/pull/38022) is good. ✅ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]
mateuslatrova commented on PR #38628: URL: https://github.com/apache/airflow/pull/38628#issuecomment-2048507160 Thanks for the idea, @potiuk! That makes much more sense. I will try to implement it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Change capture warnings output format to the JSON [airflow]
Taragolis commented on PR #38905: URL: https://github.com/apache/airflow/pull/38905#issuecomment-2048455211 Also add about configuration into the documentation, so worthwhile to check spelling and grammar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix built query with implicit cartessian product in `AirflowSecurityManagerV2` [airflow]
Taragolis closed pull request #38913: Fix built query with implicit cartessian product in `AirflowSecurityManagerV2` URL: https://github.com/apache/airflow/pull/38913 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Fix built query with implicit cartessian product in `AirflowSecurityManagerV2` [airflow]
Taragolis opened a new pull request, #38913:
URL: https://github.com/apache/airflow/pull/38913
Found during analyse warnings from the
https://github.com/apache/airflow/pull/38905
In addition raise an error if this warning detected into the Airflow during
tests run, same as proposed into the
https://github.com/apache/airflow/pull/28590, however it wasn't work before
during the previous capture system which ignore all errors
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Amazon Bedrock - Model Throughput Provisioning [airflow]
ferruzzi commented on PR #38850: URL: https://github.com/apache/airflow/pull/38850#issuecomment-2048444921 Gah, static check is just a merge issue, imported the sensor twice. Easy fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Add logic to handle on_kill for BigQueryInsertJobOperator when deferrable=True [airflow]
sunank200 opened a new pull request, #38912:
URL: https://github.com/apache/airflow/pull/38912
- Add logic to handle on_kill for BigQueryInsertJobOperator when
`deferrable=True`
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] Status of testing Providers that were prepared on April 10, 2024 [airflow]
eladkal commented on issue #38904: URL: https://github.com/apache/airflow/issues/38904#issuecomment-2048383457 > Just figure out that we accidentally exclude #37458 from the changelog. We could add it after the release, because changes itself presented into the released package. Its a mistake in the release process. But its more than just change log. This PR is a feature while the rc1 is a bug fix release thus I will exclude yandex from this wave and cut rc2 afterwards -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Status of testing Providers that were prepared on April 10, 2024 [airflow]
ellisms commented on issue #38904: URL: https://github.com/apache/airflow/issues/38904#issuecomment-2048377708 #37886 is good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(airflow) branch improve-warning-system updated (47322f1722 -> 35b4fd406b)
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch improve-warning-system in repository https://gitbox.apache.org/repos/asf/airflow.git discard 47322f1722 Change capture warnings output format to the JSON add 35b4fd406b Change capture warnings output format to the JSON This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (47322f1722) \ N -- N -- N refs/heads/improve-warning-system (35b4fd406b) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: contributing-docs/testing/unit_tests.rst | 25 + tests/models/test_taskinstance.py| 12 +--- 2 files changed, 34 insertions(+), 3 deletions(-)
Re: [PR] Change capture warnings output format to the JSON [airflow]
Taragolis commented on PR #38905: URL: https://github.com/apache/airflow/pull/38905#issuecomment-2048332619 Test `tests/ti_deps/deps/test_mapped_task_upstream_dep.py::test_step_by_step` is so annoying, it failed into the CI on the regular basis in the different PR and I can't still reproduce it locally -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add support for role arn for aws creds in storage transfer job operators [airflow]
boring-cyborg[bot] commented on PR #38911: URL: https://github.com/apache/airflow/pull/38911#issuecomment-2048325925 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst) Here are some useful points: - Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices). - Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits. Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://s.apache.org/airflow-slack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Add support for role arn for aws creds in storage transfer job operators [airflow]
wlinamchurch opened a new pull request, #38911:
URL: https://github.com/apache/airflow/pull/38911
closes: https://github.com/apache/airflow/issues/35271
related: Attempt to fix tests and checks from
https://github.com/apache/airflow/pull/35292
This PR adds support to use role arn's for AWS credentials in storage
transfer jobs. All of the substantive code changes were made in
https://github.com/apache/airflow/pull/35292.
My additional contributions were to get the tests to pass and pre-commit
checks to pass.
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]
dstandish commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2048324458 Yeah @howardyoo I think it's a good practice and will result in fewer errors. Big bang PRs, while sometimes necessary, have a tendency for things to fall between the cracks, bugs and things to go unnoticed. Let me share what I have been doing over the last month or so with my work on AIP-44 which I think has worked pretty well for all parties. 1. "get it sorta working" 2. do a soft reset to main 3. Look at your local changes, and identify small changes that make sense as a single unit. Commit that, with a good name for the commit message. 4. Step through and repeat (3) till all your local changes are recommitted. 5. Copy the output of `git log --oneline` to text editor 6. Manipulate the lines to be ``` git checkout main git checkout -b git cherry-pick ``` So e.g. each line I do a replace of ` ` with `-` and make lowercase then I can use multiline editing to quickly convert to that format. So then e.g. ``` 5db845e493 Do not log event when using db isolation (4 hours ago) eb4117c50f Fix error when setting try_number from TaskInstancePydantic (4 hours ago) a1d4eb0362 Remove unused attr _try_number on TaskInstancePydantic (4 hours ago) 19dd3f2277 Fix check of correct dag when remote call for _get_ti (4 hours ago) 4c6255b0c9 Add retry logic for RPC calls (4 hours ago) ``` becomes ``` gco main git checkout -b do-not-log-event-when-using-db-isolation git cherry-pick 5db845e493 gpsup gco main git checkout -b fix-error-when-setting-try_number-from-taskinstancepydantic git cherry-pick eb4117c50f gpsup gco main git checkout -b remove-unused-attr-_try_number-on-taskinstancepydantic git cherry-pick a1d4eb0362 gpsup gco main git checkout -b fix-check-of-correct-dag-when-remote-call-for-_get_ti git cherry-pick 19dd3f2277 gpsup gco main git checkout -b add-retry-logic-for-rpc-calls git cherry-pick 4c6255b0c9 gpsup ``` (gpsup is `git push --set-upstream origin $(git_current_branch)`) It's less painful than I thought it might be originally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Add retry logic for RPC calls [airflow]
dstandish opened a new pull request, #38910: URL: https://github.com/apache/airflow/pull/38910 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Remove unused attr _try_number on TaskInstancePydantic [airflow]
dstandish opened a new pull request, #38908: URL: https://github.com/apache/airflow/pull/38908 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Fix error when setting try_number from TaskInstancePydantic [airflow]
dstandish opened a new pull request, #38907: URL: https://github.com/apache/airflow/pull/38907 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Do not log event when using db isolation [airflow]
dstandish opened a new pull request, #38906: URL: https://github.com/apache/airflow/pull/38906 Eventually we need to fix this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Status of testing Providers that were prepared on April 10, 2024 [airflow]
gschuurman commented on issue #38904: URL: https://github.com/apache/airflow/issues/38904#issuecomment-2048281921 #38872 login is working as expected again with both email set and unset -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add option for ExternalTaskSensor to automatically infer upstream execution dates [airflow]
jannisko commented on PR #38542: URL: https://github.com/apache/airflow/pull/38542#issuecomment-2048276970 Fair point. Thank you for the hint. @hussein-awala @uranusjr I see you two have been touching the sensor every once in a while. Do you mind taking a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Change capture warnings output format to the JSON [airflow]
Taragolis closed pull request #38905: Change capture warnings output format to the JSON URL: https://github.com/apache/airflow/pull/38905 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Change capture warnings output format to the JSON [airflow]
mergeable[bot] commented on PR #38905:
URL: https://github.com/apache/airflow/pull/38905#issuecomment-2048218996
:x: Error Occurred while executing an Action
If you believe this is an unexpected error, please report it on our issue
tracker: https://github.com/mergeability/mergeable/issues/new
# Error Details
HttpError: Server Error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Change capture warnings output format to the JSON [airflow]
mergeable[bot] commented on PR #38905:
URL: https://github.com/apache/airflow/pull/38905#issuecomment-2048215410
:x: Error Occurred while executing an Action
If you believe this is an unexpected error, please report it on our issue
tracker: https://github.com/mergeability/mergeable/issues/new
# Error Details
HttpError: Server Error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] Status of testing Providers that were prepared on April 10, 2024 [airflow]
Taragolis commented on issue #38904: URL: https://github.com/apache/airflow/issues/38904#issuecomment-2048213174 Just figure out that we accidentally exclude https://github.com/apache/airflow/pull/37458 from the changelog. We could add it after the release, because changes itself presented into the released package. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Change capture warnings output format to the JSON [airflow]
Taragolis opened a new pull request, #38905:
URL: https://github.com/apache/airflow/pull/38905
This PR is a result of complete rework on warning capture system:
- Pass parameters from the pytest configurations
- Trim path to the project root or sites
- Keep information about original warning and in which test suite it
originally raised
- Avoid to use internal pytests classes
- Output format is JSON rather than pylint (we do not use it anyway).
Drawbacks is CI warnings summary might show greater number, it could be fixed
as a follow up by implement simple summarised python script.
- Output path now relative to the tests directory, but default it stored
into the `tests/warnings.txt` regardless of the current working directory
The only things the same with
[pytest-capture-warnings](https://github.com/athinkingape/pytest-capture-warnings)
is an idea, so I remove license headers, if some one have any objections feel
free to ping me to return it.
**Output before**
```console
tests/core/test_configuration.py:1411: [W0513(warning), ] The
sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option
in [database] - the old setting has been used, but please update your config.
tests/core/test_core.py:107: [W0513(warning), ] SELECT statement has a
cartesian product between FROM element(s) "task_fail" and FROM element
"dag_run". Apply join condition(s) between each element to resolve.
tests/core/test_logging_config.py:267: [W0513(warning), ] The
`airflow.utils.log.wasb_task_handler.WasbTaskHandler` class is deprecated.
Please use
`'airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler'`.
```
**Output after**
```console
{"category": "DeprecationWarning", "message": "The sql_alchemy_conn option
in [core] has been moved to the sql_alchemy_conn option in [database] - the old
setting has been used, but please update your config.", "node_id":
"tests/core/test_configuration.py::TestDeprecatedConf::test_conf_as_dict_when_deprecated_value_in_secrets",
"filename": "tests/core/test_configuration.py", "lineno": 1411, "count": 1}
{"category": "sqlalchemy.exc.SAWarning", "message": "SELECT statement has a
cartesian product between FROM element(s) \"task_fail\" and FROM element
\"dag_run\". Apply join condition(s) between each element to resolve.",
"node_id": "tests/core/test_core.py::TestCore::test_task_fail_duration",
"filename": "tests/core/test_core.py", "lineno": 107, "count": 1}
{"category": "DeprecationWarning", "message": "The
`airflow.utils.log.wasb_task_handler.WasbTaskHandler` class is deprecated.
Please use
`'airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler'`.",
"node_id":
"tests/core/test_logging_config.py::TestLoggingSettings::test_loading_remote_logging_with_wasb_handler",
"filename": "tests/core/test_logging_config.py", "lineno": 267, "count": 1}
```
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] Status of testing Providers that were prepared on April 10, 2024 [airflow]
pdebelak commented on issue #38904: URL: https://github.com/apache/airflow/issues/38904#issuecomment-2048195024 I can confirm that my change to cncf.kubernetes is working as expected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(airflow) 01/01: Change capture warnings output format to the JSON
This is an automated email from the ASF dual-hosted git repository.
taragolis pushed a commit to branch improve-warning-system
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 47322f1722798509ca2ffcadd83ff73dc244cd18
Author: Andrey Anshin
AuthorDate: Mon Apr 8 16:18:02 2024 +0400
Change capture warnings output format to the JSON
---
tests/conftest.py | 243 +++---
tests/models/test_taskinstance.py | 12 +-
2 files changed, 126 insertions(+), 129 deletions(-)
diff --git a/tests/conftest.py b/tests/conftest.py
index 6d102e7268..b38a512116 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -21,13 +21,14 @@ import json
import os
import platform
import re
+import site
import subprocess
import sys
import warnings
-from contextlib import ExitStack, suppress
+from contextlib import ExitStack, contextmanager, suppress
from datetime import datetime, timedelta, timezone
from pathlib import Path
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, NamedTuple
import pytest
import time_machine
@@ -68,8 +69,6 @@ for env_key in os.environ.copy():
if not (ko := _KEEP_CONFIGS.get(section)) or not ("*" in ko or option
in ko):
del os.environ[env_key]
-DEFAULT_WARNING_OUTPUT_PATH = Path("warnings.txt")
-warning_output_path = DEFAULT_WARNING_OUTPUT_PATH
SUPPORTED_DB_BACKENDS = ("sqlite", "postgres", "mysql")
# A bit of a Hack - but we need to check args before they are parsed by pytest
in order to
@@ -295,11 +294,22 @@ def pytest_addoption(parser):
dest="db_cleanup",
help="Disable DB clear before each test module.",
)
+group.addoption(
+"--disable-capture-warnings",
+action="store_true",
+dest="disable_capture_warnings",
+help="Disable internal capture warnings.",
+)
group.addoption(
"--warning-output-path",
action="store",
dest="warning_output_path",
-default=DEFAULT_WARNING_OUTPUT_PATH.resolve().as_posix(),
+metavar="PATH",
+help=(
+"Path for resulting captured warnings. Absolute or relative to the
`tests` directory. "
+"If not provided or environment variable `CAPTURE_WARNINGS_OUTPUT`
not set "
+"then 'warnings.txt' will be used."
+),
)
@@ -415,7 +425,25 @@ def pytest_configure(config: pytest.Config) -> None:
config.addinivalue_line("markers", "enable_redact: do not mock redact
secret masker")
os.environ["_AIRFLOW__SKIP_DATABASE_EXECUTOR_COMPATIBILITY_CHECK"] = "1"
-configure_warning_output(config)
+
+# Setup capture warnings
+if not (warning_output_path := config.getoption("warning_output_path",
default=None)):
+if not (warning_output_path :=
os.environ.get("CAPTURE_WARNINGS_OUTPUT")):
+warning_output_path = "warnings.txt"
+warning_output_path = Path(warning_output_path)
+if not warning_output_path.is_absolute():
+warning_output_path =
Path(__file__).resolve().parent.joinpath(warning_output_path)
+try:
+warning_output_path.parent.resolve(strict=True)
+except Exception as ex:
+msg = (
+f"Unable resolve {os.fspath(warning_output_path.parent)!r} path
for store warnings outputs. "
+f"Original Error:\n {type(ex).__name__}: {ex}."
+)
+pytest.exit(msg, returncode=6)
+config.option.warning_output_path = warning_output_path
+if "ignore" in sys.warnoptions:
+config.option.disable_capture_warnings = True
def pytest_unconfigure(config):
@@ -1264,139 +1292,114 @@ def _disable_redact(request: pytest.FixtureRequest,
mocker):
return
-# The code below is a modified version of capture-warning code from
-# https://github.com/athinkingape/pytest-capture-warnings
+@functools.lru_cache(maxsize=None)
+def _sites_locations() -> tuple[str, ...]:
+return tuple([*site.getsitepackages(), site.getusersitepackages()])
-# MIT License
-#
-# Portions Copyright (c) 2022 A Thinking Ape Entertainment Ltd.
-# Portions Copyright (c) 2022 Pyschojoker (Github)
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
all
-# copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR
(airflow) branch improve-warning-system created (now 47322f1722)
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch improve-warning-system in repository https://gitbox.apache.org/repos/asf/airflow.git at 47322f1722 Change capture warnings output format to the JSON This branch includes the following new commits: new 47322f1722 Change capture warnings output format to the JSON The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
Re: [PR] Amazon Bedrock - Model Throughput Provisioning [airflow]
ferruzzi commented on PR #38850: URL: https://github.com/apache/airflow/pull/38850#issuecomment-2048127209 Dependency merged; I'll get this rebased on top of that one and merge it this afternoon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Amazon Bedrock - Fix system test [airflow]
ferruzzi merged PR #38887: URL: https://github.com/apache/airflow/pull/38887 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(airflow) branch main updated: Amazon Bedrock - Fix system test (#38887)
This is an automated email from the ASF dual-hosted git repository.
ferruzzi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new b6ff085679 Amazon Bedrock - Fix system test (#38887)
b6ff085679 is described below
commit b6ff085679c283cd3ccc3edf20dd3e6b0eaec967
Author: D. Ferruzzi
AuthorDate: Wed Apr 10 10:40:34 2024 -0700
Amazon Bedrock - Fix system test (#38887)
---
.../system/providers/amazon/aws/example_bedrock.py | 92 +++---
1 file changed, 46 insertions(+), 46 deletions(-)
diff --git a/tests/system/providers/amazon/aws/example_bedrock.py
b/tests/system/providers/amazon/aws/example_bedrock.py
index 12e2461547..e25bbb8ed7 100644
--- a/tests/system/providers/amazon/aws/example_bedrock.py
+++ b/tests/system/providers/amazon/aws/example_bedrock.py
@@ -18,12 +18,12 @@ from __future__ import annotations
import json
from datetime import datetime
+from os import environ
-from botocore.exceptions import ClientError
-
-from airflow.decorators import task
+from airflow.decorators import task, task_group
from airflow.models.baseoperator import chain
from airflow.models.dag import DAG
+from airflow.operators.empty import EmptyOperator
from airflow.providers.amazon.aws.hooks.bedrock import BedrockHook
from airflow.providers.amazon.aws.operators.bedrock import (
BedrockCustomizeModelOperator,
@@ -35,6 +35,7 @@ from airflow.providers.amazon.aws.operators.s3 import (
S3DeleteBucketOperator,
)
from airflow.providers.amazon.aws.sensors.bedrock import
BedrockCustomizeModelCompletedSensor
+from airflow.utils.edgemodifier import Label
from airflow.utils.trigger_rule import TriggerRule
from tests.system.providers.amazon.aws.utils import SystemTestContextBuilder
@@ -44,10 +45,10 @@ sys_test_context_task =
SystemTestContextBuilder().add_variable(ROLE_ARN_KEY).bu
DAG_ID = "example_bedrock"
-# Creating a custom model takes nearly two hours. If SKIP_LONG_TASKS is True
then set
-# the trigger rule to an improbable state. This way we can still have the
code snippets
-# for docs, and we can manually run the full tests occasionally.
-SKIP_LONG_TASKS = True
+# Creating a custom model takes nearly two hours. If SKIP_LONG_TASKS
+# is True then these tasks will be skipped. This way we can still have
+# the code snippets for docs, and we can manually run the full tests.
+SKIP_LONG_TASKS = environ.get("SKIP_LONG_SYSTEM_TEST_TASKS", default=True)
LLAMA_MODEL_ID = "meta.llama2-13b-chat-v1"
PROMPT = "What color is an orange?"
@@ -61,15 +62,41 @@ HYPERPARAMETERS = {
}
-@task
-def delete_custom_model(model_name: str):
-try:
-BedrockHook().conn.delete_custom_model(modelIdentifier=model_name)
-except ClientError as e:
-if SKIP_LONG_TASKS and (e.response["Error"]["Code"] ==
"ValidationException"):
-# There is no model to delete. Since we skipped making one,
that's fine.
-return
-raise e
+@task_group
+def customize_model_workflow():
+# [START howto_operator_customize_model]
+customize_model = BedrockCustomizeModelOperator(
+task_id="customize_model",
+job_name=custom_model_job_name,
+custom_model_name=custom_model_name,
+role_arn=test_context[ROLE_ARN_KEY],
+
base_model_id=f"arn:aws:bedrock:us-east-1::foundation-model/{TITAN_MODEL_ID}",
+hyperparameters=HYPERPARAMETERS,
+training_data_uri=training_data_uri,
+output_data_uri=f"s3://{bucket_name}/myOutputData",
+)
+# [END howto_operator_customize_model]
+
+# [START howto_sensor_customize_model]
+await_custom_model_job = BedrockCustomizeModelCompletedSensor(
+task_id="await_custom_model_job",
+job_name=custom_model_job_name,
+)
+# [END howto_sensor_customize_model]
+
+@task
+def delete_custom_model():
+
BedrockHook().conn.delete_custom_model(modelIdentifier=custom_model_name)
+
+@task.branch
+def run_or_skip():
+return end_workflow.task_id if SKIP_LONG_TASKS else
customize_model.task_id
+
+run_or_skip = run_or_skip()
+end_workflow = EmptyOperator(task_id="end_workflow",
trigger_rule=TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS)
+
+chain(run_or_skip, Label("Long-running tasks skipped"), end_workflow)
+chain(run_or_skip, customize_model, await_custom_model_job,
delete_custom_model(), end_workflow)
with DAG(
@@ -95,7 +122,7 @@ with DAG(
upload_training_data = S3CreateObjectOperator(
task_id="upload_data",
s3_bucket=bucket_name,
-s3_key=training_data_uri,
+s3_key=input_data_s3_key,
data=json.dumps(TRAIN_DATA),
)
@@ -115,30 +142,6 @@ with DAG(
)
# [END howto_operator_invoke_titan_model]
-# [START howto_operator_customize_model]
-customize_model = BedrockCustomizeModelOperator(
-task_id="customize_model",
-
Re: [PR] Always use the executemany method when inserting rows in DbApiHook as it's way much faster [airflow]
uranusjr commented on code in PR #38715: URL: https://github.com/apache/airflow/pull/38715#discussion_r1559820105 ## airflow/providers/common/sql/hooks/sql.py: ## @@ -147,14 +147,23 @@ class DbApiHook(BaseHook): default_conn_name = "default_conn_id" # Override if this db supports autocommit. supports_autocommit = False +# Override if this db supports executemany. +supports_executemany = False # Override with the object that exposes the connect method connector: ConnectorProtocol | None = None # Override with db-specific query to check connection _test_connection_sql = "select 1" # Default SQL placeholder _placeholder: str = "%s" -def __init__(self, *args, schema: str | None = None, log_sql: bool = True, **kwargs): +def __init__( +self, +*args, +schema: str | None = None, +log_sql: bool = True, +fast_executemany: bool = False, Review Comment: Nothing seems to use this argument anywhere? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Suppress jaydebeapi.Error when setAutoCommit or getAutoCommit is unsupported by JDBC driver [airflow]
potiuk commented on PR #38707: URL: https://github.com/apache/airflow/pull/38707#issuecomment-2048066159 > Yes I saw that and I have same issue I think with MSGraphOperator but I don't fully understand how I can fix this :( Simply - You can't use code from `airfliow` in providers until the provider has `apache-airflow>=NEXT_MINOR` - until then the provider must have a "polyfill" - i.e. catch import error and have the same code that is in the provider that providers same functionality when provider is installed on airflow < NEXT_MINOR. With a note to remove it when min-airflow version is >= NEXT_MINOR (we have a policy for bumping - in two weeks we increase min-airflow-version to 2.7.0, so after that any code that was there fore < 2.7 can be removed from providers. For SQL providers, a simpler way around it is to add a code to common.sql as a new feature and use `apache-airfow-provider-common-sql >= in `provider.yaml` - but then it should be visibly marked as added in common.sql x.y.z (via since flag in the docuemntation). There was a discussion to have `common.util` provider at some point of time but it stalled a bit. Maybe we should come back to it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Suppress jaydebeapi.Error when setAutoCommit or getAutoCommit is unsupported by JDBC driver [airflow]
dabla commented on PR #38707: URL: https://github.com/apache/airflow/pull/38707#issuecomment-2048041234 > There is a problem with back-compatibility. Our providers work for `Airflow >= 2.6.0` - see the errrors raised in tests - it cannot be imported from Airlfow until we keep 2.10.0 support - you will need to duplicate it in the provider and fall-back to it and mark it as "remove after min-airflow version is set to 2.10. > > Another option (probably better) is to add it to common.sql and use from there (and add common.sql >= NEXT MINOR VERSION in provider.yaml. Yes I saw that and I have same issue I think with MSGraphOperator but I don't fully understand how I can fix this :( -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Update _index.md [airflow-site]
dabla closed pull request #933: Update _index.md URL: https://github.com/apache/airflow-site/pull/933 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Update _index.md [airflow-site]
dabla commented on PR #933: URL: https://github.com/apache/airflow-site/pull/933#issuecomment-2048037537 > I think we can close this PR in favor of [apache/airflow#38111](https://github.com/apache/airflow/pull/38111) ? Hey Elad yes this one can be closed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Replace dill package to use cloudpickle [airflow]
potiuk commented on code in PR #38531: URL: https://github.com/apache/airflow/pull/38531#discussion_r1559780893 ## airflow/models/taskinstance.py: ## @@ -1287,7 +1287,7 @@ class TaskInstance(Base, LoggingMixin): queued_dttm = Column(UtcDateTime) queued_by_job_id = Column(Integer) pid = Column(Integer) -executor_config = Column(ExecutorConfigType(pickler=dill)) +executor_config = Column(ExecutorConfigType(pickler=cloudpickle)) Review Comment: Or we could use airflow's `serde` - providing that it serializes K8S objects (or add this option). That would make it it independent and simpler "only one" serializers - and we could add migration to it, this way we could get rid of either cloudpickle or dill as "hard" dependency. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] Status of testing Providers that were prepared on April 10, 2024 [airflow]
eladkal opened a new issue, #38904: URL: https://github.com/apache/airflow/issues/38904 ### Body I have a kind request for all the contributors to the latest provider packages release. Could you please help us to test the RC versions of the providers? The guidelines on how to test providers can be found in [Verify providers by contributors](https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors) Let us know in the comment, whether the issue is addressed. Those are providers that require testing as there were some substantial changes introduced: ## Provider [airbyte: 3.7.0rc1](https://pypi.org/project/apache-airflow-providers-airbyte/3.7.0rc1) - [ ] [Ensure Airbyte Provider is Compatible with Cloud and Config APIs (#37943)](https://github.com/apache/airflow/pull/37943): @chrishronek Linked issues: - [ ] [Linked Issue #35011](https://github.com/apache/airflow/issues/35011): @vytautasnord - [ ] [fix: try002 for provider airbyte (#38786)](https://github.com/apache/airflow/pull/38786): @dondaum Linked issues: - [ ] [Linked Issue #38761](https://github.com/apache/airflow/issues/38761): @Taragolis - [ ] [add "return" statement to "yield" within a while loop in airbyte triggers (#38390)](https://github.com/apache/airflow/pull/38390): @Lee-W ## Provider [alibaba: 2.7.3rc1](https://pypi.org/project/apache-airflow-providers-alibaba/2.7.3rc1) - [ ] [Avoid use of `assert` outside of the tests (#37718)](https://github.com/apache/airflow/pull/37718): @Taragolis ## Provider [amazon: 8.20.0rc1](https://pypi.org/project/apache-airflow-providers-amazon/8.20.0rc1) - [ ] [AWS Batch Executor (#37618)](https://github.com/apache/airflow/pull/37618): @syedahsn - [ ] [Add fallback `region_name` value to AWS Executors (#38704)](https://github.com/apache/airflow/pull/38704): @Taragolis - [ ] [Introduce Amazon Bedrock service (#38602)](https://github.com/apache/airflow/pull/38602): @ferruzzi - [ ] [Amazon Bedrock - Model Customization Jobs (#38693)](https://github.com/apache/airflow/pull/38693): @ferruzzi - [ ] [ECS Executor - add support to adopt orphaned tasks. (#37786)](https://github.com/apache/airflow/pull/37786): @ferruzzi Linked issues: - [ ] [Linked Issue #35491](https://github.com/apache/airflow/issues/35491): @o-nikolas - [ ] [Update AWS auth manager CLI command to not disable AVP schema validation (#38301)](https://github.com/apache/airflow/pull/38301): @vincbeck - [ ] [Reduce s3hook memory usage (#37886)](https://github.com/apache/airflow/pull/37886): @ellisms Linked issues: - [ ] [Linked Issue #35449](https://github.com/apache/airflow/issues/35449): @Taragolis - [ ] [Add check in AWS auth manager to check if the Amazon Verified Permissions schema is up to date (#38333)](https://github.com/apache/airflow/pull/38333): @vincbeck - [ ] [fix: EmrServerlessStartJobOperator not serializing DAGs correctly whe… (#38022)](https://github.com/apache/airflow/pull/38022): @jliu0812 Linked issues: - [ ] [Linked Issue #38005](https://github.com/apache/airflow/issues/38005): @jliu0812 - [ ] [add "return" statement to "yield" within a while loop in amazon triggers (#38396)](https://github.com/apache/airflow/pull/38396): @Lee-W - [ ] [Fix set deprecated amazon operators arguments in `MappedOperator` (#38346)](https://github.com/apache/airflow/pull/38346): @Taragolis - [ ] [`ECSExecutor` API Retry bug fix (#38118)](https://github.com/apache/airflow/pull/38118): @syedahsn - [ ] [Fix `region` argument in `MappedOperator` based on `AwsBaseOperator` / `AwsBaseSensor` (#38178)](https://github.com/apache/airflow/pull/38178): @Taragolis - [ ] [ECS Executor retry task bug fix (#37979)](https://github.com/apache/airflow/pull/37979): @syedahsn - [ ] [Fix init checks for aws redshift to s3 operator (#37861)](https://github.com/apache/airflow/pull/37861): @okirialbert Linked issues: - [ ] [Linked Issue #36484](https://github.com/apache/airflow/issues/36484): @shahar1 - [ ] [Make the method `BaseAuthManager.is_authorized_custom_view` abstract (#37915)](https://github.com/apache/airflow/pull/37915): @vincbeck - [ ] [Replace "Role" by "Group" in AWS auth manager (#38078)](https://github.com/apache/airflow/pull/38078): @vincbeck - [ ] [Avoid use of `assert` outside of the tests (#37718)](https://github.com/apache/airflow/pull/37718): @Taragolis - [ ] [Use AwsLogsHook when fetching Glue job logs (#38010)](https://github.com/apache/airflow/pull/38010): @VincentChantreau - [ ] [Implement `filter_permitted_dag_ids` in AWS auth manager (#37666)](https://github.com/apache/airflow/pull/37666): @vincbeck - [ ] [AWS auth manager CLI: persist
[PR] Add documentation for packages - 2024-04-10 [airflow-site]
eladkal opened a new pull request, #1003: URL: https://github.com/apache/airflow-site/pull/1003 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(airflow-site) branch add-documentation-2024-04-10 created (now 308355acab)
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch add-documentation-2024-04-10 in repository https://gitbox.apache.org/repos/asf/airflow-site.git at 308355acab Add documentation for packages - 2024-04-10 No new revisions were added by this update.
Re: [PR] Always use the executemany method when inserting rows in DbApiHook as it's way much faster [airflow]
potiuk commented on PR #38715: URL: https://github.com/apache/airflow/pull/38715#issuecomment-2047988974 Much nicer now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
svn commit: r68420 [1/2] - /dev/airflow/providers/
Author: eladkal Date: Wed Apr 10 16:03:40 2024 New Revision: 68420 Log: Add artifacts for Airflow Providers 2024-04-10 Added: dev/airflow/providers/apache_airflow_providers_airbyte-3.7.0-py3-none-any.whl (with props) dev/airflow/providers/apache_airflow_providers_airbyte-3.7.0-py3-none-any.whl.asc dev/airflow/providers/apache_airflow_providers_airbyte-3.7.0-py3-none-any.whl.sha512 dev/airflow/providers/apache_airflow_providers_airbyte-3.7.0.tar.gz (with props) dev/airflow/providers/apache_airflow_providers_airbyte-3.7.0.tar.gz.asc dev/airflow/providers/apache_airflow_providers_airbyte-3.7.0.tar.gz.sha512 dev/airflow/providers/apache_airflow_providers_alibaba-2.7.3-py3-none-any.whl (with props) dev/airflow/providers/apache_airflow_providers_alibaba-2.7.3-py3-none-any.whl.asc dev/airflow/providers/apache_airflow_providers_alibaba-2.7.3-py3-none-any.whl.sha512 dev/airflow/providers/apache_airflow_providers_alibaba-2.7.3.tar.gz (with props) dev/airflow/providers/apache_airflow_providers_alibaba-2.7.3.tar.gz.asc dev/airflow/providers/apache_airflow_providers_alibaba-2.7.3.tar.gz.sha512 dev/airflow/providers/apache_airflow_providers_amazon-8.20.0-py3-none-any.whl (with props) dev/airflow/providers/apache_airflow_providers_amazon-8.20.0-py3-none-any.whl.asc dev/airflow/providers/apache_airflow_providers_amazon-8.20.0-py3-none-any.whl.sha512 dev/airflow/providers/apache_airflow_providers_amazon-8.20.0.tar.gz (with props) dev/airflow/providers/apache_airflow_providers_amazon-8.20.0.tar.gz.asc dev/airflow/providers/apache_airflow_providers_amazon-8.20.0.tar.gz.sha512 dev/airflow/providers/apache_airflow_providers_apache_beam-5.6.3-py3-none-any.whl (with props) dev/airflow/providers/apache_airflow_providers_apache_beam-5.6.3-py3-none-any.whl.asc dev/airflow/providers/apache_airflow_providers_apache_beam-5.6.3-py3-none-any.whl.sha512 dev/airflow/providers/apache_airflow_providers_apache_beam-5.6.3.tar.gz (with props) dev/airflow/providers/apache_airflow_providers_apache_beam-5.6.3.tar.gz.asc dev/airflow/providers/apache_airflow_providers_apache_beam-5.6.3.tar.gz.sha512 dev/airflow/providers/apache_airflow_providers_apache_cassandra-3.4.2-py3-none-any.whl (with props) dev/airflow/providers/apache_airflow_providers_apache_cassandra-3.4.2-py3-none-any.whl.asc dev/airflow/providers/apache_airflow_providers_apache_cassandra-3.4.2-py3-none-any.whl.sha512 dev/airflow/providers/apache_airflow_providers_apache_cassandra-3.4.2.tar.gz (with props) dev/airflow/providers/apache_airflow_providers_apache_cassandra-3.4.2.tar.gz.asc dev/airflow/providers/apache_airflow_providers_apache_cassandra-3.4.2.tar.gz.sha512 dev/airflow/providers/apache_airflow_providers_apache_hive-8.0.0-py3-none-any.whl (with props) dev/airflow/providers/apache_airflow_providers_apache_hive-8.0.0-py3-none-any.whl.asc dev/airflow/providers/apache_airflow_providers_apache_hive-8.0.0-py3-none-any.whl.sha512 dev/airflow/providers/apache_airflow_providers_apache_hive-8.0.0.tar.gz (with props) dev/airflow/providers/apache_airflow_providers_apache_hive-8.0.0.tar.gz.asc dev/airflow/providers/apache_airflow_providers_apache_hive-8.0.0.tar.gz.sha512 dev/airflow/providers/apache_airflow_providers_apache_spark-4.7.2-py3-none-any.whl (with props) dev/airflow/providers/apache_airflow_providers_apache_spark-4.7.2-py3-none-any.whl.asc dev/airflow/providers/apache_airflow_providers_apache_spark-4.7.2-py3-none-any.whl.sha512 dev/airflow/providers/apache_airflow_providers_apache_spark-4.7.2.tar.gz (with props) dev/airflow/providers/apache_airflow_providers_apache_spark-4.7.2.tar.gz.asc dev/airflow/providers/apache_airflow_providers_apache_spark-4.7.2.tar.gz.sha512 dev/airflow/providers/apache_airflow_providers_celery-3.6.2-py3-none-any.whl (with props) dev/airflow/providers/apache_airflow_providers_celery-3.6.2-py3-none-any.whl.asc dev/airflow/providers/apache_airflow_providers_celery-3.6.2-py3-none-any.whl.sha512 dev/airflow/providers/apache_airflow_providers_celery-3.6.2.tar.gz (with props) dev/airflow/providers/apache_airflow_providers_celery-3.6.2.tar.gz.asc dev/airflow/providers/apache_airflow_providers_celery-3.6.2.tar.gz.sha512 dev/airflow/providers/apache_airflow_providers_cncf_kubernetes-8.1.0-py3-none-any.whl (with props) dev/airflow/providers/apache_airflow_providers_cncf_kubernetes-8.1.0-py3-none-any.whl.asc dev/airflow/providers/apache_airflow_providers_cncf_kubernetes-8.1.0-py3-none-any.whl.sha512 dev/airflow/providers/apache_airflow_providers_cncf_kubernetes-8.1.0.tar.gz (with props) dev/airflow/providers/apache_airflow_providers_cncf_kubernetes-8.1.0.tar.gz.asc dev/airflow/providers/apache_airflow_providers_cncf_kubernetes-8.1.0.tar.gz.sha512
svn commit: r68420 [2/2] - /dev/airflow/providers/
Added: dev/airflow/providers/apache_airflow_providers_odbc-4.5.0-py3-none-any.whl.sha512 == --- dev/airflow/providers/apache_airflow_providers_odbc-4.5.0-py3-none-any.whl.sha512 (added) +++ dev/airflow/providers/apache_airflow_providers_odbc-4.5.0-py3-none-any.whl.sha512 Wed Apr 10 16:03:40 2024 @@ -0,0 +1 @@ +4845cdfc802c89456bdf2c2a7961ebe6ea55250e6dc3e15171aeb731778144b73af0d71dcfc9d36f40cad51b0cc2c9ea1425e9a385db47dfdcb251a7d68f59fc apache_airflow_providers_odbc-4.5.0-py3-none-any.whl Added: dev/airflow/providers/apache_airflow_providers_odbc-4.5.0.tar.gz == Binary file - no diff available. Propchange: dev/airflow/providers/apache_airflow_providers_odbc-4.5.0.tar.gz -- svn:mime-type = application/octet-stream Added: dev/airflow/providers/apache_airflow_providers_odbc-4.5.0.tar.gz.asc == --- dev/airflow/providers/apache_airflow_providers_odbc-4.5.0.tar.gz.asc (added) +++ dev/airflow/providers/apache_airflow_providers_odbc-4.5.0.tar.gz.asc Wed Apr 10 16:03:40 2024 @@ -0,0 +1,7 @@ +-BEGIN PGP SIGNATURE- + +iIkEABYKADEWIQSDQO8ECQokO9vDRUWG4IhmPszevgUCZha3YhMcZWxhZGthbEBh +cGFjaGUub3JnAAoJEIbgiGY+zN6+zhQBAOjVb4sEk0c5sETK2YV1Zk2yAG9QvPg2 +I/dmcXiqJ/WBAP4icfYztQF/19pSIk7mn8eruWohwzL867eN3QBNjQDcAA== +=ejv4 +-END PGP SIGNATURE- Added: dev/airflow/providers/apache_airflow_providers_odbc-4.5.0.tar.gz.sha512 == --- dev/airflow/providers/apache_airflow_providers_odbc-4.5.0.tar.gz.sha512 (added) +++ dev/airflow/providers/apache_airflow_providers_odbc-4.5.0.tar.gz.sha512 Wed Apr 10 16:03:40 2024 @@ -0,0 +1 @@ +0a94fefca325d339d3287744d3a301fab3033956ea957a7570a461225f7d85540aaa75736d1ba7a33fcf1ba754819fb188554bc673e9c0dccc36e876340d8ac7 apache_airflow_providers_odbc-4.5.0.tar.gz Added: dev/airflow/providers/apache_airflow_providers_openlineage-1.7.0-py3-none-any.whl == Binary file - no diff available. Propchange: dev/airflow/providers/apache_airflow_providers_openlineage-1.7.0-py3-none-any.whl -- svn:mime-type = application/octet-stream Added: dev/airflow/providers/apache_airflow_providers_openlineage-1.7.0-py3-none-any.whl.asc == --- dev/airflow/providers/apache_airflow_providers_openlineage-1.7.0-py3-none-any.whl.asc (added) +++ dev/airflow/providers/apache_airflow_providers_openlineage-1.7.0-py3-none-any.whl.asc Wed Apr 10 16:03:40 2024 @@ -0,0 +1,7 @@ +-BEGIN PGP SIGNATURE- + +iIkEABYKADEWIQSDQO8ECQokO9vDRUWG4IhmPszevgUCZha3YhMcZWxhZGthbEBh +cGFjaGUub3JnAAoJEIbgiGY+zN6+bWkA/jSsHWjdm5rDtg3ZjohLR4ar3OOkPOeP +Ufq/3JRhm/JEAP9XaLpuuW7u/63xBEy0X8EzagCjyo6gFJGbUaYT1IYcCw== +=AM9x +-END PGP SIGNATURE- Added: dev/airflow/providers/apache_airflow_providers_openlineage-1.7.0-py3-none-any.whl.sha512 == --- dev/airflow/providers/apache_airflow_providers_openlineage-1.7.0-py3-none-any.whl.sha512 (added) +++ dev/airflow/providers/apache_airflow_providers_openlineage-1.7.0-py3-none-any.whl.sha512 Wed Apr 10 16:03:40 2024 @@ -0,0 +1 @@ +3af8d4d53dfd6d68f8661faf5b78026cb5455563cca6e82594faada16c683d20eadef0e8cd219f09a25165c428f5d02029c04881df0a9d090857b99e4156ab24 apache_airflow_providers_openlineage-1.7.0-py3-none-any.whl Added: dev/airflow/providers/apache_airflow_providers_openlineage-1.7.0.tar.gz == Binary file - no diff available. Propchange: dev/airflow/providers/apache_airflow_providers_openlineage-1.7.0.tar.gz -- svn:mime-type = application/octet-stream Added: dev/airflow/providers/apache_airflow_providers_openlineage-1.7.0.tar.gz.asc == --- dev/airflow/providers/apache_airflow_providers_openlineage-1.7.0.tar.gz.asc (added) +++ dev/airflow/providers/apache_airflow_providers_openlineage-1.7.0.tar.gz.asc Wed Apr 10 16:03:40 2024 @@ -0,0 +1,7 @@ +-BEGIN PGP SIGNATURE- + +iIkEABYKADEWIQSDQO8ECQokO9vDRUWG4IhmPszevgUCZha3YhMcZWxhZGthbEBh +cGFjaGUub3JnAAoJEIbgiGY+zN6+IuEBAP7kFUBBlIsEVm3PogD7RjP+P6XXJzGP +Qf4N2NUF0xmiAQD/CVih/gJDLL3ClwIp1C8GyTK2H/4Z7ewv/PmM4NV3Bw== +=nfUY +-END PGP SIGNATURE- Added: dev/airflow/providers/apache_airflow_providers_openlineage-1.7.0.tar.gz.sha512
Re: [PR] Fix typo in README [airflow]
eladkal merged PR #38903: URL: https://github.com/apache/airflow/pull/38903 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(airflow) branch main updated: Fix typo in README (#38903)
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 3d804351aa Fix typo in README (#38903) 3d804351aa is described below commit 3d804351aa7a875dfdba824c2b27300cc5ce9e92 Author: Vincent <97131062+vincb...@users.noreply.github.com> AuthorDate: Wed Apr 10 12:55:54 2024 -0300 Fix typo in README (#38903) --- tests/system/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/system/README.md b/tests/system/README.md index 6e5217e95b..1085631dd0 100644 --- a/tests/system/README.md +++ b/tests/system/README.md @@ -71,7 +71,7 @@ pytest --system google tests/system/providers/google/cloud/bigquery/example_bigq You can specify several `--system` flags if you want to execute tests for several providers: ```commandline -pytest --system google --system aws tests/system +pytest --system google --system amazon tests/system ``` ### Running via Airflow CLI
Re: [PR] Prepare docs 1st wave (RC1) April 2024 [airflow]
eladkal merged PR #38863: URL: https://github.com/apache/airflow/pull/38863 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Fix typo in README [airflow]
vincbeck opened a new pull request, #38903:
URL: https://github.com/apache/airflow/pull/38903
The provider `aws` does not exist, it is `amazon`
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [I] 401 UNAUTHORIZED when using AUTH_ROLE_PUBLIC = "Admin" [airflow]
phanikumv commented on issue #38900: URL: https://github.com/apache/airflow/issues/38900#issuecomment-2047837967 cc @Lee-W could you please look into it tomorrow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Bugfix: Move rendering of `map_index_template` so it renders for failed tasks as long as it was defined before the point of failure [airflow]
TJaniF opened a new pull request, #38902:
URL: https://github.com/apache/airflow/pull/38902
In Airflow 2.9 `map_index_template` does not render when the task fails.
I moved the rendering into the `finally` of `_execute_callable` so it always
happens.
TaskFlow:
https://github.com/apache/airflow/assets/90063506/5bf11e20-0553-4b2a-95f3-648232b86c6d;>
Traditional operator:
https://github.com/apache/airflow/assets/90063506/6dbbe5c0-aa9d-45e1-98e6-cbad446cc323;>
Also attempted 2 unit tests :)
Apologies if there is already a PR addressing this. I did not see one.
cc: @RNHTTR
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[I] airflow hashicorp vault integration - path with subfolder is not wokring [airflow]
Gaurang033 opened a new issue, #38901:
URL: https://github.com/apache/airflow/issues/38901
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.5.3
### What happened?
I am trying to connect hashircorp vault as secret manager. if I am provide
the config as mentioend below it works, where variables_path doesn't have any
subfolder.
`{"connections_path": "connections", "mount_point": "airflow",
"variables_path": "variables", "config_path": null, "url": "https://xxx.com;,
"auth_type": "approle", "role_id":"airflow_role", "secret_id":"xxx"}`
However, If I provide subpath for variables or connection it doesn't work.
```
{"connections_path": "test-composer/connections", "mount_point": "airflow",
"variables_path": "test-composer/variables", "config_path": null, "url":
"https://xxx.com;, "auth_type": "approle", "role_id":"airflow_role",
"secret_id":"xxx"}
```
following is the policy.
```
path "airflow/*" {
capabilities = ["read", "list"]
}
path "airflow/test-composer/*" {
capabilities = ["read", "list"]
}
path "airflow/test-composer/variables/*" {
capabilities = ["read", "list"]
}
path "airflow/test-composer/variables/policy_event_variables" {
capabilities = ["read", "list"]
}
```
### What you think should happen instead?
mount_poing and path should be just concat to create a path to access key
from hashicorp vault, so not sure why it's not working.
### How to reproduce
Add a new secret backend by modifying the configuration.
**backend**: `airflow.providers.hashicorp.secrets.vault.VaultBackend`
**backend_kwargs**: ```{"connections_path": "test-composer/connections",
"mount_point": "airflow", "variables_path": "test-composer/variables",
"config_path": null, "url": "https://xxx.com;, "auth_type": "approle",
"role_id":"airflow_role", "secret_id":"xxx"}```
### Operating System
gcp composer
### Versions of Apache Airflow Providers
apache-airflow-providers-apache-beam==5.3.0
apache-airflow-providers-cncf-kubernetes==7.10.0
apache-airflow-providers-common-sql==1.8.1
apache-airflow-providers-dbt-cloud==3.4.1
apache-airflow-providers-ftp==3.6.1
apache-airflow-providers-google==10.12.0
apache-airflow-providers-hashicorp==3.5.0
apache-airflow-providers-http==4.7.0
apache-airflow-providers-imap==3.4.0
apache-airflow-providers-mysql==5.2.0
apache-airflow-providers-postgres==5.8.0
apache-airflow-providers-sendgrid==3.3.0
apache-airflow-providers-sqlite==3.5.0
apache-airflow-providers-ssh==3.8.1
### Deployment
Google Cloud Composer
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[I] 401 UNAUTHORIZED when using AUTH_ROLE_PUBLIC = "Admin" [airflow]
dstandish opened a new issue, #38900: URL: https://github.com/apache/airflow/issues/38900 ### Body When using ``` AUTH_ROLE_PUBLIC = 'Admin' ``` And not logged in... Some webserver endpoints return 401 UNAUTHORIZED E.g.  ### Committer - [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Expose count of scheduled tasks in metrics [airflow]
jonshea opened a new pull request, #38899: URL: https://github.com/apache/airflow/pull/38899 This pull request introduces a new metric that tracks the number of tasks in the SCHEDULED state. This can help with monitoring and debugging situations where tasks are stuck or otherwise accumulate in the `SCHEDULED` state. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] Add multiples repository in gitsync helm charts [airflow]
theotruvelot opened a new issue, #38898: URL: https://github.com/apache/airflow/issues/38898 ### Description Hello ! Can you add the feature to add multiples repository for gitsync in the helm chart ### Use case/motivation Because I got multiples repository for my dags... I use bitnami helm chart but she is not optimal for us. ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Add multiples repository in gitsync helm charts [airflow]
boring-cyborg[bot] commented on issue #38898: URL: https://github.com/apache/airflow/issues/38898#issuecomment-2047751789 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(airflow) branch main updated (34878507ef -> 1608231993)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
from 34878507ef Widen cheat sheet column to avoid wrapping commands (#3)
add 1608231993 Add multi-team diagrams (#38861)
No new revisions were added by this update.
Summary of changes:
..._team_airflow_architecture_with_grpc_api.md5sum | 1 +
...lti_team_airflow_architecture_with_grpc_api.png | Bin 0 -> 647436 bytes
...lti_team_airflow_architecture_with_grpc_api.py} | 38 -
...am_airflow_architecture_without_grpc_api.md5sum | 1 +
..._team_airflow_architecture_without_grpc_api.png | Bin 0 -> 597552 bytes
..._team_airflow_architecture_without_grpc_api.py} | 92 +++--
...diagram_multitenant_airflow_architecture.md5sum | 1 -
.../diagram_multitenant_airflow_architecture.png | Bin 601270 -> 0 bytes
hatch_build.py | 1 +
9 files changed, 52 insertions(+), 82 deletions(-)
create mode 100644
docs/apache-airflow/img/diagram_multi_team_airflow_architecture_with_grpc_api.md5sum
create mode 100644
docs/apache-airflow/img/diagram_multi_team_airflow_architecture_with_grpc_api.png
copy docs/apache-airflow/img/{diagram_multitenant_airflow_architecture.py =>
diagram_multi_team_airflow_architecture_with_grpc_api.py} (87%)
create mode 100644
docs/apache-airflow/img/diagram_multi_team_airflow_architecture_without_grpc_api.md5sum
create mode 100644
docs/apache-airflow/img/diagram_multi_team_airflow_architecture_without_grpc_api.png
rename docs/apache-airflow/img/{diagram_multitenant_airflow_architecture.py =>
diagram_multi_team_airflow_architecture_without_grpc_api.py} (73%)
delete mode 100644
docs/apache-airflow/img/diagram_multitenant_airflow_architecture.md5sum
delete mode 100644
docs/apache-airflow/img/diagram_multitenant_airflow_architecture.png
Re: [PR] Add multi-team diagrams [airflow]
potiuk merged PR #38861: URL: https://github.com/apache/airflow/pull/38861 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(airflow) branch constraints-main updated: Updating constraints. Github run id:8632236281
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 568037b72b Updating constraints. Github run id:8632236281 568037b72b is described below commit 568037b72be7ab11b8bad97e66ff81beb7d2f463 Author: Automated GitHub Actions commit AuthorDate: Wed Apr 10 14:33:50 2024 + Updating constraints. Github run id:8632236281 This update in constraints is automatically committed by the CI 'constraints-push' step based on 'refs/heads/main' in the 'apache/airflow' repository with commit sha 34878507ef032648c90f05b00d64e8171da4982e. The action that build those constraints can be found at https://github.com/apache/airflow/actions/runs/8632236281/ The image tag used for that build was: 34878507ef032648c90f05b00d64e8171da4982e. You can enter Breeze environment with this image by running 'breeze shell --image-tag 34878507ef032648c90f05b00d64e8171da4982e' All tests passed in this build so we determined we can push the updated constraints. See https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for details. --- constraints-3.10.txt | 30 +++--- constraints-3.11.txt | 30 +++--- constraints-3.12.txt | 30 +++--- constraints-3.8.txt | 30 +++--- constraints-3.9.txt | 30 +++--- constraints-no-providers-3.10.txt | 8 constraints-no-providers-3.11.txt | 8 constraints-no-providers-3.12.txt | 8 constraints-no-providers-3.8.txt | 6 +++--- constraints-no-providers-3.9.txt | 8 constraints-source-providers-3.10.txt | 30 +++--- constraints-source-providers-3.11.txt | 30 +++--- constraints-source-providers-3.12.txt | 30 +++--- constraints-source-providers-3.8.txt | 30 +++--- constraints-source-providers-3.9.txt | 30 +++--- 15 files changed, 169 insertions(+), 169 deletions(-) diff --git a/constraints-3.10.txt b/constraints-3.10.txt index 0ce7d5e69b..b1614ae467 100644 --- a/constraints-3.10.txt +++ b/constraints-3.10.txt @@ -1,6 +1,6 @@ # -# This constraints file was automatically generated on 2024-04-09T09:41:59.600284 +# This constraints file was automatically generated on 2024-04-10T13:55:29.037866 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow. # This variant of constraints install uses the HEAD of the branch version for 'apache-airflow' but installs # the providers from PIP-released packages at the moment of the constraint generation. @@ -77,7 +77,7 @@ aioresponses==0.7.6 aiosignal==1.3.1 alabaster==0.7.16 alembic==1.13.1 -alibabacloud-adb20211201==1.3.1 +alibabacloud-adb20211201==1.3.2 alibabacloud-tea==0.3.6 alibabacloud_credentials==0.3.2 alibabacloud_endpoint_util==0.0.3 @@ -201,9 +201,9 @@ aws-xray-sdk==2.13.0 azure-batch==14.2.0 azure-common==1.1.28 azure-core==1.30.1 -azure-cosmos==4.5.1 +azure-cosmos==4.6.0 azure-datalake-store==0.0.53 -azure-identity==1.15.0 +azure-identity==1.16.0 azure-keyvault-secrets==4.8.0 azure-kusto-data==4.4.0 azure-mgmt-containerinstance==10.1.0 @@ -297,7 +297,7 @@ facebook_business==19.0.2 fastavro==1.9.4 fasteners==0.19 fastjsonschema==2.19.1 -filelock==3.13.3 +filelock==3.13.4 flower==2.0.1 frozenlist==1.4.1 fsspec==2023.12.2 @@ -458,7 +458,7 @@ msrestazure==0.6.4 multi_key_dict==2.0.3 multidict==6.0.5 mypy-boto3-appflow==1.34.0 -mypy-boto3-rds==1.34.65 +mypy-boto3-rds==1.34.81 mypy-boto3-redshift-data==1.34.0 mypy-boto3-s3==1.34.65 mypy-extensions==1.0.0 @@ -478,9 +478,9 @@ objsize==0.7.0 openai==1.16.2 openapi-schema-validator==0.6.2 openapi-spec-validator==0.7.1 -openlineage-integration-common==1.11.3 -openlineage-python==1.11.3 -openlineage_sql==1.11.3 +openlineage-integration-common==1.12.0 +openlineage-python==1.12.0 +openlineage_sql==1.12.0 openpyxl==3.1.2 opensearch-py==2.5.0 opentelemetry-api==1.24.0 @@ -513,10 +513,10 @@ pexpect==4.9.0 pgvector==0.2.5 pinecone-client==2.2.4 pinotdb==5.1.6 -pipdeptree==2.17.0 +pipdeptree==2.18.0 pipx==1.5.0 pkginfo==1.10.0 -platformdirs==3.11.0 +platformdirs==4.2.0 pluggy==1.4.0 ply==3.11 plyvel==1.5.1 @@ -618,7 +618,7 @@ scrapbook==0.5.0 semver==3.0.2 sendgrid==6.11.0 sentinels==1.0.0 -sentry-sdk==1.44.1 +sentry-sdk==1.45.0 setproctitle==1.3.3 shapely==2.0.3 shellingham==1.5.4 @@ -629,7 +629,7 @@ smbprotocol==1.13.0 smmap==5.0.1 sniffio==1.3.1 snowballstemmer==2.2.0 -snowflake-connector-python==3.7.1 +snowflake-connector-python==3.8.1 snowflake-sqlalchemy==1.5.1
(airflow-site) branch gh-pages updated (46289ab974 -> d676dd8530)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch gh-pages in repository https://gitbox.apache.org/repos/asf/airflow-site.git discard 46289ab974 Rewritten history to remove past gh-pages deployments new d676dd8530 Rewritten history to remove past gh-pages deployments This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (46289ab974) \ N -- N -- N refs/heads/gh-pages (d676dd8530) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: 404.html | 3 - announcements/index.html | 3 - blocktype/index.html | 3 - blocktype/testimonial/index.html | 3 - blocktype/use-case/index.html | 3 - blocktype/use-case/index.xml | 80 ++-- blog/airflow-1.10.10/index.html| 7 +- blog/airflow-1.10.12/index.html| 7 +- blog/airflow-1.10.8-1.10.9/index.html | 7 +- blog/airflow-2.2.0/index.html | 7 +- blog/airflow-2.3.0/index.html | 7 +- blog/airflow-2.4.0/index.html | 7 +- blog/airflow-2.5.0/index.html | 7 +- blog/airflow-2.6.0/index.html | 7 +- blog/airflow-2.7.0/index.html | 7 +- blog/airflow-2.8.0/index.html | 7 +- blog/airflow-2.9.0/index.html | 7 +- blog/airflow-survey-2020/index.html| 7 +- blog/airflow-survey-2022/index.html| 7 +- blog/airflow-survey/index.html | 7 +- blog/airflow-two-point-oh-is-here/index.html | 7 +- blog/airflow_summit_2021/index.html| 7 +- blog/airflow_summit_2022/index.html| 7 +- blog/announcing-new-website/index.html | 7 +- blog/apache-airflow-for-newcomers/index.html | 7 +- .../index.html | 7 +- .../index.html | 7 +- .../index.html | 7 +- .../index.html | 7 +- blog/fab-oid-vulnerability/index.html | 7 +- .../index.html | 7 +- blog/index.html| 3 - blog/introducing_setup_teardown/index.html | 7 +- .../index.html | 7 +- blog/tags/airflow-summit/index.html| 3 - blog/tags/community/index.html | 3 - blog/tags/development/index.html | 3 - blog/tags/documentation/index.html | 3 - blog/tags/release/index.html | 3 - blog/tags/rest-api/index.html | 3 - blog/tags/survey/index.html| 3 - blog/tags/users/index.html | 3 - blog/tags/vulnerabilities/index.html | 3 - categories/index.html | 3 - code-of-conduct/index.html | 3 - community/index.html | 3 - docs/index.html| 3 - ecosystem/index.html | 3 - index.html | 3 - meetups/index.html | 3 - privacy-notice/index.html | 3 - roadmap/index.html | 3 - search/index.html | 7 +- sitemap.xml| 136 ++--- survey/index.html | 3 - tags/index.html| 3 - use-cases/adobe/index.html | 7 +- use-cases/adyen/index.html | 7 +- use-cases/big-fish-games/index.html| 7 +- use-cases/business_operations/index.html | 29 + use-cases/dish/index.html
Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]
potiuk commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2047713345 That works, maybe even split it to smaller pieces. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Validate dbt `trigger_reason` field to be less than 255 characters [airflow]
boraberke commented on PR #38896: URL: https://github.com/apache/airflow/pull/38896#issuecomment-2047689569 Hey @josh-fell, would greatly appreciate your thoughts and suggestions on this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] bump uv to 0.1.31 [airflow]
potiuk commented on PR #38874: URL: https://github.com/apache/airflow/pull/38874#issuecomment-2047688734 Sure :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Validate dbt `trigger_reason` field to be less than 255 characters [airflow]
boraberke opened a new pull request, #38896: URL: https://github.com/apache/airflow/pull/38896 Validate and truncate `trigger_reason` field if it is longer than the limit of 255 characters. closes: #34676 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] add log for running callback [airflow]
romsharon98 commented on code in PR #38892:
URL: https://github.com/apache/airflow/pull/38892#discussion_r1559532674
##
tests/models/test_taskinstance.py:
##
@@ -2856,8 +2856,9 @@ def on_execute_callable(context):
],
)
@patch("logging.Logger.exception")
+@patch("logging.Logger.info")
Review Comment:
Thanks for notice it, I changed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Resolve internal warnings for TestLocalTaskJob and TestSigTermOnRunner [airflow]
Owen-CH-Leung commented on code in PR #38893:
URL: https://github.com/apache/airflow/pull/38893#discussion_r1559514632
##
tests/jobs/test_local_task_job.py:
##
@@ -886,53 +907,53 @@ def test_process_sigterm_works_with_retries(
execution_date = DEFAULT_DATE
run_id = f"test-{execution_date.date().isoformat()}"
-# Run LocalTaskJob in separate process
-proc = mp_context.Process(
-target=self._sigterm_local_task_runner,
-args=(dag_id, task_id, run_id, execution_date, task_started,
retry_callback_called),
-name="LocalTaskJob-TestProcess",
-daemon=daemon,
-)
-proc.start()
-
-try:
-with timeout(wait_timeout, "Timeout during waiting start
LocalTaskJob"):
-while task_started.value == 0:
-time.sleep(0.2)
-os.kill(proc.pid, signal.SIGTERM)
-
-with timeout(wait_timeout, "Timeout during waiting callback"):
-while retry_callback_called.value == 0:
-time.sleep(0.2)
-finally:
-proc.kill()
-
-assert retry_callback_called.value == 1
-# Internally callback finished before TaskInstance commit changes in
DB (as of Jan 2022).
-# So we can't easily check TaskInstance.state without any race
conditions drawbacks,
-# and fact that process with LocalTaskJob could be already killed.
-# We could add state validation (`UP_FOR_RETRY`) if callback mechanism
changed.
-
-pytest_capture = request.config.option.capture
-if pytest_capture == "no":
-# Since we run `LocalTaskJob` in the separate process we can grab
ut easily by `caplog`.
-# However, we could grab it from stdout/stderr but only if `-s`
flag set, see:
-# https://github.com/pytest-dev/pytest/issues/5997
-captured = capfd.readouterr()
+with tempfile.NamedTemporaryFile() as tmpfile:
Review Comment:
Thanks. I've revised to use the `tmp_path` fixture.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [Landing pages] Replace use case video iframes with links [airflow-site]
potiuk commented on PR #1002: URL: https://github.com/apache/airflow-site/pull/1002#issuecomment-2047642637 proably best choice -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(airflow-site) branch main updated: [Landing pages] Replace use case video iframes with links (#1002)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
The following commit(s) were added to refs/heads/main by this push:
new b6ce741de7 [Landing pages] Replace use case video iframes with links
(#1002)
b6ce741de7 is described below
commit b6ce741de7743dc0b60d16dc875d5f9cf6ff891a
Author: Michael Robinson <68482867+merobi-...@users.noreply.github.com>
AuthorDate: Wed Apr 10 10:04:13 2024 -0400
[Landing pages] Replace use case video iframes with links (#1002)
* Fix for google news error.
Signed-off-by: merobi-hub
* Replace use case video ifrmaes with links.
Signed-off-by: merobi-hub
-
Signed-off-by: merobi-hub
---
.../content/en/use-cases/business_operations.md| 22 +++---
.../site/content/en/use-cases/etl_analytics.md | 21 +++--
.../en/use-cases/infrastructure-management.md | 21 +++--
landing-pages/site/content/en/use-cases/mlops.md | 21 +++--
.../site/layouts/partials/hooks/head-end.html | 3 ---
5 files changed, 12 insertions(+), 76 deletions(-)
diff --git a/landing-pages/site/content/en/use-cases/business_operations.md
b/landing-pages/site/content/en/use-cases/business_operations.md
index 5daf45cc19..8c829ae8f9 100644
--- a/landing-pages/site/content/en/use-cases/business_operations.md
+++ b/landing-pages/site/content/en/use-cases/business_operations.md
@@ -20,27 +20,11 @@ This video shows an example of using Airflow to run the
pipelines that power a c
-
+https://www.youtube.com/embed/2CEApKN0z1U?autoplay=1;>
+
+
-
-document.getElementById('videoPlaceholder').addEventListener('click',
function() {
-var iframe = document.createElement('iframe');
-iframe.setAttribute('src',
'https://www.youtube.com/embed/2CEApKN0z1U?autoplay=1');
-iframe.setAttribute('width', '100%');
-iframe.setAttribute('height', '315');
-iframe.setAttribute('frameborder', '0');
-iframe.setAttribute('allow', 'accelerometer; autoplay;
clipboard-write; encrypted-media; gyroscope; picture-in-picture');
-iframe.setAttribute('allowfullscreen', 'allowfullscreen');
-iframe.style.display = 'block';
-
-var videoContainer = document.getElementById('videoContainer');
-videoContainer.innerHTML = '';
-videoContainer.appendChild(iframe);
-});
-
-
-
## Why use Airflow for Business Operations?
diff --git a/landing-pages/site/content/en/use-cases/etl_analytics.md
b/landing-pages/site/content/en/use-cases/etl_analytics.md
index b0f612cfbb..c578c3e22c 100644
--- a/landing-pages/site/content/en/use-cases/etl_analytics.md
+++ b/landing-pages/site/content/en/use-cases/etl_analytics.md
@@ -19,26 +19,11 @@ The video below shows a simple ETL/ELT pipeline in Airflow
that extracts climate
-
+https://www.youtube.com/embed/ljBU_VyihVQ?autoplay=1;>
+
+
-
-document.getElementById('videoPlaceholder').addEventListener('click',
function() {
-var iframe = document.createElement('iframe');
-iframe.setAttribute('src',
'https://www.youtube.com/embed/ljBU_VyihVQ?autoplay=1');
-iframe.setAttribute('width', '90%');
-iframe.setAttribute('height', '315');
-iframe.setAttribute('frameborder', '0');
-iframe.setAttribute('allow', 'accelerometer; autoplay;
clipboard-write; encrypted-media; gyroscope; picture-in-picture');
-iframe.setAttribute('allowfullscreen', 'allowfullscreen');
-iframe.style.display = 'block';
-
-var videoContainer = document.getElementById('videoContainer');
-videoContainer.innerHTML = '';
-videoContainer.appendChild(iframe);
-});
-
-
## Why use Airflow for ETL/ELT pipelines?
diff --git
a/landing-pages/site/content/en/use-cases/infrastructure-management.md
b/landing-pages/site/content/en/use-cases/infrastructure-management.md
index bab2aea859..60adf5543e 100644
--- a/landing-pages/site/content/en/use-cases/infrastructure-management.md
+++ b/landing-pages/site/content/en/use-cases/infrastructure-management.md
@@ -18,26 +18,11 @@ Airflow can interact with any API, which makes it a great
tool to manage your in
Infrastructure management is often needed within the context of other use
cases, such as MLOps, or implementing data quality checks. This video shows an
example of how it might be used for an MLOps pipeline. You can find the code
shown in this example
[here](https://github.com/astronomer/use-case-setup-teardown-data-quality).
-
+https://www.youtube.com/embed/JkURWnl76GQ?autoplay=1;>
+
+
-
-
