[GitHub] [airflow] potiuk commented on pull request #28256: Include full path to Python files under zip path while clearing import errors.
potiuk commented on PR #28256: URL: https://github.com/apache/airflow/pull/28256#issuecomment-1399202367 It can be. You can just recursively find if the files exist splitting by directory separator. The advantage will be that you will do it only once during migration so it might take even multiple minutes to complete. Rest the file path as cache spreading up testing for modified files and migration as a way to build the cache from existing data (and continuously updating the cache after migration) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow-client-python] tag 2.5.0 created (now 1a6f6ee)
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to tag 2.5.0 in repository https://gitbox.apache.org/repos/asf/airflow-client-python.git at 1a6f6ee (commit) No new revisions were added by this update.
[airflow-client-python] branch master updated: Update Changelog for 2.5.0 release (#58)
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow-client-python.git
The following commit(s) were added to refs/heads/master by this push:
new 1a6f6ee Update Changelog for 2.5.0 release (#58)
1a6f6ee is described below
commit 1a6f6ee780563ab203d615d793ac4d3a90975c6c
Author: Ephraim Anierobi
AuthorDate: Sat Jan 21 07:39:02 2023 +0100
Update Changelog for 2.5.0 release (#58)
---
CHANGELOG.md | 20
1 file changed, 20 insertions(+)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f12a943..c1597e9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -17,6 +17,26 @@
under the License.
-->
+# v2.5.0
+
+Apache Airflow API version: 2.5.x
+
+### Major changes:
+
+- NA
+
+### Major fixes:
+
+- NA
+
+### New API supported:
+
+- GET /datasets/events | Get dataset events
+- GET /datasets | Get datasets
+- GET /datasets/{id} | Get a dataset
+- POST /dags/{dag_id}/dagRuns/{dag_run_id}/clear | Clear a dagrun endpoint
+- GET /dags/{dag_id}/dagRuns/{dag_run_id}/upstreamDatasetEvents | Get dataset
events for a DAG run
+
# v2.3.0
Apache Airflow API version: 2.3.x
[GitHub] [airflow-client-python] ephraimbuddy merged pull request #58: Update Changelog for 2.5.0 release
ephraimbuddy merged PR #58: URL: https://github.com/apache/airflow-client-python/pull/58 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow-client-python] ephraimbuddy commented on pull request #58: Update Changelog for 2.5.0 release
ephraimbuddy commented on PR #58: URL: https://github.com/apache/airflow-client-python/pull/58#issuecomment-1399192717 Ok. Looks like a review is not necessary. It's a small change too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] rkarish opened a new pull request, #29079: Use sqlalchemy engine with pandas in DbApiHook (apache#28969)
rkarish opened a new pull request, #29079: URL: https://github.com/apache/airflow/pull/29079 Closes: apache#28969 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow-client-python] branch update-changelog-for-2.5.0 created (now 2870404)
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to branch update-changelog-for-2.5.0 in repository https://gitbox.apache.org/repos/asf/airflow-client-python.git at 2870404 Update Changelog for 2.5.0 release This branch includes the following new commits: new 2870404 Update Changelog for 2.5.0 release The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[airflow-client-python] 01/01: Update Changelog for 2.5.0 release
This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch update-changelog-for-2.5.0
in repository https://gitbox.apache.org/repos/asf/airflow-client-python.git
commit 2870404823fe3060b5f4cd6011a8829f577dcade
Author: Ephraim Anierobi
AuthorDate: Sat Jan 21 07:10:22 2023 +0100
Update Changelog for 2.5.0 release
---
CHANGELOG.md | 20
1 file changed, 20 insertions(+)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f12a943..c1597e9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -17,6 +17,26 @@
under the License.
-->
+# v2.5.0
+
+Apache Airflow API version: 2.5.x
+
+### Major changes:
+
+- NA
+
+### Major fixes:
+
+- NA
+
+### New API supported:
+
+- GET /datasets/events | Get dataset events
+- GET /datasets | Get datasets
+- GET /datasets/{id} | Get a dataset
+- POST /dags/{dag_id}/dagRuns/{dag_run_id}/clear | Clear a dagrun endpoint
+- GET /dags/{dag_id}/dagRuns/{dag_run_id}/upstreamDatasetEvents | Get dataset
events for a DAG run
+
# v2.3.0
Apache Airflow API version: 2.3.x
[GitHub] [airflow] sudeepgupta90 opened a new pull request, #29078: enhance production guide [#20999][#28637]
sudeepgupta90 opened a new pull request, #29078:
URL: https://github.com/apache/airflow/pull/29078
closes: #20999 #28637
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] rkarish commented on issue #28969: "UserWarning: pandas only supports SQLAlchemy connectable" when calling MsSqlHook pandas methods
rkarish commented on issue #28969: URL: https://github.com/apache/airflow/issues/28969#issuecomment-1399188456 I'll take this one -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] tirkarthi commented on pull request #28256: Include full path to Python files under zip path while clearing import errors.
tirkarthi commented on PR #28256: URL: https://github.com/apache/airflow/pull/28256#issuecomment-1399175298 I am not sure if there is a reliable way to extract "/home/karthikeyan/airflow/dags/error_dag.zip" from "/home/karthikeyan/airflow/dags/error_dag.zip/error_dag.py" through zipfile module for migration of existing errors. If it's just clearing based on zip file then wouldn't this become a query like below just deleting all zip file related errors though the zip files haven't changed? It might become like a query like below that already exists in processor.py ```python query = session.query(errors.ImportError) if self._file_paths: for file_path in self._file_paths: query = query.filter(errors.ImportError.filename.startswith(file_path)) query.delete(synchronize_session="fetch") session.commit() ``` https://github.com/apache/airflow/blob/09b3a29972430e5749d772359692fe4a9d528e48/airflow/dag_processing/processor.py#L563-L566 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #27616: Flush the session in `handle_failure()` prior to invoking callbacks
Taragolis commented on code in PR #27616:
URL: https://github.com/apache/airflow/pull/27616#discussion_r1083237739
##
airflow/models/taskinstance.py:
##
@@ -1808,13 +1808,13 @@ def handle_failure(
except Exception:
self.log.exception("Failed to send email to: %s", task.email)
-if callback and context:
-self._run_finished_callback(callback, context, callback_type)
-
if not test_mode:
session.merge(self)
session.flush()
+if callback and context:
+self._run_finished_callback(callback, context, callback_type)
+
Review Comment:
I don't think this is the right way for solve #27616.
If we call `sesion.flush` before callbacks we will lock row in DB until
callback completed as result you won't able to reset task from the UI as well
as other side effects related to this issue.
How to reproduce
```python
import time
from airflow.decorators import task
from airflow.models import DAG
import pendulum
def endless_callback(context):
while True:
print("Kill me if you can!")
time.sleep(10)
with DAG(
dag_id="infinity_callback",
schedule=None,
start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
catchup=False,
tags=["infinity callback", "lock db"],
):
@task(on_retry_callback=endless_callback, retries=1)
def failed_task():
1 / 0
x = failed_task()
```
Ideally we should commit changes before run any callbacks but we can't do
this - `LocalTaskJob` would decide that the state is set externally and kill
process with callback.
The general issues with callbacks right now if callback run for a long time
then:
1. DB Session doesn't return into the poll
2. Our transaction would be idled which not a good (I'd rather say terrible)
and lead to a lot different problems.
I'm not sure there is good solution exist without decouple "callbacks" from
Task Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[airflow] branch main updated (ca9a59b3e8 -> 09b3a29972)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from ca9a59b3e8 emit dagrun failed duration when timeout (#29076) add 09b3a29972 Migrate DagFileProcessorManager.clear_nonexistent_import_errors to Internal API (#28976) No new revisions were added by this update. Summary of changes: airflow/api_internal/endpoints/rpc_api_endpoint.py | 2 ++ airflow/dag_processing/manager.py | 12 2 files changed, 10 insertions(+), 4 deletions(-)
[GitHub] [airflow] potiuk closed issue #28785: AIP-44 Migrate DagFileProcessorManager.clear_nonexistent_import_errors to Internal API
potiuk closed issue #28785: AIP-44 Migrate DagFileProcessorManager.clear_nonexistent_import_errors to Internal API URL: https://github.com/apache/airflow/issues/28785 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #28976: Migrate DagFileProcessorManager.clear_nonexistent_import_errors to internal API
potiuk merged PR #28976: URL: https://github.com/apache/airflow/pull/28976 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (9dedf81fa1 -> ca9a59b3e8)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 9dedf81fa1 Check for run_id url param when linking to graph/gantt views (#29066) add ca9a59b3e8 emit dagrun failed duration when timeout (#29076) No new revisions were added by this update. Summary of changes: airflow/jobs/scheduler_job.py | 2 ++ 1 file changed, 2 insertions(+)
[GitHub] [airflow] potiuk closed issue #29013: Metrics dagrun.duration.failed. not updated when the dag run failed due to timeout
potiuk closed issue #29013: Metrics dagrun.duration.failed. not updated when the dag run failed due to timeout URL: https://github.com/apache/airflow/issues/29013 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #29076: Emit dagrun duration when it fails because of a timeout
potiuk merged PR #29076: URL: https://github.com/apache/airflow/pull/29076 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #28846: Updated app to support configuring the caching hash method for FIPS
potiuk commented on code in PR #28846: URL: https://github.com/apache/airflow/pull/28846#discussion_r1083233955 ## airflow/compat/hashlib_wrapper.py: ## @@ -0,0 +1,41 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import hashlib Review Comment: NICE! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29034: Add functionality for operators to template all eligible fields (apac…
potiuk commented on PR #29034: URL: https://github.com/apache/airflow/pull/29034#issuecomment-1399164723 LGTM. @uranusjr - I think you were earlier involved in discussions about that one - WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (48cab7cfeb -> 9dedf81fa1)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 48cab7cfeb bump (#28602) add 9dedf81fa1 Check for run_id url param when linking to graph/gantt views (#29066) No new revisions were added by this update. Summary of changes: airflow/www/views.py | 9 - 1 file changed, 8 insertions(+), 1 deletion(-)
[GitHub] [airflow] potiuk merged pull request #29066: Check for run_id url param when linking to graph/gantt views
potiuk merged PR #29066: URL: https://github.com/apache/airflow/pull/29066 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk closed issue #28155: Links to dag graph some times display incorrect dagrun
potiuk closed issue #28155: Links to dag graph some times display incorrect dagrun URL: https://github.com/apache/airflow/issues/28155 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28880: Can't configure Kubernetes and Celery workers in Helm Chart
potiuk commented on issue #28880: URL: https://github.com/apache/airflow/issues/28880#issuecomment-1399161324 Just make a PR - we (or others) can easier discuss it there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29056: Handling error on cluster policy itself
potiuk commented on PR #29056: URL: https://github.com/apache/airflow/pull/29056#issuecomment-1399158895 > @potiuk (thanks for your kind review..!) I agree with you that exception catch logic on dagbag will be simplified, if using just `except Exception`. so, I reflected in the code. > > But, I think anothor Exception class(`AirflowClusterPolicyError`) is necessary, because it facilitates intuitive identification of the cause of the error, just from exception CLASS NAME, although we can find the error cause from Web UI stacktrace(based on import_error table row) without it. > > In my opionion, if new exception class(`AirflowClusterPolicyError`) is not needed, explicit `AirflowClusterPolicyViolation` exception class is also not necessary. Because, user can raise any exception class(like `ValueError`, `FileNotFoundError`, or any custom exception) to suit the situation of the error. Yep. This way is OK. I understood the reasoning when I saw your changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20308: Multi Schedulers don't synchronize task instance state when using Triggerer
potiuk commented on issue #20308: URL: https://github.com/apache/airflow/issues/20308#issuecomment-1399158502 No idea @jskalasariya . There is nothing like prioritization in Airflow. Things get fixed when someone fix them because that someone decides it is important. If it is not fixed, then it means no-one though it is important enough to fix it. Not sure why you asking us 2 for prioritisation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] okayhooni commented on pull request #29056: Handling error on cluster policy itself
okayhooni commented on PR #29056: URL: https://github.com/apache/airflow/pull/29056#issuecomment-1399158194 @potiuk (thanks for your kind review..!) I agree with you that exception catch logic on dagbag will be simplified, if using just `except Exception`. so, I reflected in the code. But, I think anothor Exception class(`AirflowClusterPolicyError`) is necessary, because it facilitates intuitive identification of the cause of the error, just from exception CLASS NAME, although we can find the error cause from Web UI stacktrace(based on import_error table row) without it. In my opionion, if new exception class(`AirflowClusterPolicyError`) is not needed, explicit `AirflowClusterPolicyViolation` exception class is also not necessary. Because, user can raise any exception class(like `ValueError`, `FileNotFoundError`, or any custom exception) to suit the situation of the error. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29056: Handling error on cluster policy itself
potiuk commented on PR #29056: URL: https://github.com/apache/airflow/pull/29056#issuecomment-1399157535 LGTM now. WDYT Others (@ashb @uranusjr @jedcunningham) - maybe there was a good reason why we decided to only handle specific errors that I have not thought about. I have a feeling that how it is proposed now is much more robust and user friendly - as all kind of exceptions will be handled now with user-facing information. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28256: Include full path to Python files under zip path while clearing import errors.
potiuk commented on PR #28256: URL: https://github.com/apache/airflow/pull/28256#issuecomment-1399157056 > @potiuk @uranusjr Sorry, I am little confused. Currently `ImportError` model has filename column that stores full path to Python along with zipfile like `/home/karthikeyan/airflow/dags/error_dag.zip/error_dag.py` . Do you want to add a migration where `file_path` column stores `/home/karthikeyan/airflow/dags/error_dag.zip` value and use it in the deletion query? Yes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jskalasariya commented on issue #20308: Multi Schedulers don't synchronize task instance state when using Triggerer
jskalasariya commented on issue #20308: URL: https://github.com/apache/airflow/issues/20308#issuecomment-1399156944 @ashb @potiuk are we prioritising this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] syedahsn commented on a diff in pull request #29077: add retries to stop_pipeline on conflict
syedahsn commented on code in PR #29077:
URL: https://github.com/apache/airflow/pull/29077#discussion_r1083228639
##
airflow/providers/amazon/aws/hooks/sagemaker.py:
##
@@ -1154,19 +1154,35 @@ def stop_pipeline(
:return: Status of the pipeline execution after the operation.
One of 'Executing'|'Stopping'|'Stopped'|'Failed'|'Succeeded'.
"""
-try:
-
self.conn.stop_pipeline_execution(PipelineExecutionArn=pipeline_exec_arn)
-except ClientError as ce:
-# we have to rely on the message to catch the right error here,
because its type
-# (ValidationException) is shared with other kinds of error (for
instance, badly formatted ARN)
-if (
-not fail_if_not_running
-and "Only pipelines with 'Executing' status can be stopped" in
ce.response["Error"]["Message"]
-):
-self.log.warning("Cannot stop pipeline execution, as it was
not running: %s", ce)
-else:
-self.log.error(ce)
-raise
+retries = 2 # i.e. 3 calls max, 1 initial + 2 retries
Review Comment:
Why not make this a configurable parameter? I understand that this code
change is to catch an unlikely transient error, but I think adding "magic"
numbers reduces code readability. If not a configurable parameter, maybe add a
constant at the top with a clear name of what the number is for.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29048: Allow downloading of dbt Cloud artifacts to non-existent paths
potiuk commented on PR #29048: URL: https://github.com/apache/airflow/pull/29048#issuecomment-1399156631 Random failures. Nice addition and nice improvements in tests. Merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29052: Fix KubernetesPodOperator xcom push when `get_logs=False`
potiuk commented on PR #29052: URL: https://github.com/apache/airflow/pull/29052#issuecomment-1399155916 LGTM but I think I need also second opinion (@jedcunningham :) again )? I also think having unit tests covering the cases that would have failed previously are needed here. I am always suspicious about changes introducing behavioural change where no tests are touched. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29000: Skip KubernetesPodOperator task when it returns a provided exit code
potiuk commented on PR #29000: URL: https://github.com/apache/airflow/pull/29000#issuecomment-1399155414 @jedcunningham - maybe you can take a look as well especially at the exit-code retrieval part. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] syedahsn commented on a diff in pull request #29077: add retries to stop_pipeline on conflict
syedahsn commented on code in PR #29077:
URL: https://github.com/apache/airflow/pull/29077#discussion_r1083228639
##
airflow/providers/amazon/aws/hooks/sagemaker.py:
##
@@ -1154,19 +1154,35 @@ def stop_pipeline(
:return: Status of the pipeline execution after the operation.
One of 'Executing'|'Stopping'|'Stopped'|'Failed'|'Succeeded'.
"""
-try:
-
self.conn.stop_pipeline_execution(PipelineExecutionArn=pipeline_exec_arn)
-except ClientError as ce:
-# we have to rely on the message to catch the right error here,
because its type
-# (ValidationException) is shared with other kinds of error (for
instance, badly formatted ARN)
-if (
-not fail_if_not_running
-and "Only pipelines with 'Executing' status can be stopped" in
ce.response["Error"]["Message"]
-):
-self.log.warning("Cannot stop pipeline execution, as it was
not running: %s", ce)
-else:
-self.log.error(ce)
-raise
+retries = 2 # i.e. 3 calls max, 1 initial + 2 retries
Review Comment:
Why not make this a configurable parameter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28795: Migrate Models Variable to Internal API
potiuk commented on PR #28795: URL: https://github.com/apache/airflow/pull/28795#issuecomment-1399154745 Needs conflict resolution after other merges. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29065: When clearing task instances try to get associated DAGs from database
potiuk commented on PR #29065: URL: https://github.com/apache/airflow/pull/29065#issuecomment-1399152967 looks like an important one to merge if the diagnosis is correct. I have not yet looked at the details and do not get that deep knowledge about this part but might be worth seriously looking at it @ashb @ephraimbuddy . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28602: Bump OPENAPI_GENERATOR_CLI_VER to latest v5 image(5.4.0)
potiuk commented on PR #28602: URL: https://github.com/apache/airflow/pull/28602#issuecomment-1399150521 Client announced . Merging :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (886d87dc8a -> 48cab7cfeb)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 886d87dc8a Chart: add doc note about podtemplate images (#29032) add 48cab7cfeb bump (#28602) No new revisions were added by this update. Summary of changes: clients/gen/common.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[GitHub] [airflow] potiuk merged pull request #28602: Bump OPENAPI_GENERATOR_CLI_VER to latest v5 image(5.4.0)
potiuk merged PR #28602: URL: https://github.com/apache/airflow/pull/28602 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #29032: Chart: add doc note about podtemplate images
potiuk merged PR #29032: URL: https://github.com/apache/airflow/pull/29032 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (5da085bcb2 -> 886d87dc8a)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 5da085bcb2 Upgrade bitnami/postgresql helm depenency from 10.5.3 to 12.1.9 (#29071) add 886d87dc8a Chart: add doc note about podtemplate images (#29032) No new revisions were added by this update. Summary of changes: chart/values.schema.json | 4 ++-- chart/values.yaml| 4 2 files changed, 6 insertions(+), 2 deletions(-)
[GitHub] [airflow] potiuk commented on pull request #27357: Update bitnami/postgresql dependency chart to latest version
potiuk commented on PR #27357: URL: https://github.com/apache/airflow/pull/27357#issuecomment-1399144100 Closed as #29071 has been merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #27357: Update bitnami/postgresql dependency chart to latest version
potiuk closed pull request #27357: Update bitnami/postgresql dependency chart to latest version URL: https://github.com/apache/airflow/pull/27357 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
svn commit: r59508 - /release/airflow/clients/python/2.5.0/
Author: ephraimanierobi Date: Sat Jan 21 01:43:30 2023 New Revision: 59508 Log: Release Apache Airflow Python Client 2.5.0 from 2.5.0rc1 Added: release/airflow/clients/python/2.5.0/ release/airflow/clients/python/2.5.0/apache-airflow-client-2.5.0-bin.tar.gz - copied unchanged from r59507, dev/airflow/clients/python/2.5.0rc1/apache-airflow-client-2.5.0-bin.tar.gz release/airflow/clients/python/2.5.0/apache-airflow-client-2.5.0-bin.tar.gz.asc - copied unchanged from r59507, dev/airflow/clients/python/2.5.0rc1/apache-airflow-client-2.5.0-bin.tar.gz.asc release/airflow/clients/python/2.5.0/apache-airflow-client-2.5.0-bin.tar.gz.sha512 - copied unchanged from r59507, dev/airflow/clients/python/2.5.0rc1/apache-airflow-client-2.5.0-bin.tar.gz.sha512 release/airflow/clients/python/2.5.0/apache-airflow-client-2.5.0-source.tar.gz - copied unchanged from r59507, dev/airflow/clients/python/2.5.0rc1/apache-airflow-client-2.5.0-source.tar.gz release/airflow/clients/python/2.5.0/apache-airflow-client-2.5.0-source.tar.gz.asc - copied unchanged from r59507, dev/airflow/clients/python/2.5.0rc1/apache-airflow-client-2.5.0-source.tar.gz.asc release/airflow/clients/python/2.5.0/apache-airflow-client-2.5.0-source.tar.gz.sha512 - copied unchanged from r59507, dev/airflow/clients/python/2.5.0rc1/apache-airflow-client-2.5.0-source.tar.gz.sha512 release/airflow/clients/python/2.5.0/apache_airflow_client-2.5.0-py3-none-any.whl - copied unchanged from r59507, dev/airflow/clients/python/2.5.0rc1/apache_airflow_client-2.5.0-py3-none-any.whl release/airflow/clients/python/2.5.0/apache_airflow_client-2.5.0-py3-none-any.whl.asc - copied unchanged from r59507, dev/airflow/clients/python/2.5.0rc1/apache_airflow_client-2.5.0-py3-none-any.whl.asc release/airflow/clients/python/2.5.0/apache_airflow_client-2.5.0-py3-none-any.whl.sha512 - copied unchanged from r59507, dev/airflow/clients/python/2.5.0rc1/apache_airflow_client-2.5.0-py3-none-any.whl.sha512
[GitHub] [airflow] csm10495 commented on pull request #28808: Allow setting the name for the base container within K8s Pod Operator
csm10495 commented on PR #28808: URL: https://github.com/apache/airflow/pull/28808#issuecomment-1399136793 Also does that change it just for the instance or for the class if done that way? .. I'd have to check to know for sure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] csm10495 commented on pull request #28808: Allow setting the name for the base container within K8s Pod Operator
csm10495 commented on PR #28808: URL: https://github.com/apache/airflow/pull/28808#issuecomment-1399135597 It doesn't really match how params are normally passed through. Also it would look kind of weird in a taskflow dag. I think the PR as is makes a bit more sense. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] vandonr-amz opened a new pull request, #29077: add retries to stop_pipeline on conflict
vandonr-amz opened a new pull request, #29077: URL: https://github.com/apache/airflow/pull/29077 Explained in comments in the code, but there is a race condition when stopping a pipeline that can make the operation fail if it's in the middle of two steps. It should be very rare in real life, but happens more often on our system tests, which run a dummy pipeline made of very short steps. Boto doesn't retry it because its retry strategy doesn't include either 409 HTTP codes nor ConflictExceptions, which can happen for good reasons when trying to create resources that already exist for instance. Sagemaker team might implement a fix/retry on the server side, but in the meantime, we can fix it here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (2b7071c600 -> 5da085bcb2)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 2b7071c600 FTP operator has logic in __init__ (#29073) add 5da085bcb2 Upgrade bitnami/postgresql helm depenency from 10.5.3 to 12.1.9 (#29071) No new revisions were added by this update. Summary of changes: BREEZE.rst| 2 +- chart/Chart.lock | 7 +++--- chart/Chart.yaml | 3 ++- chart/charts/postgresql-10.5.3.tgz| Bin 50647 -> 0 bytes chart/charts/postgresql-12.1.9.tgz| Bin 0 -> 57966 bytes chart/templates/NOTES.txt | 6 + chart/values.schema.json | 44 +++--- chart/values.yaml | 7 -- newsfragments/27353.significant.rst | 8 +++ tests/charts/test_basic_helm_chart.py | 4 ++-- tests/charts/test_rbac.py | 6 - 11 files changed, 68 insertions(+), 19 deletions(-) delete mode 100644 chart/charts/postgresql-10.5.3.tgz create mode 100644 chart/charts/postgresql-12.1.9.tgz create mode 100644 newsfragments/27353.significant.rst
[GitHub] [airflow] potiuk closed issue #27353: Upgrade to the latest version of bitnami/postgresql
potiuk closed issue #27353: Upgrade to the latest version of bitnami/postgresql URL: https://github.com/apache/airflow/issues/27353 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #29071: Upgrade bitnami/postgresql helm depenency from 10.5.3 to 12.1.9
potiuk merged PR #29071: URL: https://github.com/apache/airflow/pull/29071 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #16618: Access specific values within an `XCom` value using Taskflow API
jedcunningham commented on issue #16618: URL: https://github.com/apache/airflow/issues/16618#issuecomment-1399101632 @ash, @josh-fell and I had a chat about this. One approach that may work out is having a `output.return_value`, where `[]` get recorded in the `XcomArg` instead of changing the key. Basically `essentially output.return_value['outer']['inner'] == XComArg(key="return_value", path=["outer", "inner"])`. (@ash please correct me if I misunderstood). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #29073: FTP operator has logic in __init__
potiuk merged PR #29073: URL: https://github.com/apache/airflow/pull/29073 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (8eb348911f -> 2b7071c600)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 8eb348911f Mark `test_process_sigterm_works_with_retries` quarantined again (#29075) add 2b7071c600 FTP operator has logic in __init__ (#29073) No new revisions were added by this update. Summary of changes: airflow/providers/ftp/operators/ftp.py| 50 +-- airflow/providers/sftp/operators/sftp.py | 4 +-- tests/providers/ftp/operators/test_ftp.py | 4 +-- 3 files changed, 30 insertions(+), 28 deletions(-)
[GitHub] [airflow] potiuk closed issue #29070: FTP operator has logic in __init__
potiuk closed issue #29070: FTP operator has logic in __init__ URL: https://github.com/apache/airflow/issues/29070 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow-site] branch gh-pages updated: Deploying to gh-pages from @ 966804c0a1779876ddc8301778e7b041a05dd36c
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch gh-pages in repository https://gitbox.apache.org/repos/asf/airflow-site.git The following commit(s) were added to refs/heads/gh-pages by this push: new 13960ba868 Deploying to gh-pages from @ 966804c0a1779876ddc8301778e7b041a05dd36c 13960ba868 is described below commit 13960ba868e7df41c731bdaeb8f3501b78c5 Author: potiuk AuthorDate: Sat Jan 21 00:28:32 2023 + Deploying to gh-pages from @ 966804c0a1779876ddc8301778e7b041a05dd36c --- blog/airflow-1.10.10/index.html| 4 +- blog/airflow-1.10.12/index.html| 4 +- blog/airflow-1.10.8-1.10.9/index.html | 4 +- blog/airflow-2.2.0/index.html | 4 +- blog/airflow-2.3.0/index.html | 4 +- blog/airflow-2.4.0/index.html | 4 +- blog/airflow-2.5.0/index.html | 4 +- blog/airflow-survey-2020/index.html| 4 +- blog/airflow-survey-2022/index.html| 4 +- blog/airflow-survey/index.html | 4 +- blog/airflow-two-point-oh-is-here/index.html | 4 +- blog/airflow_summit_2021/index.html| 4 +- blog/airflow_summit_2022/index.html| 4 +- blog/announcing-new-website/index.html | 4 +- blog/apache-airflow-for-newcomers/index.html | 4 +- .../index.html | 4 +- .../index.html | 4 +- .../index.html | 4 +- .../index.html | 4 +- .../index.html | 4 +- .../index.html | 4 +- index.html | 32 +++ meetups.json | 36 +--- search/index.html | 4 +- sitemap.xml| 100 ++--- use-cases/adobe/index.html | 4 +- use-cases/adyen/index.html | 4 +- use-cases/big-fish-games/index.html| 4 +- use-cases/dish/index.html | 4 +- use-cases/experity/index.html | 4 +- use-cases/onefootball/index.html | 4 +- use-cases/plarium-krasnodar/index.html | 4 +- use-cases/seniorlink/index.html| 4 +- use-cases/sift/index.html | 4 +- 34 files changed, 153 insertions(+), 139 deletions(-) diff --git a/blog/airflow-1.10.10/index.html b/blog/airflow-1.10.10/index.html index 969cabc29a..856a1743cc 100644 --- a/blog/airflow-1.10.10/index.html +++ b/blog/airflow-1.10.10/index.html @@ -34,11 +34,11 @@ - + - + diff --git a/blog/airflow-1.10.12/index.html b/blog/airflow-1.10.12/index.html index fa6f015aa4..750974123c 100644 --- a/blog/airflow-1.10.12/index.html +++ b/blog/airflow-1.10.12/index.html @@ -34,11 +34,11 @@ - + - + diff --git a/blog/airflow-1.10.8-1.10.9/index.html b/blog/airflow-1.10.8-1.10.9/index.html index b5925c7261..1615dd99f6 100644 --- a/blog/airflow-1.10.8-1.10.9/index.html +++ b/blog/airflow-1.10.8-1.10.9/index.html @@ -34,11 +34,11 @@ - + - + diff --git a/blog/airflow-2.2.0/index.html b/blog/airflow-2.2.0/index.html index 02fc4361f4..ad5d898c53 100644 --- a/blog/airflow-2.2.0/index.html +++ b/blog/airflow-2.2.0/index.html @@ -34,11 +34,11 @@ - + - + diff --git a/blog/airflow-2.3.0/index.html b/blog/airflow-2.3.0/index.html index 4ebd0aa009..2be11a14d4 100644 --- a/blog/airflow-2.3.0/index.html +++ b/blog/airflow-2.3.0/index.html @@ -34,11 +34,11 @@ - + - + diff --git a/blog/airflow-2.4.0/index.html b/blog/airflow-2.4.0/index.html index 53449a3bfd..3a1517e013 100644 --- a/blog/airflow-2.4.0/index.html +++ b/blog/airflow-2.4.0/index.html @@ -34,11 +34,11 @@ - + - + diff --git a/blog/airflow-2.5.0/index.html b/blog/airflow-2.5.0/index.html index 78c66ec6a8..8fbba2d10e 100644 --- a/blog/airflow-2.5.0/index.html +++ b/blog/airflow-2.5.0/index.html @@ -34,11 +34,11 @@ - + - + diff --git a/blog/airflow-survey-2020/index.html b/blog/airflow-survey-2020/index.html index 05b165de92..795125a00b 100644 --- a/blog/airflow-survey-2020/index.html +++ b/blog/airflow-survey-2020/index.html @@ -34,11 +34,11 @@ - + - + diff --git a/blog/airflow-survey-2022/index.html b/blog/airflow-survey-2022/index.html index ef20f4adf7..ab88cc5ec2 100644 --- a/blog/airflow-survey-2022/index.html +++ b/blog/airflow-survey-2022/index.html @@ -34,11 +34,11 @@ - + - + diff --git a/blog/airflow-survey/index.html b/blog/airflow-survey/index.html index 21619ecec3..19a9cbc016
[airflow] branch main updated (49ad77c50a -> 8eb348911f)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 49ad77c50a Refactor `TestLocalTaskJob.test_process_sigterm_works_with_retries` (#28990) add 8eb348911f Mark `test_process_sigterm_works_with_retries` quarantined again (#29075) No new revisions were added by this update. Summary of changes: tests/jobs/test_local_task_job.py | 1 + 1 file changed, 1 insertion(+)
[GitHub] [airflow] potiuk merged pull request #29075: Mark `test_process_sigterm_works_with_retries` quarantined again
potiuk merged PR #29075: URL: https://github.com/apache/airflow/pull/29075 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow-site] branch main updated: Update Meetups page content (#726)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
The following commit(s) were added to refs/heads/main by this push:
new 966804c0a1 Update Meetups page content (#726)
966804c0a1 is described below
commit 966804c0a1779876ddc8301778e7b041a05dd36c
Author: John Thomas <49878111+tohnjho...@users.noreply.github.com>
AuthorDate: Fri Jan 20 17:18:43 2023 -0700
Update Meetups page content (#726)
* Update Meetups page content
* fix json indentation
---
landing-pages/site/static/meetups.json | 36 +++---
1 file changed, 25 insertions(+), 11 deletions(-)
diff --git a/landing-pages/site/static/meetups.json
b/landing-pages/site/static/meetups.json
index fbb216504d..17aeff1b8d 100644
--- a/landing-pages/site/static/meetups.json
+++ b/landing-pages/site/static/meetups.json
@@ -3,77 +3,91 @@
"city": "Amsterdam",
"country": "Netherlands",
"date": "",
-"members": 378,
+"members": 400,
"url": "https://www.meetup.com/Amsterdam-Airflow-meetup/;
},
{
"city": "Tel Aviv-Yafo",
"country": "Israel",
"date": "",
-"members": 40,
+"members": 843,
"url": "https://www.meetup.com/tel-aviv-apache-airflow-meetup/;
},
{
"city": "San Francisco",
"country": "USA",
"date": "",
-"members": 1855,
+"members": 1927,
"url": "https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/;
},
{
"city": "London",
"country": "United Kingdom",
"date": "",
-"members": 728,
+"members": 898,
"url": "https://www.meetup.com/London-Apache-Airflow-Meetup/;
},
{
"city": "New York",
"country": "USA",
"date": "",
-"members": 1181,
+"members": 1469,
"url": "https://www.meetup.com/NYC-Apache-Airflow-Meetup/;
},
+ {
+"city": "Washington D.C.",
+"country": "USA",
+"date": "",
+"members": 40,
+"url": "https://www.meetup.com/dc-area-apache-airflow-meetup/;
+ },
+ {
+"city": "Chicago",
+"country": "USA",
+"date": "",
+"members": 49,
+"url": "https://www.meetup.com/chicago-apache-airflow-meetup-group/;
+ },
{
"city": "Melbourne",
"country": "Australia",
"date": "",
-"members": 159,
+"members": 200,
"url": "https://www.meetup.com/Melbourne-Apache-Airflow-Meetup/;
},
{
"city": "Paris",
"country": "France",
"date": "",
-"members": 602,
+"members": 938,
"url": "https://www.meetup.com/Paris-Apache-Airflow-Meetup/;
},
{
"city": "Portland",
"country": "USA",
"date": "",
-"members": 73,
+"members": 91,
"url": "https://www.meetup.com/Portland-Apache-Airflow-Meetup/;
},
{
"city": "Tokyo",
"country": "Japan",
"date": "",
-"members": 134,
+"members": 176,
"url": "https://www.meetup.com/Tokyo-Apache-Airflow-incubating-Meetup/;
},
{
"city": "Bangalore",
"country": "India",
"date": "",
-"members": 733,
+"members": 1400,
"url": "https://www.meetup.com/Bangalore-Apache-Airflow-Meetup/;
},
{
"city": "Warsaw",
"country": "Poland",
"date": "",
-"members": 390,
+"members": 456,
"url": "https://www.meetup.com/Warsaw-Airflow-Meetup/;
}
]
[GitHub] [airflow-site] potiuk merged pull request #726: Update Meetups page content
potiuk merged PR #726: URL: https://github.com/apache/airflow-site/pull/726 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #27762: Remove get_db_hook method
github-actions[bot] closed pull request #27762: Remove get_db_hook method URL: https://github.com/apache/airflow/pull/27762 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #27947: refactored Amazon Redshift-data functionality into the hook
github-actions[bot] commented on PR #27947: URL: https://github.com/apache/airflow/pull/27947#issuecomment-1399092128 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #28141: Fix GoogleDriveHook writing files to trashed folders on upload
github-actions[bot] commented on PR #28141: URL: https://github.com/apache/airflow/pull/28141#issuecomment-1399092106 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #28524: Use only parameters for client_type that are required
github-actions[bot] commented on issue #28524: URL: https://github.com/apache/airflow/issues/28524#issuecomment-1399092083 This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] hussein-awala opened a new pull request, #29076: [FIX] emit dagrun duration when it fails because of a timeout
hussein-awala opened a new pull request, #29076:
URL: https://github.com/apache/airflow/pull/29076
closes: #29013
---
**^ Add meaningful description above**
Currently the duration of the finished dagruns are emitted in the method
`update_state`, which is called only if the dagrun is not timeout, and we don't
emit any metric when the dagrun is timeout.
This PR add the missing `Stats.timing` to the timeout case.
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #29075: Mark `test_process_sigterm_works_with_retries` quarantined again
Taragolis commented on PR #29075: URL: https://github.com/apache/airflow/pull/29075#issuecomment-1399085492 Oh... this check not make any sense, we initially call all callbacks and only after that we commit to query. https://github.com/apache/airflow/blob/49ad77c50aa0141123ca1f2cc8c9e904b3b400c1/airflow/models/taskinstance.py#L1859-L1870 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #28658: Fix Quarantine tests
Taragolis commented on issue #28658: URL: https://github.com/apache/airflow/issues/28658#issuecomment-1399079589 #29075 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] eladkal opened a new issue, #28658: Fix Quarantine tests
eladkal opened a new issue, #28658: URL: https://github.com/apache/airflow/issues/28658 ### Body We have several tests marked in the code base with `@pytest.mark.quarantined` It means that the tests are flaky and if fail in CI it does not fail the build. The goal is to fix the tests and make them stable. This task is to gather all of them under the same issue instead of dedicated issue per test. - [x] [TestImpersonation](https://github.com/apache/airflow/blob/bfcae349b88fd959e32bfacd027a5be976fe2132/tests/core/test_impersonation_tests.py#L117) - [x] [TestImpersonationWithCustomPythonPath](https://github.com/apache/airflow/blob/bfcae349b88fd959e32bfacd027a5be976fe2132/tests/core/test_impersonation_tests.py#L181) - [x] [test_exception_propagation](https://github.com/apache/airflow/blob/76f81cd4a7433b7eeddb863b2ae6ee59176cf816/tests/jobs/test_local_task_job.py#L772) - [x] [test_localtaskjob_maintain_heart_rate](https://github.com/apache/airflow/blob/76f81cd4a7433b7eeddb863b2ae6ee59176cf816/tests/jobs/test_local_task_job.py#L402) - [x] [test_exception_propagation](https://github.com/apache/airflow/blob/4d0fd8ef6adc35f683c7561f05688a65fd7451f4/tests/executors/test_celery_executor.py#L103) - [ ] [test_process_sigterm_works_with_retries](https://github.com/apache/airflow/blob/65010fda091242870a410c65478eae362899763b/tests/jobs/test_local_task_job.py#L770) ### Committer - [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Taragolis opened a new pull request, #29075: Mark `test_process_sigterm_works_with_retries` quarantined again
Taragolis opened a new pull request, #29075: URL: https://github.com/apache/airflow/pull/29075 Unfortunately this test still not stable and first run 臘♂️ 臘♂️ 臘♂️ in main failed again. https://github.com/apache/airflow/actions/runs/3971462849/jobs/6808560333: ```console finally: proc.kill() assert retry_callback_called.value == 1 with create_session() as session: ti = ( session.query(TaskInstance) .filter( TaskInstance.dag_id == dag_id, TaskInstance.task_id == task_id, TaskInstance.run_id == run_id, ) .one() ) > assert ti.state == State.UP_FOR_RETRY E AssertionError: assert 'running' == E - up_for_retry E + running tests/jobs/test_local_task_job.py:895: AssertionError ``` This error basically mean that we send SIGKILL `proc.kill()` before state changed but callback already executed. Note: We do not check state in the previous version of the test, only callback result -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jedcunningham commented on pull request #27848: git-sync 3.6.3
jedcunningham commented on PR #27848: URL: https://github.com/apache/airflow/pull/27848#issuecomment-1399076067 @danielhoherd, can you add a newsfragement for this ([here is an example](https://github.com/apache/airflow/blob/main/chart/newsfragments/28074.significant.rst))? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #29062: Several improvements to the Params doc
jedcunningham commented on code in PR #29062:
URL: https://github.com/apache/airflow/pull/29062#discussion_r1083133413
##
docs/apache-airflow/core-concepts/params.rst:
##
@@ -42,15 +42,31 @@ Use a dictionary that maps Param names to a either a
:class:`~airflow.models.par
"x": Param(5, type="integer", minimum=3),
"y": 6
},
-) as the_dag:
+):
+
+Task-level Params
+-
+
+You can also add Params to individual tasks.
+
+.. code-block::
+
+PythonOperator(
+task_id="print_x",
+params={"x": 10},
+python_callable=print_it,
+)
+
+Task-level params take precedence over DAG-level params, and user-supplied
params (when triggering the DAG)
+take precedence over task-level params.
Referencing Params in a Task
-Params are stored as ``params`` in the :ref:`template context `.
-So you can reference them in a template.
+Params can be referenced in :ref:`templated strings `. For
example:
Review Comment:
```suggestion
Params can be referenced in :ref:`templated strings ` under
``params``. For example:
```
I think this is a key part, but feel free to reword it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] hussein-awala commented on pull request #28713: Add a param for get_dags endpoint to list only unpaused dags
hussein-awala commented on PR #28713: URL: https://github.com/apache/airflow/pull/28713#issuecomment-1399067293 @ashb I just added the tests, can you re-check please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #29074: Update Airflow version to 2.5.1
jedcunningham commented on code in PR #29074: URL: https://github.com/apache/airflow/pull/29074#discussion_r1083087956 ## RELEASE_NOTES.rst: ## @@ -21,6 +21,119 @@ .. towncrier release notes start +Airflow 2.5.1 (2023-01-20) +-- + +Significant Changes +^^^ + +Trigger gevent ``monkeypatching`` via environment variable (#28283) Review Comment: You also should delete the newsfragment for this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] pierrejeambrun opened a new pull request, #29074: Update Airflow version to 2.5.1
pierrejeambrun opened a new pull request, #29074: URL: https://github.com/apache/airflow/pull/29074 Update airflow version for main following the 2.5.1 release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] rkarish commented on a diff in pull request #29034: Add functionality for operators to template all eligible fields (apac…
rkarish commented on code in PR #29034: URL: https://github.com/apache/airflow/pull/29034#discussion_r1083085422 ## airflow/models/baseoperator.py: ## @@ -1184,7 +1197,29 @@ def render_template_fields( """ if not jinja_env: jinja_env = self.get_template_env() -self._do_render_template_fields(self, self.template_fields, context, jinja_env, set()) + +if self.template_all_fields: +templatable_fields = ( +set(self.__dict__.keys()) +- set(BaseOperator.__dict__.keys()) +- set(partial(operator_class=self.__class__, task_id=self.task_id).kwargs.keys()) +) Review Comment: Was wondering if anyone has any comment on how I did this. Goal was to get attributes that belong only to child classes of `BaseOperator`, would like to know if this approach is satisfactory, or if there is a better way to accomplish it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #28981: Implemented log_containers option to read from all containers in KubernetesPodOperator
dstandish commented on code in PR #28981:
URL: https://github.com/apache/airflow/pull/28981#discussion_r1083084084
##
airflow/providers/cncf/kubernetes/utils/pod_manager.py:
##
@@ -264,15 +279,69 @@ def consume_logs(*, since_time: DateTime | None = None,
follow: bool = True) ->
)
time.sleep(1)
-def await_container_completion(self, pod: V1Pod, container_name: str) ->
None:
+def fetch_input_container_logs(
+self, pod: V1Pod, log_containers: list[str] | bool, follow=False
+) -> list[PodLoggingStatus]:
+"""
+Follow the logs of containers in the pod specified by input parameter
and stream to airflow logging.
+Returns when all the containers exit
+"""
+pod_logging_statuses = []
+all_container_names = self.get_container_names(pod)
+if len(all_container_names) == 0:
+self.log.error("Failed to retrieve container names from the pod,
unable to collect logs")
+else:
+# if log_containers is list type, collect logs of the input
container names
+if type(log_containers) == list:
+for container_name in log_containers:
+if container_name in all_container_names:
+status = self.fetch_container_logs(
+pod=pod, container_name=container_name,
follow=follow
+)
+pod_logging_statuses.append(status)
+else:
+self.log.error(
+"container name '%s' specified in input parameter
is not found in the pod",
+container_name,
+)
+# if log_containers is bool value True, collect logs from all
containers
+# if log_containers is bool value False, collect logs from the
base (first) container
+elif type(log_containers) == bool:
+if log_containers is True:
+for container_name in all_container_names:
+status = self.fetch_container_logs(
+pod=pod, container_name=container_name,
follow=follow
+)
+pod_logging_statuses.append(status)
+else:
+status = self.fetch_container_logs(
+pod=pod, container_name=all_container_names[0],
follow=follow
+)
+pod_logging_statuses.append(status)
Review Comment:
@mlnsharma what is the expected behavior here when there are multiple
containers? Will it follow just _one_ container until done and then jump to
next container? That would seem to be tad bit problematic because you won't
get all container logs continuously. What if you are following logs from a
very uninteresting sidecar?
If that's true, just to think of alternatives, we could imagine launching a
"logs follower" (perhaps simply a callable) in a thread for each container,
that would funnel messages back to main thread via a queue. The main thread
would just consume the queue and emit messages from it as they arrive until all
containers are done.
And if only one container, no thread. WDYT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] rkarish commented on a diff in pull request #29034: Add functionality for operators to template all eligible fields (apac…
rkarish commented on code in PR #29034: URL: https://github.com/apache/airflow/pull/29034#discussion_r1083083888 ## airflow/models/baseoperator.py: ## @@ -1184,7 +1197,29 @@ def render_template_fields( """ if not jinja_env: jinja_env = self.get_template_env() -self._do_render_template_fields(self, self.template_fields, context, jinja_env, set()) + +if self.template_all_fields: +templatable_fields = ( Review Comment: Was wondering if anyone has any comment on how I did this. Goal was to get attributes that belong only to child classes of `BaseOperator`, would like to know if this approach is satisfactory, or if there is a better way to accomplish it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] rkarish commented on pull request #29034: Add functionality for operators to template all eligible fields (apac…
rkarish commented on PR #29034: URL: https://github.com/apache/airflow/pull/29034#issuecomment-1398997523 Hi Jarek, quick update on this. The failing static checks was an issue in main, it was resolved with a rebase. All other failures were related to my code changes and I've resolved those failing tests by updating some code in the serialization package. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] snjypl opened a new pull request, #29073: FTP operator has logic in __init__
snjypl opened a new pull request, #29073:
URL: https://github.com/apache/airflow/pull/29073
Fixes: https://github.com/apache/airflow/issues/29070
Move the logic from __init__ to execute for FTP operator
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29070: FTP operator has logic in __init__
potiuk commented on issue #29070: URL: https://github.com/apache/airflow/issues/29070#issuecomment-1398987908 did :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28271: AIP-44 Migrate Variable to Internal API
potiuk commented on issue #28271: URL: https://github.com/apache/airflow/issues/28271#issuecomment-1398983633 Changed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] snjypl commented on issue #29070: FTP operator has logic in __init__
snjypl commented on issue #29070: URL: https://github.com/apache/airflow/issues/29070#issuecomment-1398981934 @potiuk can you please assign this to me? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28990: Refactor `TestLocalTaskJob.test_process_sigterm_works_with_retries`
potiuk commented on PR #28990: URL: https://github.com/apache/airflow/pull/28990#issuecomment-1398981894 stable enough :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #28990: Refactor `TestLocalTaskJob.test_process_sigterm_works_with_retries`
potiuk merged PR #28990: URL: https://github.com/apache/airflow/pull/28990 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (100bb8d79a -> 49ad77c50a)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 100bb8d79a listener plugin example added (#27905) add 49ad77c50a Refactor `TestLocalTaskJob.test_process_sigterm_works_with_retries` (#28990) No new revisions were added by this update. Summary of changes: tests/jobs/test_local_task_job.py | 165 +- 1 file changed, 125 insertions(+), 40 deletions(-)
[GitHub] [airflow] potiuk closed issue #28658: Fix Quarantine tests
potiuk closed issue #28658: Fix Quarantine tests URL: https://github.com/apache/airflow/issues/28658 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] danielhoherd commented on a diff in pull request #27848: git-sync 3.6.3
danielhoherd commented on code in PR #27848: URL: https://github.com/apache/airflow/pull/27848#discussion_r1083071160 ## chart/values.yaml: ## @@ -93,7 +93,7 @@ images: pullPolicy: IfNotPresent gitSync: repository: k8s.gcr.io/git-sync/git-sync -tag: v3.4.0 +tag: v3.6.1 Review Comment: OK, I undid everything except the tag version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #27905: listener plugin example added
potiuk merged PR #27905: URL: https://github.com/apache/airflow/pull/27905 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (c0d9862d76 -> 100bb8d79a)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
from c0d9862d76 Emit DataDog statsd metrics with metadata tags (#28961)
add 100bb8d79a listener plugin example added (#27905)
No new revisions were added by this update.
Summary of changes:
airflow/example_dags/plugins/event_listener.py | 156 +
.../plugins/listener_plugin.py}| 12 +-
docs/apache-airflow/howto/index.rst| 1 +
docs/apache-airflow/howto/listener-plugin.rst | 95 +
docs/spelling_wordlist.txt | 1 +
5 files changed, 259 insertions(+), 6 deletions(-)
create mode 100644 airflow/example_dags/plugins/event_listener.py
copy airflow/{providers/amazon/aws/utils/rds.py =>
example_dags/plugins/listener_plugin.py} (78%)
create mode 100644 docs/apache-airflow/howto/listener-plugin.rst
[GitHub] [airflow] pierrejeambrun closed issue #29026: Status of testing of Apache Airflow 2.5.1rc2
pierrejeambrun closed issue #29026: Status of testing of Apache Airflow 2.5.1rc2 URL: https://github.com/apache/airflow/issues/29026 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow-site] potiuk commented on pull request #726: Update Meetups page content
potiuk commented on PR #726: URL: https://github.com/apache/airflow-site/pull/726#issuecomment-1398967947 pre-commit/liniting issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (326b5d9791 -> c0d9862d76)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 326b5d9791 Fix minor typo in breeze example command. (#29072) add c0d9862d76 Emit DataDog statsd metrics with metadata tags (#28961) No new revisions were added by this update. Summary of changes: airflow/config_templates/config.yml | 7 +++ airflow/config_templates/default_airflow.cfg | 3 ++ airflow/configuration.py | 1 + airflow/dag_processing/manager.py| 14 ++--- airflow/dag_processing/processor.py | 18 +-- airflow/jobs/scheduler_job.py| 9 +++- airflow/models/dag.py| 4 +- airflow/models/taskinstance.py | 15 -- airflow/stats.py | 77 +--- tests/core/test_stats.py | 21 ++-- tests/dag_processing/test_processor.py | 23 ++--- tests/jobs/test_scheduler_job.py | 25 +++-- tests/models/test_dag.py | 5 +- tests/models/test_taskinstance.py| 4 +- 14 files changed, 164 insertions(+), 62 deletions(-)
[GitHub] [airflow] potiuk merged pull request #28961: Emit DataDog statsd metrics with metadata tags
potiuk merged PR #28961: URL: https://github.com/apache/airflow/pull/28961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] utkarsharma2 commented on a diff in pull request #28934: Remove hard-coded executor-database coupling
utkarsharma2 commented on code in PR #28934:
URL: https://github.com/apache/airflow/pull/28934#discussion_r1083060525
##
airflow/cli/commands/standalone_command.py:
##
@@ -159,14 +159,28 @@ def calculate_env(self):
# Make sure we're using a local executor flavour
executor_class, _ = ExecutorLoader.import_default_executor_cls()
if not executor_class.is_local:
-if "sqlite" in conf.get("database", "sql_alchemy_conn"):
-self.print_output("standalone", "Forcing executor to
SequentialExecutor")
-env["AIRFLOW__CORE__EXECUTOR"] =
executor_constants.SEQUENTIAL_EXECUTOR
-else:
-self.print_output("standalone", "Forcing executor to
LocalExecutor")
-env["AIRFLOW__CORE__EXECUTOR"] =
executor_constants.LOCAL_EXECUTOR
+env["AIRFLOW__CORE__EXECUTOR"] = self.get_local_executor(
+database=conf.get("database", "sql_alchemy_conn")
+)
+self.print_output("standalone", f"Forcing executor to
{env['AIRFLOW__CORE__EXECUTOR']}")
Review Comment:
@o-nikolas Thanks for the context, this makes sense now. I'll revoke these
changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #29018: Able to pass arguments to `--conf` not specified in `parameters` argument
potiuk commented on issue #29018: URL: https://github.com/apache/airflow/issues/29018#issuecomment-1398961154 yes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow-site] TohnJhomas opened a new pull request, #726: Update Meetups page content
TohnJhomas opened a new pull request, #726: URL: https://github.com/apache/airflow-site/pull/726 Adding Chicago and Washington DC meetup groups, update group membership numbers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mrn-aglic closed issue #29067: TaskGroup dynamic task mapping fails to populate all mapping metadata
mrn-aglic closed issue #29067: TaskGroup dynamic task mapping fails to populate all mapping metadata URL: https://github.com/apache/airflow/issues/29067 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mrn-aglic commented on issue #29067: TaskGroup dynamic task mapping fails to populate all mapping metadata
mrn-aglic commented on issue #29067: URL: https://github.com/apache/airflow/issues/29067#issuecomment-1398952175 Seems to be working in 2.5.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (e926bb9bb0 -> 326b5d9791)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from e926bb9bb0 Add deferrable mode to DataFusionStartPipelineOperator (#28690) add 326b5d9791 Fix minor typo in breeze example command. (#29072) No new revisions were added by this update. Summary of changes: BREEZE.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[GitHub] [airflow] potiuk merged pull request #29072: Fix minor typo in breeze example command.
potiuk merged PR #29072: URL: https://github.com/apache/airflow/pull/29072 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
