Re: [I] ArgoCD reverting the app to Missing, OutOfSync even when successfully deployed [airflow]

[via GitHub]

potiuk closed issue #37460: ArgoCD reverting the app to Missing, OutOfSync even 
when successfully deployed
URL: https://github.com/apache/airflow/issues/37460


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] airflow db upgrade: FOREIGN KEY constraint failed [airflow]

[via GitHub]

github-actions[bot] commented on issue #23453:
URL: https://github.com/apache/airflow/issues/23453#issuecomment-1947855033

   This issue has been automatically marked as stale because it has been open 
for 365 days without any activity. There has been several Airflow releases 
since last activity on this issue. Kindly asking to recheck the report against 
latest Airflow version and let us know if the issue is reproducible. The issue 
will be closed in next 30 days if no further activity occurs from the issue 
author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] ArgoCD reverting the app to Missing, OutOfSync even when successfully deployed [airflow]

[via GitHub]

potiuk commented on issue #37460:
URL: https://github.com/apache/airflow/issues/37460#issuecomment-1947846025

   Yep I think discussion is better.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Kubernetes executor pod reconcilation is slow due to recurssive deep copy [airflow]

[via GitHub]

potiuk commented on issue #37131:
URL: https://github.com/apache/airflow/issues/37131#issuecomment-1947843819

   It's just another way to achieve isolation (heavy handed, so maybe 
side-edlffects will be too big and it's a bad idea). 
   
   Basically the reason we are doing deep copy here is because we want to have 
objects in memory that we want to modify but we want to keep the original 
object intact. This is what we really achieve by deepcopy. And apparently the 
objects we deepcopy are so complex that it takes couple of seconds to do (which 
I find quite strange on its own but I trust the original diagnosis - it would 
have to be likely many 100s of objects and rather complex structure to get few 
seconds here). But yeah, those objects are likely complex.
   
   So if we are talking about out-of-the box thinkinging and saving the cost of 
deepcopying - forking process does the same thing - it allows to have 
separate.process with the same object in memory which is effectively a copy, 
and the memory segments for the object in question on heap will get duplicated 
the moment you write them (which should be much faster than copying a 
complex.object structure in Python as you copy all affected memory segments 
using low-level processor instructions). Effect is the same - you have a copy 
of the object that you can modify without affecting the original object.
   
   So my proposal is just one of the solution to 'let's get an isolated copy of 
the object that is long to copy'.
   
   Very remote and out-of-the-box and can have other side effects - but maybe 
it's something we are looking for.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] add endpoint to delete DatasetDagRunQueue [airflow]

[via GitHub]

Lee-W commented on code in PR #37176:
URL: https://github.com/apache/airflow/pull/37176#discussion_r1492014740


##
airflow/providers/fab/auth_manager/security_manager/override.py:
##
@@ -276,6 +276,7 @@ class 
FabAirflowSecurityManagerOverride(AirflowSecurityManagerV2):
 (permissions.ACTION_CAN_EDIT, permissions.RESOURCE_VARIABLE),
 (permissions.ACTION_CAN_DELETE, permissions.RESOURCE_VARIABLE),
 (permissions.ACTION_CAN_DELETE, permissions.RESOURCE_XCOM),
+(permissions.ACTION_CAN_DELETE, permissions.RESOURCE_DATASET),

Review Comment:
   > does this mean a FAB provider upgrade would be necessary for the endpoints 
to work?
   
   I think so
   
   > Do we need to do something about this?
   
   Without this, `@security.requires_access_dataset("DELETE")` won't work



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Check permissions for ImportError [airflow]

[via GitHub]

jedcunningham commented on code in PR #37468:
URL: https://github.com/apache/airflow/pull/37468#discussion_r1492012822


##
airflow/api_connexion/endpoints/import_error_endpoint.py:
##
@@ -16,39 +16,59 @@
 # under the License.
 from __future__ import annotations
 
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Sequence
 
 from sqlalchemy import func, select
 
 from airflow.api_connexion import security
-from airflow.api_connexion.exceptions import NotFound
+from airflow.api_connexion.exceptions import NotFound, PermissionDenied
 from airflow.api_connexion.parameters import apply_sorting, check_limit, 
format_parameters
 from airflow.api_connexion.schemas.error_schema import (
 ImportErrorCollection,
 import_error_collection_schema,
 import_error_schema,
 )
-from airflow.auth.managers.models.resource_details import AccessView
+from airflow.auth.managers.models.resource_details import AccessView, 
DagDetails
+from airflow.models.dag import DagModel
 from airflow.models.errors import ImportError as ImportErrorModel
 from airflow.utils.session import NEW_SESSION, provide_session
+from airflow.www.extensions.init_auth_manager import get_auth_manager
 
 if TYPE_CHECKING:
 from sqlalchemy.orm import Session
 
 from airflow.api_connexion.types import APIResponse
+from airflow.auth.managers.models.batch_apis import IsAuthorizedDagRequest
 
 
 @security.requires_access_view(AccessView.IMPORT_ERRORS)
 @provide_session
 def get_import_error(*, import_error_id: int, session: Session = NEW_SESSION) 
-> APIResponse:
 """Get an import error."""
 error = session.get(ImportErrorModel, import_error_id)
-
 if error is None:
 raise NotFound(
 "Import error not found",
 detail=f"The ImportError with import_error_id: `{import_error_id}` 
was not found",
 )
+session.expunge(error)
+
+can_read_all_dags = get_auth_manager().is_authorized_dag(method="GET")
+if not can_read_all_dags:
+readable_dag_ids = security.get_readable_dags()
+file_dag_ids = {
+dag_id[0]
+for dag_id in 
session.query(DagModel.dag_id).filter(DagModel.fileloc == error.filename).all()
+}

Review Comment:
   I'm short circuiting complexity with this. If we know the user has read on 
all DAGs, we can avoid all of it.



##
airflow/api_connexion/endpoints/import_error_endpoint.py:
##
@@ -65,10 +85,42 @@ def get_import_errors(
 """Get all import errors."""
 to_replace = {"import_error_id": "id"}
 allowed_filter_attrs = ["import_error_id", "timestamp", "filename"]
-total_entries = session.scalars(func.count(ImportErrorModel.id)).one()
+count_query = select(func.count(ImportErrorModel.id))
 query = select(ImportErrorModel)
 query = apply_sorting(query, order_by, to_replace, allowed_filter_attrs)
+
+can_read_all_dags = get_auth_manager().is_authorized_dag(method="GET")
+
+if not can_read_all_dags:
+# if the user doesn't have access to all DAGs, only display errors 
from visible DAGs
+readable_dag_ids = security.get_readable_dags()
+query = query.join(DagModel, DagModel.fileloc == 
ImportErrorModel.filename).where(
+DagModel.dag_id.in_(readable_dag_ids)
+)
+count_query = count_query.join(DagModel, DagModel.fileloc == 
ImportErrorModel.filename).where(

Review Comment:
   This is the total count. The other query is paginated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] add endpoint to delete DatasetDagRunQueue [airflow]

[via GitHub]

uranusjr commented on code in PR #37176:
URL: https://github.com/apache/airflow/pull/37176#discussion_r1492007495


##
airflow/providers/fab/auth_manager/security_manager/override.py:
##
@@ -276,6 +276,7 @@ class 
FabAirflowSecurityManagerOverride(AirflowSecurityManagerV2):
 (permissions.ACTION_CAN_EDIT, permissions.RESOURCE_VARIABLE),
 (permissions.ACTION_CAN_DELETE, permissions.RESOURCE_VARIABLE),
 (permissions.ACTION_CAN_DELETE, permissions.RESOURCE_XCOM),
+(permissions.ACTION_CAN_DELETE, permissions.RESOURCE_DATASET),

Review Comment:
   Hmm, does this mean a FAB provider upgrade would be necessary for the 
endpoints to work?Do we need to do something about this? Documentation? No 
quite sure how this works. cc @vincbeck 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Partially enable `D100`, `D102`, `D103` and `D104` [airflow]

[via GitHub]

rawwar commented on code in PR #37469:
URL: https://github.com/apache/airflow/pull/37469#discussion_r1491968585


##
airflow/security/utils.py:
##


Review Comment:
   So, is it safe to remove from the modules?  If so, I can check for all the 
modules where E402 is ignored and probably move the license text to a separate 
license file. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] ERROR: could not queue task TaskInstanceKey [airflow]

[via GitHub]

harshith-bolar-rapido commented on issue #20047:
URL: https://github.com/apache/airflow/issues/20047#issuecomment-1947761572

   Seeing this in airflow 2.5.3


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] ArgoCD reverting the app to Missing, OutOfSync even when successfully deployed [airflow]

[via GitHub]

jaruji commented on issue #37460:
URL: https://github.com/apache/airflow/issues/37460#issuecomment-1947700115

   I'm mainly opening the issue to see whether somebody else encountered 
something similar and managed to fix it. I do not have any expectations 
regarding you doing the necessary research and figuring it out for me. Guess 
this can be converted into a discussion if you feel like an issue is not 
appropriate in this case.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Kubernetes executor pod reconcilation is slow due to recurssive deep copy [airflow]

[via GitHub]

dirrao commented on issue #37131:
URL: https://github.com/apache/airflow/issues/37131#issuecomment-1947695416

   Copy on write is intuitive and makes sense. I would look into the options.
   However, I believe the Kubernetes executor utils calls coming from either 
the scheduler (pod launch) or web server (pod spec rendering). I don't think 
it's being used anywhere else. I am just curious to check, why are we thinking 
about multi-process/threads in this case? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Core tests fail when run separately [airflow]

[via GitHub]

drajguru closed pull request #37467:  Core tests fail when run separately
URL: https://github.com/apache/airflow/pull/37467


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Core tests fail when run separately [airflow]

[via GitHub]

drajguru closed issue #37466: Core tests fail when run separately
URL: https://github.com/apache/airflow/issues/37466


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Core tests fail when run separately [airflow]

[via GitHub]

drajguru closed issue #37466: Core tests fail when run separately
URL: https://github.com/apache/airflow/issues/37466


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Core tests fail when run separately [airflow]

[via GitHub]

drajguru commented on issue #37466:
URL: https://github.com/apache/airflow/issues/37466#issuecomment-1947691550

   Oh I see... I just retested on the latest main and the issue is indeed now 
fixed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Partially enable `D100`, `D102`, `D103` and `D104` [airflow]

[via GitHub]

Taragolis commented on code in PR #37469:
URL: https://github.com/apache/airflow/pull/37469#discussion_r1491874137


##
airflow/security/utils.py:
##


Review Comment:
   And also think it listed in 3rd-party-licenses: 
https://github.com/apache/airflow/blob/main/3rd-party-licenses/LICENSE-hue.txt
   
   Added in
   https://github.com/apache/airflow/pull/438
   
   Source
   
https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/kt_renewer.py
   



##
airflow/security/utils.py:
##


Review Comment:
   And also think it listed in 3rd-party-licenses: 
https://github.com/apache/airflow/blob/main/3rd-party-licenses/LICENSE-hue.txt
   
   Added in https://github.com/apache/airflow/pull/438
   
   Source
   
https://github.com/cloudera/hue/blob/master/desktop/core/src/desktop/kt_renewer.py
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Partially enable `D100`, `D102`, `D103` and `D104` [airflow]

[via GitHub]

Taragolis commented on code in PR #37469:
URL: https://github.com/apache/airflow/pull/37469#discussion_r1491867626


##
airflow/api/auth/backend/kerberos_auth.py:
##
@@ -14,13 +14,9 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+"""Kerberos authentication module."""
 from __future__ import annotations
 
-import warnings

Review Comment:
   @potiuk I guess this license notice might be removed because it is already 
listed in 
https://github.com/apache/airflow/blob/main/3rd-party-licenses/LICENSE-flask-kerberos.txt



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Filter datasets by dag_id in rest API [airflow]

[via GitHub]

Satoshi-Sh commented on issue #37423:
URL: https://github.com/apache/airflow/issues/37423#issuecomment-1947589825

   I can take on this task.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Partially enable `D100`, `D102`, `D103` and `D104` [airflow]

[via GitHub]

rawwar commented on code in PR #37469:
URL: https://github.com/apache/airflow/pull/37469#discussion_r1491840368


##
airflow/api/auth/backend/kerberos_auth.py:
##
@@ -14,13 +14,9 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+"""Kerberos authentication module."""
 from __future__ import annotations
 
-import warnings

Review Comment:
   @potiuk , pre-commit hooks reformatted this module after I moved doc string 
to Line 17 from Line 48. Should I revert this change? 
   
   I thought instead of merging `E`*s and `D`*s , like you mentioned, I could 
just fix the conflicts. Reason being, I felt `E`*s (at least `E402`) is 
supposed to be ignored



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Add Teradata Provider [airflow]

[via GitHub]

SatishChGit commented on PR #36953:
URL: https://github.com/apache/airflow/pull/36953#issuecomment-1947589696

   Thank you every one for your support!. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Partially enable `D100`, `D102`, `D103` and `D104` [airflow]

[via GitHub]

rawwar commented on code in PR #37469:
URL: https://github.com/apache/airflow/pull/37469#discussion_r1491840368


##
airflow/api/auth/backend/kerberos_auth.py:
##
@@ -14,13 +14,9 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+"""Kerberos authentication module."""
 from __future__ import annotations
 
-import warnings

Review Comment:
   @potiuk , pre-commit hooks reformatted this module after I moved doc string 
to Line 17 from Line 48. Should I revert this change? 
   
   I thought instead of merging `E`*s and `D`*s , like you mentioned, I could 
just fix the conflicts. Reason being, I felt `E`*s can't be fixed 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Partially enable `D100`, `D102`, `D103` and `D104` [airflow]

[via GitHub]

Taragolis commented on code in PR #37469:
URL: https://github.com/apache/airflow/pull/37469#discussion_r1491844488


##
pyproject.toml:
##
@@ -1381,83 +1387,1472 @@ combine-as-imports = true
 "tests/providers/qdrant/hooks/test_qdrant.py" = ["E402"]
 "tests/providers/qdrant/operators/test_qdrant.py" = ["E402"]
 
-# All the modules which do not follow D105 yet, please remove as soon as it 
becomes compatible
-"airflow/callbacks/callback_requests.py" = ["D105"]
-"airflow/cli/commands/task_command.py" = ["D105"]
-"airflow/configuration.py" = ["D105"]
-"airflow/datasets/__init__.py" = ["D105"]
-"airflow/decorators/base.py" = ["D105"]
-"airflow/decorators/setup_teardown.py" = ["D105"]
+# All the modules which do not follow D100-D105 yet, please remove as soon as 
it becomes compatible
+"airflow/__main__.py" = ["D103"]
+"airflow/api/auth/__init__.py" = ["D104"]
+"airflow/api/auth/backend/__init__.py" = ["D104"]
+"airflow/api/auth/backend/basic_auth.py" = ["D103"]
+"airflow/api/client/local_client.py" = ["D102"]
+"airflow/api/common/__init__.py" = ["D104"]
+"airflow/api/common/airflow_health.py" = ["D100"]
+"airflow/api_connexion/__init__.py" = ["D104"]
+"airflow/api_connexion/endpoints/__init__.py" = ["D104"]
+"airflow/api_connexion/endpoints/config_endpoint.py" = ["D100", "D103"]
+"airflow/api_connexion/endpoints/connection_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/dag_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/dag_run_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/dag_source_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/dag_warning_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/dataset_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/event_log_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/extra_link_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/forward_to_fab_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/health_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/import_error_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/log_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/plugin_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/pool_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/provider_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/request_dict.py" = ["D100"]
+"airflow/api_connexion/endpoints/task_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/task_instance_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/update_mask.py" = ["D100", "D103"]
+"airflow/api_connexion/endpoints/variable_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/version_endpoint.py" = ["D100"]
+"airflow/api_connexion/endpoints/xcom_endpoint.py" = ["D100"]
+"airflow/api_connexion/exceptions.py" = ["D100"]
+"airflow/api_connexion/parameters.py" = ["D100"]
+"airflow/api_connexion/schemas/__init__.py" = ["D104"]
+"airflow/api_connexion/schemas/common_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/config_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/connection_schema.py" = ["D100", "D102"]
+"airflow/api_connexion/schemas/dag_run_schema.py" = ["D100", "D102"]
+"airflow/api_connexion/schemas/dag_schema.py" = ["D100", "D102"]
+"airflow/api_connexion/schemas/dag_source_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/dag_warning_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/dataset_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/enum_schemas.py" = ["D100"]
+"airflow/api_connexion/schemas/error_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/event_log_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/health_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/job_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/log_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/plugin_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/pool_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/provider_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/role_and_permission_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/sla_miss_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/task_instance_schema.py" = ["D100", "D102"]
+"airflow/api_connexion/schemas/task_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/trigger_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/user_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/variable_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/version_schema.py" = ["D100"]
+"airflow/api_connexion/schemas/xcom_schema.py" = ["D100"]
+"airflow/api_connexion/security.py" = ["D100", "D103"]
+"airflow/api_connexion/types.py" = ["D100"]
+"airflow/api_internal/__init__.py" = ["D104"]
+"airflow/api_internal/endpoints/__init__.py" = ["D104"]
+"airflow/api_internal/endpoints/rpc_api_endpoint.py" = ["D100"]
+"airflow/api_internal/internal_api_call.py" = ["D100", "D102"]
+"airflow/auth/__init__.py" = ["D104"]
+"airflow/auth/managers/__init__.py" = 

Re: [PR] Partially enable `D100`, `D102`, `D103` and `D104` [airflow]

[via GitHub]

rawwar commented on code in PR #37469:
URL: https://github.com/apache/airflow/pull/37469#discussion_r1491840368


##
airflow/api/auth/backend/kerberos_auth.py:
##
@@ -14,13 +14,9 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+"""Kerberos authentication module."""
 from __future__ import annotations
 
-import warnings

Review Comment:
   @potiuk , pre-commit hooks reformatted this module after I moved doc string 
to Line 17 from Line 48. Should I revert this change? 
   
   I thought instead of merging `E`*s and `D`*s , like you mentioned, I could 
just fix the conflicts



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Enable More PyDocStyle Checks [airflow]

[via GitHub]

Taragolis commented on issue #10742:
URL: https://github.com/apache/airflow/issues/10742#issuecomment-1947582702

   And personally not sure about D107


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Enable More PyDocStyle Checks [airflow]

[via GitHub]

rawwar commented on issue #10742:
URL: https://github.com/apache/airflow/issues/10742#issuecomment-1947582616

   
   > I don’t think it is even possible not to combine them if we want to enable 
all of them. Only problem to create issues with all of them
   
   I just raised this PR - https://github.com/apache/airflow/pull/37469 . can 
you please take a look. ruff did ignore multiple lint erros in a single line.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Partially enable `D100`, `D101`, `D102`, `D103` and `D104` [airflow]

[via GitHub]

rawwar commented on code in PR #37469:
URL: https://github.com/apache/airflow/pull/37469#discussion_r1491840368


##
airflow/api/auth/backend/kerberos_auth.py:
##
@@ -14,13 +14,9 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+"""Kerberos authentication module."""
 from __future__ import annotations
 
-import warnings

Review Comment:
   @potiuk , pre-commit hooks reformatted this module after I moved doc string 
to Line 17 from Line 48. Should I revert this change? 
   
   I thought, instead of merging `E`*s and `D`*s , like you mentioned, I could 
just fix the conflicts



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Enable More PyDocStyle Checks [airflow]

[via GitHub]

Taragolis commented on issue #10742:
URL: https://github.com/apache/airflow/issues/10742#issuecomment-1947578178

   > We could combine several `D*`s if we want in one row. I don't think there 
is a particular reason why we want to keep them in separate sections. 
@Taragolis ?
   
   I don’t think it is even possible not to combine them if we want to enable 
all of them. Only problem to create issues with all of them


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Reduce irrelevant ERROR level logs from KubernertesPodOperator on pod runtime failure [airflow]

[via GitHub]

sudiptob2 commented on issue #36077:
URL: https://github.com/apache/airflow/issues/36077#issuecomment-1947569963

   @ketozhang Thank you so much. There were problem in RBAC, I updated it and 
now I can get Pod events
   
   ```
   [2024-02-15, 23:26:26 UTC] {pod.py:980} ERROR - (403)
   [2024-02-15, 23:26:26 UTC] {taskinstance.py:2698} ERROR - Task failed with 
exception
   [2024-02-15, 23:26:26 UTC] {standard_task_runner.py:107} ERROR - Failed to 
execute job 2 for task minimal_test_fail_task_without_cr (Pod 
minimal-test-fail-task-without-cr-rp9u952i returned a failure.
   [2024-02-16, 00:45:26 UTC] {pod.py:732} ERROR - Pod Event: Scheduled - 
Successfully assigned default/minimal-test-fail-task-without-cr-kfnmnrcd to 
kind-worker
   [2024-02-16, 00:45:26 UTC] {pod.py:732} ERROR - Pod Event: Pulling - Pulling 
image "alpine:latest"
   [2024-02-16, 00:45:26 UTC] {pod.py:732} ERROR - Pod Event: Pulled - 
Successfully pulled image "alpine:latest" in 566ms (566ms including waiting)
   [2024-02-16, 00:45:26 UTC] {pod.py:732} ERROR - Pod Event: Created - Created 
container base
   [2024-02-16, 00:45:26 UTC] {pod.py:732} ERROR - Pod Event: Started - Started 
container base
   [2024-02-16, 00:45:26 UTC] {taskinstance.py:2698} ERROR - Task failed with 
exception
   [2024-02-16, 00:45:26 UTC] {standard_task_runner.py:107} ERROR - Failed to 
execute job 2 for task minimal_test_fail_task_without_cr (Pod 
minimal-test-fail-task-without-cr-kfnmnrcd returned a failure.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Check permissions for ImportError [airflow]

[via GitHub]

vincbeck commented on code in PR #37468:
URL: https://github.com/apache/airflow/pull/37468#discussion_r1491830495


##
airflow/api_connexion/endpoints/import_error_endpoint.py:
##
@@ -65,10 +85,42 @@ def get_import_errors(
 """Get all import errors."""
 to_replace = {"import_error_id": "id"}
 allowed_filter_attrs = ["import_error_id", "timestamp", "filename"]
-total_entries = session.scalars(func.count(ImportErrorModel.id)).one()
+count_query = select(func.count(ImportErrorModel.id))
 query = select(ImportErrorModel)
 query = apply_sorting(query, order_by, to_replace, allowed_filter_attrs)
+
+can_read_all_dags = get_auth_manager().is_authorized_dag(method="GET")
+
+if not can_read_all_dags:
+# if the user doesn't have access to all DAGs, only display errors 
from visible DAGs
+readable_dag_ids = security.get_readable_dags()
+query = query.join(DagModel, DagModel.fileloc == 
ImportErrorModel.filename).where(
+DagModel.dag_id.in_(readable_dag_ids)
+)
+count_query = count_query.join(DagModel, DagModel.fileloc == 
ImportErrorModel.filename).where(

Review Comment:
   From the query you cannot get the count? We are doing twice the same query 
here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Check permissions for ImportError [airflow]

[via GitHub]

vincbeck commented on code in PR #37468:
URL: https://github.com/apache/airflow/pull/37468#discussion_r1491829265


##
airflow/api_connexion/endpoints/import_error_endpoint.py:
##
@@ -16,39 +16,59 @@
 # under the License.
 from __future__ import annotations
 
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Sequence
 
 from sqlalchemy import func, select
 
 from airflow.api_connexion import security
-from airflow.api_connexion.exceptions import NotFound
+from airflow.api_connexion.exceptions import NotFound, PermissionDenied
 from airflow.api_connexion.parameters import apply_sorting, check_limit, 
format_parameters
 from airflow.api_connexion.schemas.error_schema import (
 ImportErrorCollection,
 import_error_collection_schema,
 import_error_schema,
 )
-from airflow.auth.managers.models.resource_details import AccessView
+from airflow.auth.managers.models.resource_details import AccessView, 
DagDetails
+from airflow.models.dag import DagModel
 from airflow.models.errors import ImportError as ImportErrorModel
 from airflow.utils.session import NEW_SESSION, provide_session
+from airflow.www.extensions.init_auth_manager import get_auth_manager
 
 if TYPE_CHECKING:
 from sqlalchemy.orm import Session
 
 from airflow.api_connexion.types import APIResponse
+from airflow.auth.managers.models.batch_apis import IsAuthorizedDagRequest
 
 
 @security.requires_access_view(AccessView.IMPORT_ERRORS)
 @provide_session
 def get_import_error(*, import_error_id: int, session: Session = NEW_SESSION) 
-> APIResponse:
 """Get an import error."""
 error = session.get(ImportErrorModel, import_error_id)
-
 if error is None:
 raise NotFound(
 "Import error not found",
 detail=f"The ImportError with import_error_id: `{import_error_id}` 
was not found",
 )
+session.expunge(error)
+
+can_read_all_dags = get_auth_manager().is_authorized_dag(method="GET")
+if not can_read_all_dags:
+readable_dag_ids = security.get_readable_dags()
+file_dag_ids = {
+dag_id[0]
+for dag_id in 
session.query(DagModel.dag_id).filter(DagModel.fileloc == error.filename).all()
+}

Review Comment:
   We do already this check here: 
https://github.com/apache/airflow/blob/main/airflow/auth/managers/base_auth_manager.py#L364.
 I dont think it is necessary to do it twice. I would just do
   
   
   ```suggestion
   readable_dag_ids = security.get_readable_dags()
   file_dag_ids = {
   dag_id[0]
   for dag_id in 
session.query(DagModel.dag_id).filter(DagModel.fileloc == error.filename).all()
   }
   
   ```



##
airflow/api_connexion/endpoints/import_error_endpoint.py:
##
@@ -65,10 +85,42 @@ def get_import_errors(
 """Get all import errors."""
 to_replace = {"import_error_id": "id"}
 allowed_filter_attrs = ["import_error_id", "timestamp", "filename"]
-total_entries = session.scalars(func.count(ImportErrorModel.id)).one()
+count_query = select(func.count(ImportErrorModel.id))
 query = select(ImportErrorModel)
 query = apply_sorting(query, order_by, to_replace, allowed_filter_attrs)
+
+can_read_all_dags = get_auth_manager().is_authorized_dag(method="GET")
+
+if not can_read_all_dags:

Review Comment:
   Same



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Enable More PyDocStyle Checks [airflow]

[via GitHub]

potiuk commented on issue #10742:
URL: https://github.com/apache/airflow/issues/10742#issuecomment-1947559270

   > Issue is that same module have D*s and also E*s.
   
   We could combine `D*`'s and `E*s` together as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Enable More PyDocStyle Checks [airflow]

[via GitHub]

rawwar commented on issue #10742:
URL: https://github.com/apache/airflow/issues/10742#issuecomment-1947557987

   > We could combine several `D*`s if we want in one row. I don't think there 
is a particular reason why we want to keep them in separate sections. 
@Taragolis ?
   
   Issue is that same module have `D*`s and also `E*`s. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Enable More PyDocStyle Checks [airflow]

[via GitHub]

potiuk commented on issue #10742:
URL: https://github.com/apache/airflow/issues/10742#issuecomment-1947556694

   We could `D*` if we want. I don't think there is a particular reason why we 
want to keep them in separate sections. @Taragolis ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Enable More PyDocStyle Checks [airflow]

[via GitHub]

rawwar commented on issue #10742:
URL: https://github.com/apache/airflow/issues/10742#issuecomment-1947555396

   And, list of all issues that we can add to `pyproject.toml`. I did some 
basic pre-processing.
   
   
   ```
   "airflow/__main__.py" = ["D103"]
   "airflow/api/auth/__init__.py" = ["D104"]
   "airflow/api/auth/backend/__init__.py" = ["D104"]
   "airflow/api/auth/backend/basic_auth.py" = ["D103"]
   "airflow/api/auth/backend/kerberos_auth.py" = ["D100"]
   "airflow/api/client/local_client.py" = ["D102"]
   "airflow/api/common/__init__.py" = ["D104"]
   "airflow/api/common/airflow_health.py" = ["D100"]
   "airflow/api_connexion/__init__.py" = ["D104"]
   "airflow/api_connexion/endpoints/__init__.py" = ["D104"]
   "airflow/api_connexion/endpoints/config_endpoint.py" = ["D100", "D103"]
   "airflow/api_connexion/endpoints/connection_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/dag_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/dag_run_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/dag_source_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/dag_warning_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/dataset_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/event_log_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/extra_link_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/forward_to_fab_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/health_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/import_error_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/log_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/plugin_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/pool_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/provider_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/request_dict.py" = ["D100"]
   "airflow/api_connexion/endpoints/task_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/task_instance_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/update_mask.py" = ["D100", "D103"]
   "airflow/api_connexion/endpoints/variable_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/version_endpoint.py" = ["D100"]
   "airflow/api_connexion/endpoints/xcom_endpoint.py" = ["D100"]
   "airflow/api_connexion/exceptions.py" = ["D100"]
   "airflow/api_connexion/parameters.py" = ["D100"]
   "airflow/api_connexion/schemas/__init__.py" = ["D104"]
   "airflow/api_connexion/schemas/common_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/config_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/connection_schema.py" = ["D100", "D102"]
   "airflow/api_connexion/schemas/dag_run_schema.py" = ["D100", "D102"]
   "airflow/api_connexion/schemas/dag_schema.py" = ["D100", "D102"]
   "airflow/api_connexion/schemas/dag_source_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/dag_warning_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/dataset_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/enum_schemas.py" = ["D100"]
   "airflow/api_connexion/schemas/error_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/event_log_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/health_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/job_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/log_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/plugin_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/pool_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/provider_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/role_and_permission_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/sla_miss_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/task_instance_schema.py" = ["D100", "D102"]
   "airflow/api_connexion/schemas/task_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/trigger_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/user_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/variable_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/version_schema.py" = ["D100"]
   "airflow/api_connexion/schemas/xcom_schema.py" = ["D100"]
   "airflow/api_connexion/security.py" = ["D100", "D103"]
   "airflow/api_connexion/types.py" = ["D100"]
   "airflow/api_internal/__init__.py" = ["D104"]
   "airflow/api_internal/endpoints/__init__.py" = ["D104"]
   "airflow/api_internal/endpoints/rpc_api_endpoint.py" = ["D100"]
   "airflow/api_internal/internal_api_call.py" = ["D100", "D102"]
   "airflow/auth/__init__.py" = ["D104"]
   "airflow/auth/managers/__init__.py" = ["D104"]
   "airflow/auth/managers/base_auth_manager.py" = ["D100"]
   "airflow/auth/managers/fab/__init__.py" = ["D104"]
   "airflow/auth/managers/fab/api/__init__.py" = ["D104"]
   "airflow/auth/managers/fab/api/auth/__init__.py" = ["D104"]
   "airflow/auth/managers/fab/api/auth/backend/__init__.py" = ["D104"]
   "airflow/auth/managers/fab/api/auth/backend/basic_auth.py" = ["D103"]
   

Re: [PR] Make Datasets hashable [airflow]

[via GitHub]

potiuk commented on PR #37465:
URL: https://github.com/apache/airflow/pull/37465#issuecomment-1947554480

   LGTM. But I would prefer Others to also take a look - maybe there is a good 
reason for not having it hashable :). And thanks for the thorough research on 
the past !
   
   BTW. Static checks need fixing on that one (I heartily recommend installing 
pre-commit and largely stop caring about thsoe failures).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Enable More PyDocStyle Checks [airflow]

[via GitHub]

rawwar commented on issue #10742:
URL: https://github.com/apache/airflow/issues/10742#issuecomment-1947553871

   > Oh absolutley. Moving it out is just theorethically possible, but we do 
not want to do it (or at least we have not decided about it yet. I personally 
think there is a value in keeping all project settings in a single 
pyproject.toml.
   
   I was just trying to do this. But, I see an issue with updating 
`pyproject.tom`
   
   one of the module `"airflow/api/auth/backend/kerberos_auth.py"`  has `D100` 
issue and also `E402`. Since, we are adding separate sections with module as a 
key in pyproject.toml, that's causing a conflict. 
   
   As of now, I only see three modules causing above conflict:
   
   ```
   airflow/security/kerberos.py
   airflow/security/permissions.py
   airflow/security/utils.py
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Enable More PyDocStyle Checks [airflow]

[via GitHub]

potiuk commented on issue #10742:
URL: https://github.com/apache/airflow/issues/10742#issuecomment-1947550688

   > Can we just update it in the current pyproject.toml? I would like to 
contribute and I'll finish them off one by one
   
   Oh absolutley. Moving it out is just theorethically possible, but we do not 
want to do it (or at least we have not decided about it yet. I personally think 
there is a value in keeping  all project settings in a single pyproject.toml.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Reduce irrelevant ERROR level logs from KubernertesPodOperator on pod runtime failure [airflow]

[via GitHub]

sudiptob2 commented on issue #36077:
URL: https://github.com/apache/airflow/issues/36077#issuecomment-1947549730

   Thanks for the hint ✅ 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Reduce irrelevant ERROR level logs from KubernertesPodOperator on pod runtime failure [airflow]

[via GitHub]

ketozhang commented on issue #36077:
URL: https://github.com/apache/airflow/issues/36077#issuecomment-1947548134

   It looks like you're not getting any event failures because of some 
Kubernetes permission issue. In your logs, you're getting a 403 error:
   
   > HTTP response body: 
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"events
 is forbidden: User \"system:serviceaccount:default:default\" cannot list 
resource \"events\" in API group \"\" in the namespace 
\"default\"","reason":"Forbidden","details":{"kind":"events"},"code":403}
   
   I'm not familiar with the Helm chart you're using, ` airflow-k8s-maxcotec`. 
I'd look into your chart's documentation or try a different helm chart.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Core tests fail when run separately [airflow]

[via GitHub]

potiuk commented on issue #37466:
URL: https://github.com/apache/airflow/issues/37466#issuecomment-1947544510

   BTW. You Absolutely should not open issue for such  things - just opening PR 
is more than enough, we hate beauraucracy (even the word is ugly) and in 
Airlfow, when you plan to open a PR, there is absolutely no need to open issue 
for it, it adds unnecessary overhead.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Check permissions for ImportError [airflow]

[via GitHub]

jedcunningham opened a new pull request, #37468:
URL: https://github.com/apache/airflow/pull/37468

   
   
   
   
   
   
   
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Core tests fail when run separately [airflow]

[via GitHub]

potiuk commented on issue #37466:
URL: https://github.com/apache/airflow/issues/37466#issuecomment-1947542936

   Are you ABSOLUTELY sure you tested in on the latest main ? 
   
   Becauase I believe I fixed this very problem few days ago here: 
https://github.com/apache/airflow/pull/37320 - where indeed original 
monkeypatching was not removed and all those things you are patching now were 
initialized as side effect.
   
   But combining _AIRFLOW_SKIP_DB_TESTS and patching just Session and Engine 
which is a bit easier, does the trick in a much simpler way.
   
   I just run `breeze testing tests --test-type Core` in main and it did not 
fail with the error you suggest, so likely you assessed it's wrong based on the 
version that was before.
   
   Can you please re-verify if you still can reproduce the problem with latest 
main ? My feeling is that it is good now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Continuously logging and Batch logging mode for the aws Glue job's execution logs [airflow]

[via GitHub]

github-actions[bot] commented on issue #26203:
URL: https://github.com/apache/airflow/issues/26203#issuecomment-1947539465

   This issue has been closed because it has not received response from the 
issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Not able to view the DAG page but jump straight to an error page when testing customized timetable [airflow]

[via GitHub]

github-actions[bot] closed issue #35630: Not able to view the DAG page but jump 
straight to an error page when testing customized timetable
URL: https://github.com/apache/airflow/issues/35630


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Not able to view the DAG page but jump straight to an error page when testing customized timetable [airflow]

[via GitHub]

github-actions[bot] commented on issue #35630:
URL: https://github.com/apache/airflow/issues/35630#issuecomment-1947539386

   This issue has been closed because it has not received response from the 
issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Continuously logging and Batch logging mode for the aws Glue job's execution logs [airflow]

[via GitHub]

github-actions[bot] closed issue #26203: Continuously logging and Batch logging 
mode for the aws Glue job's execution logs
URL: https://github.com/apache/airflow/issues/26203


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] `GoogleBaseHook` is ignoring `key_secret_name` in `provide_gcp_credential_file_as_context` method [airflow]

[via GitHub]

github-actions[bot] commented on issue #35190:
URL: https://github.com/apache/airflow/issues/35190#issuecomment-1947539409

   This issue has been automatically marked as stale because it has been open 
for 14 days with no response from the author. It will be closed in next 7 days 
if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Enable More PyDocStyle Checks [airflow]

[via GitHub]

rawwar commented on issue #10742:
URL: https://github.com/apache/airflow/issues/10742#issuecomment-1947534083

   > In theory we could enable all rules [#10742 
(comment)](https://github.com/apache/airflow/issues/10742#issuecomment-1947242983)
 but place exclusions into the separate file, e.g. in ruff.toml
   > 
   > ```toml
   > extend = "pyproject.toml"
   > 
   > [lint.extend-per-file-ignores]
   > "airflow/api/auth/backend/kerberos_auth.py" = ["D100", "D107"]
   > "airflow/api/common/airflow_health.py" = ["D100"]
   > "airflow/api_connexion/endpoints/config_endpoint.py" = ["D100"]
   > 
   > # Another 1500 lines of exclusions
   > 
   > "airflow/providers/teradata/hooks/teradata.py" = ["D107"]
   > "airflow/providers/yandex/secrets/lockbox.py" = ["D107"]
   > 
   > "hatch_build.py" = ["D100", "D102"]
   > ```
   
   Can we just update it in the current `pyproject.toml`?  I would like to 
contribute and I'll finish them off one by one 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Enable More PyDocStyle Checks [airflow]

[via GitHub]

rawwar commented on issue #10742:
URL: https://github.com/apache/airflow/issues/10742#issuecomment-1947530957

   > In theory we could enable all rules [#10742 
(comment)](https://github.com/apache/airflow/issues/10742#issuecomment-1947242983)
 but place exclusions into the separate file, e.g. in ruff.toml
   > 
   > ```toml
   > extend = "pyproject.toml"
   > 
   > [lint.extend-per-file-ignores]
   > "airflow/api/auth/backend/kerberos_auth.py" = ["D100", "D107"]
   > "airflow/api/common/airflow_health.py" = ["D100"]
   > "airflow/api_connexion/endpoints/config_endpoint.py" = ["D100"]
   > 
   > # Another 1500 lines of exclusions
   > 
   > "airflow/providers/teradata/hooks/teradata.py" = ["D107"]
   > "airflow/providers/yandex/secrets/lockbox.py" = ["D107"]
   > 
   > "hatch_build.py" = ["D100", "D102"]
   > ```
   
   I'll raise a PR for this with all exclusions for `D100-D104`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] ArgoCD reverting the app to Missing, OutOfSync even when successfully deployed [airflow]

[via GitHub]

potiuk commented on issue #37460:
URL: https://github.com/apache/airflow/issues/37460#issuecomment-1947527524

   > Specifically, the run-airflow-migrations and create-user jobs are 
executing every couple of minutes in this case. If the automated selfHeal is 
turned off, the app just shifts to Missing and OutOfSync after a couple of 
minute but it keeps working just fine. Just wanted to give a better description 
of what is happening!
   
   Just bear with us that for most people here who do not use ArgoCD - none of 
this - syncing, out-of-syncing has any significant meaning, and for people who 
voluntarily maintain Ariflow, they won't spend awfully lot of time digging in 
the details of Argo.
   
   For us we deliver Helm Chart is a helm chart and we generally deploy and 
test it in plain K8S. If there is anything that Argo Does which is specific to 
Argo, it's pretty much up to you to see what's happening and translate it into 
"what really happens".  
   
   By looking at logs etc. and trying to figure out thibgs - while Argo is 
popular, it's still pretty much 3rd-party technology that we have little to 
none expert knowedge there and we have to rely on our users - who are part of 
the community and actually have the skills and incentive to investigate things 
like that. If you can diagnose it and translate it to what happens in the 
"non-argo" terms but - generally what happens with resources, memory, 
networking and other things - then we can try to see how we can help with that. 
   
   Maybe other Argo users might chime in and provide some input - but from 
maintainers point of view it's like that . 
   
   Just to set some expectations here. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Core tests fail when run separately #37466 [airflow]

[via GitHub]

drajguru opened a new pull request, #37467:
URL: https://github.com/apache/airflow/pull/37467

   Add orm mocking to settings test
   
   Prevent putting the session into a state that prevents subsequent test 
modules from passing.
   closes: #37466


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Reduce irrelevant ERROR level logs from KubernertesPodOperator on pod runtime failure [airflow]

[via GitHub]

sudiptob2 commented on issue #36077:
URL: https://github.com/apache/airflow/issues/36077#issuecomment-1947510738

   Hi @ketozhang 
   I ran 2 dags according to your suggestion. But could not get the Pod Event 
log.
   1. 
[minimal_test_fail_dag_without_cr.py](https://github.com/sudiptob2/airflow-kpo-test/blob/main/dev/dags/minimal_test_fail_dag_without_cr.py)
 : Running without `container_resource` set, See 
[Logfile](https://github.com/sudiptob2/airflow-kpo-test/blob/abb61c5d67e4961a41584b9fa89973d8e5b51fbd/data/dag_id%3Dminimal_test_fail_task_without_cr/run_id%3Dscheduled__2022-01-01T00%3A00%3A00%2B00%3A00/task_id%3Dminimal_test_fail_task_without_cr/attempt%3D1.log)
   2. 
[minimal_test_fail.py](https://github.com/sudiptob2/airflow-kpo-test/blob/main/dev/dags/minimal_test_fail_dag.py):
 Run with container resources set, See 
[Logfile](https://github.com/sudiptob2/airflow-kpo-test/blob/abb61c5d67e4961a41584b9fa89973d8e5b51fbd/data/dag_id%3Dminimal_test_fail_dag/run_id%3Dscheduled__2022-01-01T00%3A00%3A00%2B00%3A00/task_id%3Dminimal_test_fail_task/attempt%3D1.log).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[I] Core tests fail when run separately [airflow]

[via GitHub]

drajguru opened a new issue, #37466:
URL: https://github.com/apache/airflow/issues/37466

   ### Apache Airflow version
   
   main (development)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   test_sqlite_relative_path() in test_settings.py seemingly puts the session 
into a state that prevents subsequent test modules from passing. I think during 
this test the session is re-initialized and this interferes other test modules 
due to it being a global object.
   
   ### What you think should happen instead?
   
   The test should mock out the session, engine... objects to avoid mutating 
the global state.
   
   ### How to reproduce
   
   breeze testing tests --test-type Core
   
   ! _pytest.outcomes.Exit: Unable clear test DB, got error 
sqlalchemy.exc.OperationalE
   [SQL: DELETE FROM job]
   (Background on this error at: https://sqlalche.me/e/14/e3q8) !
    193 passed, 4 skipped, 5 warnings in 65.27s (0:01:05) 
=
   
   ### Operating System
   
   ubuntu 22.04
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Virtualenv installation
   
   ### Deployment details
   
   Using breeze
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Add environment var example to mssql.rst [airflow]

[via GitHub]

boring-cyborg[bot] commented on PR #37418:
URL: https://github.com/apache/airflow/pull/37418#issuecomment-1947489220

   Awesome work, congrats on your first merged pull request! You are invited to 
check our [Issue Tracker](https://github.com/apache/airflow/issues) for 
additional contributions.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch main updated: Add environment var example to mssql.rst (#37418)

[jscheffl]

This is an automated email from the ASF dual-hosted git repository.

jscheffl pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 9f4f8b3553 Add environment var example to mssql.rst (#37418)
9f4f8b3553 is described below

commit 9f4f8b3553ea02e021712b893d9d076f9c9b5e11
Author: Dikarabo-Molele 
AuthorDate: Fri Feb 16 01:11:41 2024 +0200

Add environment var example to mssql.rst (#37418)

* Add environment var example to mssql.rst

Adding example code for configuring MSSQL connection using environment 
variables for easier reference.

* Update mssql.rst to add JSON serializing example

Update mssql.rst to add JSON serializing example

* Update docs/apache-airflow-providers-microsoft-mssql/connections/mssql.rst

Remove unnecessary ..

Co-authored-by: Jens Scheffler <95105677+jsche...@users.noreply.github.com>

* Update docs/apache-airflow-providers-microsoft-mssql/connections/mssql.rst

Remove unnecessary ..

Co-authored-by: Wei Lee 

-

Co-authored-by: Jens Scheffler <95105677+jsche...@users.noreply.github.com>
Co-authored-by: Wei Lee 
---
 .../connections/mssql.rst  | 23 ++
 1 file changed, 23 insertions(+)

diff --git 
a/docs/apache-airflow-providers-microsoft-mssql/connections/mssql.rst 
b/docs/apache-airflow-providers-microsoft-mssql/connections/mssql.rst
index 9e66429044..a2990742ae 100644
--- a/docs/apache-airflow-providers-microsoft-mssql/connections/mssql.rst
+++ b/docs/apache-airflow-providers-microsoft-mssql/connections/mssql.rst
@@ -51,3 +51,26 @@ Extra (optional)
 
 More details on all MSSQL parameters supported can be found in
 `MSSQL documentation 
`_.
+
+When specifying the connection as URI (in :envvar:`AIRFLOW_CONN_{CONN_ID}` 
variable) you should specify it
+following the standard syntax of DB connections - where extras are passed as 
parameters
+of the URI. Note that all components of the URI should be URL-encoded.
+
+For example:
+
+.. code-block:: bash
+
+   export 
AIRFLOW_CONN_MSSQL_DEFAULT='mssql://username:passw...@server.com:1433/database_name'
+
+If serializing with JSON:
+
+.. code-block:: bash
+
+export AIRFLOW_CONN_MSSQL_DEFAULT='{
+"conn_type": "mssql",
+"login": "username",
+"password": "password",
+"host": "server.com",
+"port": 1433,
+"schema": "database_name"
+}'

Re: [PR] Add environment var example to mssql.rst [airflow]

[via GitHub]

jscheffl merged PR #37418:
URL: https://github.com/apache/airflow/pull/37418


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch main updated: Replace usage of `datetime.utcnow` and `datetime.utcfromtimestamp` in providers (#37138)

[taragolis]

This is an automated email from the ASF dual-hosted git repository.

taragolis pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 42f8d048d2 Replace usage of `datetime.utcnow` and 
`datetime.utcfromtimestamp` in providers (#37138)
42f8d048d2 is described below

commit 42f8d048d2dccfcf59a44e00e9b1e8a3e63090a0
Author: Andrey Anshin 
AuthorDate: Fri Feb 16 03:07:32 2024 +0400

Replace usage of `datetime.utcnow` and `datetime.utcfromtimestamp` in 
providers (#37138)

* Replace usage of `datetime.utcnow` and `datetime.utcfromtimestamp` in 
providers

* Apply suggested changes

Co-authored-by: Tzu-ping Chung 

* Remove redundand context

Co-authored-by: Tzu-ping Chung 

-

Co-authored-by: Tzu-ping Chung 
---
 .../amazon/aws/log/cloudwatch_task_handler.py  |  4 +--
 .../providers/amazon/aws/utils/task_log_fetcher.py |  4 +--
 .../providers/google/cloud/transfers/s3_to_gcs.py  |  4 +--
 .../providers/google/common/hooks/base_google.py   |  2 +-
 kubernetes_tests/test_base.py  |  4 +--
 tests/providers/amazon/aws/hooks/test_sagemaker.py |  4 +--
 .../amazon/aws/log/test_cloudwatch_task_handler.py |  4 +--
 .../celery/executors/test_celery_executor.py   |  4 +--
 .../executors/test_kubernetes_executor.py  |  8 ++---
 tests/providers/google/cloud/sensors/test_gcs.py   | 42 +-
 tests/providers/google/cloud/triggers/test_gcs.py  |  4 +--
 .../example_cloud_storage_transfer_service_aws.py  |  4 +--
 .../example_cloud_storage_transfer_service_gcp.py  |  4 +--
 13 files changed, 41 insertions(+), 51 deletions(-)

diff --git a/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py 
b/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py
index 3a801093ca..865b30233d 100644
--- a/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py
+++ b/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py
@@ -17,7 +17,7 @@
 # under the License.
 from __future__ import annotations
 
-from datetime import date, datetime, timedelta
+from datetime import date, datetime, timedelta, timezone
 from functools import cached_property
 from typing import TYPE_CHECKING, Any
 
@@ -163,7 +163,7 @@ class CloudwatchTaskHandler(FileTaskHandler, LoggingMixin):
 return "\n".join(self._event_to_str(event) for event in events)
 
 def _event_to_str(self, event: dict) -> str:
-event_dt = datetime.utcfromtimestamp(event["timestamp"] / 1000.0)
+event_dt = datetime.fromtimestamp(event["timestamp"] / 1000.0, 
tz=timezone.utc)
 formatted_event_dt = event_dt.strftime("%Y-%m-%d %H:%M:%S,%f")[:-3]
 message = event["message"]
 return f"[{formatted_event_dt}] {message}"
diff --git a/airflow/providers/amazon/aws/utils/task_log_fetcher.py 
b/airflow/providers/amazon/aws/utils/task_log_fetcher.py
index 63110edada..a4cad6c099 100644
--- a/airflow/providers/amazon/aws/utils/task_log_fetcher.py
+++ b/airflow/providers/amazon/aws/utils/task_log_fetcher.py
@@ -18,7 +18,7 @@
 from __future__ import annotations
 
 import time
-from datetime import datetime, timedelta
+from datetime import datetime, timedelta, timezone
 from threading import Event, Thread
 from typing import TYPE_CHECKING, Generator
 
@@ -87,7 +87,7 @@ class AwsTaskLogFetcher(Thread):
 
 @staticmethod
 def event_to_str(event: dict) -> str:
-event_dt = datetime.utcfromtimestamp(event["timestamp"] / 1000.0)
+event_dt = datetime.fromtimestamp(event["timestamp"] / 1000.0, 
tz=timezone.utc)
 formatted_event_dt = event_dt.strftime("%Y-%m-%d %H:%M:%S,%f")[:-3]
 message = event["message"]
 return f"[{formatted_event_dt}] {message}"
diff --git a/airflow/providers/google/cloud/transfers/s3_to_gcs.py 
b/airflow/providers/google/cloud/transfers/s3_to_gcs.py
index cc745059a1..92dc97b4cc 100644
--- a/airflow/providers/google/cloud/transfers/s3_to_gcs.py
+++ b/airflow/providers/google/cloud/transfers/s3_to_gcs.py
@@ -17,7 +17,7 @@
 # under the License.
 from __future__ import annotations
 
-from datetime import datetime
+from datetime import datetime, timezone
 from tempfile import NamedTemporaryFile
 from typing import TYPE_CHECKING, Any, Sequence
 
@@ -276,7 +276,7 @@ class S3ToGCSOperator(S3ListOperator):
 )
 
 def submit_transfer_jobs(self, files: list[str], gcs_hook: GCSHook, 
s3_hook: S3Hook) -> list[str]:
-now = datetime.utcnow()
+now = datetime.now(tz=timezone.utc)
 one_time_schedule = {"day": now.day, "month": now.month, "year": 
now.year}
 
 gcs_bucket, gcs_prefix = _parse_gcs_url(self.dest_gcs)
diff --git a/airflow/providers/google/common/hooks/base_google.py 
b/airflow/providers/google/common/hooks/base_google.py
index a83524422a..bee2798267 100644
--- a/airflow/providers/google/common/hooks/base_google.py
+++ 

Re: [PR] Replace usage of `datetime.utcnow` and `datetime.utcfromtimestamp` in providers [airflow]

[via GitHub]

Taragolis merged PR #37138:
URL: https://github.com/apache/airflow/pull/37138


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Make Datasets hashable [airflow]

[via GitHub]

mpeteuil opened a new pull request, #37465:
URL: https://github.com/apache/airflow/pull/37465

   Currently DAGs accept a 
[`Collection["Dataset"]`](https://github.com/apache/airflow/blob/0c02ead4d8a527cbf0a916b6344f255c520e637f/airflow/models/dag.py#L171)
 as an option for the `schedule`, but that collection cannot be a `set` because 
Datasets are not a hashable type. The interesting thing is that [the 
`DatasetModel` is actually already 
hashable](https://github.com/apache/airflow/blob/dec78ab3f140f35e507de825327652ec24d03522/airflow/models/dataset.py#L93-L100),
 so this introduces a bit of duplication since it's mostly the same 
implementation. However, Airflow users are primarily interfacing with 
`Dataset`, not `DatasetModel` so I think it makes sense for `Dataset` to be 
hashable. I'm not sure how to square the duplication or what `__eq__` and 
`__hash__` provide for `DatasetModel` though.
   
   There was discussion [on the original PR that created the 
`DatasetModel`](https://github.com/apache/airflow/pull/24613) about whether to 
create two classes or one. In that discussion @kaxil mentioned:
   
   > I would slightly favour a separate `DatasetModel` and `Dataset` so 
`Dataset` becomes an extensible class, and `DatasetModel` just stores the info 
about the class. So users don't need to care about SQLAlchmey stuff when 
extending it.
   
   That provides a bit of background on why they both exist for anyone reading 
this who is curious.
   
   
   
   
   
   
   
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Reduce irrelevant ERROR level logs from KubernertesPodOperator on pod runtime failure [airflow]

[via GitHub]

sudiptob2 commented on issue #36077:
URL: https://github.com/apache/airflow/issues/36077#issuecomment-1947442082

   Yes sure, Just to mention, I already did try 
[this](https://github.com/sudiptob2/airflow-kpo-test/blob/main/dev/dags/test_fail_dag_2.py)
 Dag which is similar to the one you suggested. But did not get Pod event. 
   
   Testing your suggested DAG now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] ArgoCD reverting the app to Missing, OutOfSync even when successfully deployed [airflow]

[via GitHub]

jaruji commented on issue #37460:
URL: https://github.com/apache/airflow/issues/37460#issuecomment-1947441852

   I can open an issue there tomorrow. I did some more research, and enabled 
automated prune and selfHeal in the argoCD app like so:
   ```
syncPolicy:
   automated:
   prune: true
   selfHeal: true
   ```
   The argo airflow app is now re-syncing every couple of minutes to keep the 
Healthy and Synced status. Specifically, the run-airflow-migrations and 
create-user jobs are executing every couple of minutes in this case. If the 
automated selfHeal is turned off, the app just shifts to Missing and OutOfSync 
after a couple of minutes - but it keeps working just fine. Just wanted to give 
a better description of what is happening!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Reduce irrelevant ERROR level logs from KubernertesPodOperator on pod runtime failure [airflow]

[via GitHub]

ketozhang commented on issue #36077:
URL: https://github.com/apache/airflow/issues/36077#issuecomment-1947438237

   If you wouldn't mind. Can you do one run with and another without 
`container_resources` set?
   
   I forgot the original issue I outlined here was we are getting Pod Event 
failures when we are not supposed to. With `container_resources`, you should 
expected a Pod Event failure. Without `container_resources`, you should expect 
no Pod Event failure — I did see one which is why I raised this ticket.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Inlets not handled properly with @task functions [airflow]

[via GitHub]

ninowalker commented on issue #37463:
URL: https://github.com/apache/airflow/issues/37463#issuecomment-1947437366

   For additional context,  with the python debugger inside `execute_callable` 
of the PythonOperator (which wraps the function in question):
   
   1. It's calling `self.python_callable(*self.op_args, **self.op_kwargs)`
   2. `self.op_kwargs` are unresolved.
   3. However, `self.inlets` does contain the correct values.
   4. Thus, the problem is with the PythonOperator not supporting `inlets`.
   
   Is this by design? Seems not.
   
   
   ```python
   > 
/Users/rfc/miniforge3/envs/openwtf/lib/python3.11/site-packages/airflow/operators/python.py(216)execute_callable()
   -> return self.python_callable(*self.op_args, **self.op_kwargs)
   (Pdb) p self.op_kwargs
   {'inlets': ['auto'], 'execution_date': '2024-02-15T22:14:46.598812+00:00'}
   (Pdb) p self.inlets
   ['filename']
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Reduce irrelevant ERROR level logs from KubernertesPodOperator on pod runtime failure [airflow]

[via GitHub]

sudiptob2 commented on issue #36077:
URL: https://github.com/apache/airflow/issues/36077#issuecomment-1947433926

   @ketozhang Thanks! will try this shortly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Reduce irrelevant ERROR level logs from KubernertesPodOperator on pod runtime failure [airflow]

[via GitHub]

ketozhang commented on issue #36077:
URL: https://github.com/apache/airflow/issues/36077#issuecomment-1947432083

   You seem to be getting a different error more so about Kubernetes API 
permission issue which isn't related to the Pod.
   
   Let's try to use reduce your DAG file into minimal viable example:
   
   ```python
   dag = DAG('test_fail_dag', schedule_interval='@once', catchup=False)
   
   task = KubernetesPodOperator(
   task_id='test_fail_task',
   dag=dag,
   container_resources={"requests": {"memory": "1000Gi", "cpu": "1000"}},
   log_events_on_failure=True,
   )
   ```
   
   Assuming your cluster isn't that large, this should give you in the logs 
something like `Pod Event: FailedScheduling - ...`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Filter datasets graph by dag_id [airflow]

[via GitHub]

bbovenzi opened a new pull request, #37464:
URL: https://github.com/apache/airflow/pull/37464

   Read in all the dag_ids from the dataset dependencies endpoint and allow 
filtering the datasets graph by dag_id
   A use rcan filter via the multi-select or from a tooltip when clicking on a 
Dag
   
   https://github.com/apache/airflow/assets/4600967/e7a640d9-93a3-4060-9ea2-d15c2524b2d6;>
   
   https://github.com/apache/airflow/assets/4600967/5d32c319-9a6c-4100-8e5b-e72833a4a821;>
   
   ![Feb-15-2024 
17-06-35](https://github.com/apache/airflow/assets/4600967/488935b3-229d-4068-87e0-6bd7d9f53bfd)
   
   
   
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] ArgoCD reverting the app to Missing, OutOfSync even when successfully deployed [airflow]

[via GitHub]

potiuk commented on issue #37460:
URL: https://github.com/apache/airflow/issues/37460#issuecomment-1947407531

   Have you opened the issue to ArgoCD? What do they say ? It could be caused 
by some issues that we already know and plan to fix 
https://github.com/apache/airflow/issues/37399 or 
https://github.com/apache/airflow/pull/37356 - can you please check with Argo 
CD support? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] smtp_default connection host/port is ignored when sending email [airflow]

[via GitHub]

Taragolis commented on issue #37442:
URL: https://github.com/apache/airflow/issues/37442#issuecomment-1947401299

   Maybe you rights, attached link to core modules.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] smtp_default connection host/port is ignored when sending email [airflow]

[via GitHub]

potiuk commented on issue #37442:
URL: https://github.com/apache/airflow/issues/37442#issuecomment-1947393912

   Ah. Maybe I misunderstood :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] smtp_default connection host/port is ignored when sending email [airflow]

[via GitHub]

Taragolis commented on issue #37442:
URL: https://github.com/apache/airflow/issues/37442#issuecomment-1947391262

   Just wondering is it issue about EmailOperator from core or from [SFTP 
Provider](https://airflow.apache.org/docs/apache-airflow-providers-smtp/stable/index.html)?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[I] Inlets not handled properly with @task functions [airflow]

[via GitHub]

ninowalker opened a new issue, #37463:
URL: https://github.com/apache/airflow/issues/37463

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   2.8.0
   
   ### What happened?
   
   A task with an outlet should convey it's outlets as inlets to a `@task` 
decorated function, when instantiated with `inlets=AUTO`. Instead, the string 
"auto" (the value of `airflow.lineage.AUTO`) is passed as the value, rather 
than the resolved outlets.
   
   Consider a minimal test case:
   
   ```python
   from airflow import DAG
   from airflow.decorators import task
   from airflow.lineage import AUTO
   from datetime import datetime, timedelta
   
   from airflow.operators.empty import EmptyOperator
   
   DAG_ID = "test_lineage"
   
   with DAG(
   dag_id=DAG_ID,
   schedule=None,
   start_date=datetime(2021, 1, 1),
   dagrun_timeout=timedelta(minutes=5),
   catchup=False,
   ) as dag:
   f = "filename"
   op1 = EmptyOperator(
   task_id="leave1",
   outlets=[f],
   )
   
   @task
   def test_inlets(inlets):
   assert inlets == [f], f"{inlets} != {[f]}"
   return True
   
   op1 >> test_inlets(inlets=AUTO)
   ```
   
   ### What you think should happen instead?
   
   `inlets` should resolve to the `outlets` of the upstream task, instead of 
being unresolved.
   
   Instead:
   ```
   AssertionError: ['auto'] != ['filename']'
   ```
   
   ### How to reproduce
   
   1. Create the DAG, given the test code above.
   2. Run the DAG.
   3. Be sad :(
   
   ### Operating System
   
   OSX
   
   ### Versions of Apache Airflow Providers
   
   ```
   apache-airflow-providers-amazon==8.13.0
   apache-airflow-providers-common-sql==1.9.0
   apache-airflow-providers-discord==3.5.0
   apache-airflow-providers-ftp @ 
file:///home/conda/feedstock_root/build_artifacts/apache-airflow-providers-ftp_1702441975571/work
   apache-airflow-providers-google==10.12.0
   apache-airflow-providers-http @ 
file:///home/conda/feedstock_root/build_artifacts/apache-airflow-providers-http_1706649884002/work
   apache-airflow-providers-imap @ 
file:///home/conda/feedstock_root/build_artifacts/apache-airflow-providers-imap_1702443030927/work
   apache-airflow-providers-papermill==3.6.0
   apache-airflow-providers-postgres==5.9.0
   apache-airflow-providers-sqlite==3.6.0
   ```
   
   ### Deployment
   
   Other
   
   ### Deployment details
   
   Running standalone.
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Inlets not handled properly with @task functions [airflow]

[via GitHub]

boring-cyborg[bot] commented on issue #37463:
URL: https://github.com/apache/airflow/issues/37463#issuecomment-1947378746

   Thanks for opening your first issue here! Be sure to follow the issue 
template! If you are willing to raise PR to address this issue please do so, no 
need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch constraints-main updated: Updating constraints. Github run id:7921989999

[github-bot]

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a commit to branch constraints-main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/constraints-main by this push:
 new 2ad99b909f Updating constraints. Github run id:792198
2ad99b909f is described below

commit 2ad99b909f1b7c306b764a901272f807ae917f9e
Author: Automated GitHub Actions commit 
AuthorDate: Thu Feb 15 21:25:01 2024 +

Updating constraints. Github run id:792198

This update in constraints is automatically committed by the CI 
'constraints-push' step based on
'refs/heads/main' in the 'apache/airflow' repository with commit sha 
3c1d05140e352ecc612e4a0c92627a59201b7f80.

The action that build those constraints can be found at 
https://github.com/apache/airflow/actions/runs/792198/

The image tag used for that build was: 
3c1d05140e352ecc612e4a0c92627a59201b7f80. You can enter Breeze environment
with this image by running 'breeze shell --image-tag 
3c1d05140e352ecc612e4a0c92627a59201b7f80'

All tests passed in this build so we determined we can push the updated 
constraints.

See 
https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for 
details.
---
 constraints-3.10.txt  | 95 ++-
 constraints-3.11.txt  | 95 ++-
 constraints-3.8.txt   | 95 ++-
 constraints-3.9.txt   | 95 ++-
 constraints-no-providers-3.10.txt |  8 +--
 constraints-no-providers-3.11.txt |  8 +--
 constraints-no-providers-3.8.txt  |  8 +--
 constraints-no-providers-3.9.txt  |  8 +--
 constraints-source-providers-3.10.txt | 28 ++-
 constraints-source-providers-3.11.txt | 28 ++-
 constraints-source-providers-3.8.txt  | 28 ++-
 constraints-source-providers-3.9.txt  | 28 ++-
 12 files changed, 272 insertions(+), 252 deletions(-)

diff --git a/constraints-3.10.txt b/constraints-3.10.txt
index 0c573522f0..438207e540 100644
--- a/constraints-3.10.txt
+++ b/constraints-3.10.txt
@@ -1,6 +1,6 @@
 
 #
-# This constraints file was automatically generated on 
2024-02-14T14:51:04.593358
+# This constraints file was automatically generated on 
2024-02-15T20:59:57.093254
 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow.
 # This variant of constraints install uses the HEAD of the branch version for 
'apache-airflow' but installs
 # the providers from PIP-released packages at the moment of the constraint 
generation.
@@ -45,7 +45,7 @@ Flask-SQLAlchemy==2.5.1
 Flask-Session==0.5.0
 Flask-WTF==1.2.1
 Flask==2.2.5
-GitPython==3.1.41
+GitPython==3.1.42
 JPype1==1.5.0
 JayDeBeApi==1.2.3
 Jinja2==3.1.3
@@ -96,55 +96,55 @@ anyio==4.2.0
 apache-airflow-providers-airbyte==3.6.0
 apache-airflow-providers-alibaba==2.7.2
 apache-airflow-providers-amazon==8.17.0
-apache-airflow-providers-apache-beam==5.6.0
+apache-airflow-providers-apache-beam==5.6.1
 apache-airflow-providers-apache-cassandra==3.4.1
-apache-airflow-providers-apache-drill==2.6.0
-apache-airflow-providers-apache-druid==3.8.0
+apache-airflow-providers-apache-drill==2.6.1
+apache-airflow-providers-apache-druid==3.8.1
 apache-airflow-providers-apache-flink==1.3.0
 apache-airflow-providers-apache-hdfs==4.3.2
-apache-airflow-providers-apache-hive==6.4.2
+apache-airflow-providers-apache-hive==7.0.0
 apache-airflow-providers-apache-impala==1.3.0
 apache-airflow-providers-apache-kafka==1.3.1
 apache-airflow-providers-apache-kylin==3.5.0
-apache-airflow-providers-apache-livy==3.7.1
+apache-airflow-providers-apache-livy==3.7.2
 apache-airflow-providers-apache-pig==4.3.0
 apache-airflow-providers-apache-pinot==4.3.0
 apache-airflow-providers-apache-spark==4.7.1
-apache-airflow-providers-apprise==1.2.1
+apache-airflow-providers-apprise==1.2.2
 apache-airflow-providers-arangodb==2.4.1
 apache-airflow-providers-asana==2.4.1
 apache-airflow-providers-atlassian-jira==2.5.1
-apache-airflow-providers-celery==3.5.2
+apache-airflow-providers-celery==3.6.0
 apache-airflow-providers-cloudant==3.4.1
 apache-airflow-providers-cncf-kubernetes==7.14.0
 apache-airflow-providers-cohere==1.1.2
-apache-airflow-providers-common-io==1.2.0
+apache-airflow-providers-common-io==1.3.0
 apache-airflow-providers-common-sql==1.10.1
-apache-airflow-providers-databricks==6.1.0
+apache-airflow-providers-databricks==6.2.0
 apache-airflow-providers-datadog==3.5.1
-apache-airflow-providers-dbt-cloud==3.6.0
+apache-airflow-providers-dbt-cloud==3.6.1
 apache-airflow-providers-dingding==3.4.0
 apache-airflow-providers-discord==3.6.0
 apache-airflow-providers-docker==3.9.1
-apache-airflow-providers-elasticsearch==5.3.2
+apache-airflow-providers-elasticsearch==5.3.3
 apache-airflow-providers-exasol==4.4.2
 

Re: [PR] [POC] Validate is it possible to build wheel from sdist distribution [airflow]

[via GitHub]

Taragolis commented on code in PR #37444:
URL: https://github.com/apache/airflow/pull/37444#discussion_r1491687286


##
dev/breeze/src/airflow_breeze/commands/release_management_commands.py:
##
@@ -711,8 +804,8 @@ def prepare_provider_packages(
 sys.exit(0)
 get_console().print("\n[success]Successfully built packages!\n\n")
 get_console().print("\n[info]Packages available in dist:\n")
-for file in sorted(DIST_DIR.glob("apache*")):
-get_console().print(file.name)
+for dist_info in _dist_packages(package_format=package_format, 
build_type="providers"):
+get_console().print(str(dist_info))

Review Comment:
   Yeah, will push it into the airflow repo tomorrow, after check how it impact 
on providers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Azure blob remote logging - The specified container does not exist. [airflow]

[via GitHub]

jaruji commented on issue #37459:
URL: https://github.com/apache/airflow/issues/37459#issuecomment-1947350151

   Okay that fixed it. The docs for this should get updated, as the information 
written there regarding the azure blob remote logging setup is not up to date. 
It took me a fair bit of time to figure out that the issue was not on my side. 
Thanks for the patience and help!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [POC] Validate is it possible to build wheel from sdist distribution [airflow]

[via GitHub]

potiuk commented on code in PR #37444:
URL: https://github.com/apache/airflow/pull/37444#discussion_r1491675071


##
dev/breeze/src/airflow_breeze/commands/release_management_commands.py:
##
@@ -711,8 +804,8 @@ def prepare_provider_packages(
 sys.exit(0)
 get_console().print("\n[success]Successfully built packages!\n\n")
 get_console().print("\n[info]Packages available in dist:\n")
-for file in sorted(DIST_DIR.glob("apache*")):
-get_console().print(file.name)
+for dist_info in _dist_packages(package_format=package_format, 
build_type="providers"):
+get_console().print(str(dist_info))

Review Comment:
   BTW. Since it is sdist - it is only enabled in canary build, so you can 
either change the condtion in ci.yml it to test it in CI or push it to 
"airflow" repo and assign `canary` label to get it work in the same way as 
canary build.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Add colour to logs [airflow]

[via GitHub]

potiuk commented on issue #37443:
URL: https://github.com/apache/airflow/issues/37443#issuecomment-1947343754

   > Sorry, I don't mean to say that its not possible. I think, the current log 
tab is just a window showing text and I was talking about it.
   
   Well it's a javascript-controlled window. It can display anything, it's just 
a matter of formatting the text you put there. And yes that woudl be a nice 
option to be able to see the logs colored - we already show them colored in 
terminal, so showing them colored in the UI would be nice. I think both 
options, coloring with ANSI colors similarly to what we do in terminal or the 
grafana-like approach is good. Someone would need to design, propose get 
positive feedback and implement it though


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] smtp_default connection host/port is ignored when sending email [airflow]

[via GitHub]

potiuk commented on issue #37442:
URL: https://github.com/apache/airflow/issues/37442#issuecomment-1947335380

   So far that has been deliberate choice - see 
https://airflow.apache.org/docs/apache-airflow/stable/howto/email-config.html. 
SMTP for the emails that are sent by Airflow have - traditinally been sent 
using configuration done by Deployment Manager in Airflow configuraiton and 
it's been a deliberate decision to spearate "platform" configuration from "DAG" 
configuration. Using user/password from Connection table has been added later 
as a way to make security credentials kept using FERNET-encrypted DB entry 
mechanism we already have, but there was no intention to use all the 
configuration there - but if you define a way how to handle backwards 
compatibility, and explain it in the docs including how to change existing 
confifuration to the new one and also make it in the way to avoid even more 
confusion, then i think it's quite ok to have it.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Azure blob remote logging - The specified container does not exist. [airflow]

[via GitHub]

Taragolis commented on issue #37459:
URL: https://github.com/apache/airflow/issues/37459#issuecomment-1947316985

   It is separate section, something like that.
   
   ```yaml
   config:
   logging:
 remote_logging: 'True'
 remote_base_log_folder: wasb-airflow/logs
 remote_log_conn_id: azure_blob_storage
   
   azure_remote_logging:
 remote_wasb_log_container: airflow
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Partially enable D105 rule [airflow]

[via GitHub]

Taragolis commented on PR #37462:
URL: https://github.com/apache/airflow/pull/37462#issuecomment-1947310104

   Anyway, I will create appropriate task tomorrow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Partially enable D105 rule [airflow]

[via GitHub]

potiuk commented on PR #37462:
URL: https://github.com/apache/airflow/pull/37462#issuecomment-1947309064

like it we have in pyproject.toml (and have mechanism to see the 
"important" changes there :).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Partially enable D105 rule [airflow]

[via GitHub]

potiuk commented on PR #37462:
URL: https://github.com/apache/airflow/pull/37462#issuecomment-1947305106

   https://github.com/apache/airflow/assets/595491/e6255807-312b-45a3-a783-2889bdf8e5b7;>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Partially enable D105 rule [airflow]

[via GitHub]

Taragolis commented on PR #37462:
URL: https://github.com/apache/airflow/pull/37462#issuecomment-1947304958

   We could move this changes in the separate file 
https://github.com/apache/airflow/issues/10742#issuecomment-1947303063 but it 
just an idea, because if we enable all rules we have to add 1500+ exclusions


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Enable More PyDocStyle Checks [airflow]

[via GitHub]

Taragolis commented on issue #10742:
URL: https://github.com/apache/airflow/issues/10742#issuecomment-1947303063

   In theory we could enable all rules 
https://github.com/apache/airflow/issues/10742#issuecomment-1947242983 but 
place exclusions into the separate file, e.g. in ruff.toml
   
   ```toml
   extend = "pyproject.toml"
   
   [lint.extend-per-file-ignores]
   "airflow/api/auth/backend/kerberos_auth.py" = ["D100", "D107"]
   "airflow/api/common/airflow_health.py" = ["D100"]
   "airflow/api_connexion/endpoints/config_endpoint.py" = ["D100"]
   
   # Another 1500 lines of exclusions
   
   "airflow/providers/teradata/hooks/teradata.py" = ["D107"]
   "airflow/providers/yandex/secrets/lockbox.py" = ["D107"]
   
   "hatch_build.py" = ["D100", "D102"]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Partially enable D105 rule [airflow]

[via GitHub]

potiuk commented on PR #37462:
URL: https://github.com/apache/airflow/pull/37462#issuecomment-1947302356

   I really like how those kinds of pyproject.toml changes now do not generate 
full build :D . YAY... Achievement unlocked.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch main updated: Partially enable D105 rule (#37462)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new 3c1d05140e Partially enable D105 rule (#37462)
3c1d05140e is described below

commit 3c1d05140e352ecc612e4a0c92627a59201b7f80
Author: Andrey Anshin 
AuthorDate: Fri Feb 16 00:36:22 2024 +0400

Partially enable D105 rule (#37462)
---
 pyproject.toml | 88 --
 1 file changed, 86 insertions(+), 2 deletions(-)

diff --git a/pyproject.toml b/pyproject.toml
index 20ea3572ac..f6a7c1e020 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1302,6 +1302,10 @@ extend-select = [
 "RUF100", # Unused noqa (auto-fixable)
 # We ignore more pydocstyle than we enable, so be more selective at what 
we enable
 "D101",
+# We add modules that do not follow the rule `Missing docstring in magic 
method`
+# into the `tool.ruff.per-file-ignores`, and should remove it from that 
list as soon as it follows.
+# See: https://github.com/apache/airflow/issues/10742
+"D105",
 "D106",
 "D2",
 "D3",
@@ -1355,11 +1359,14 @@ combine-as-imports = true
 "provider_packages/*" = ["D"]
 "*/example_dags/*" = ["D"]
 "chart/*" = ["D"]
-# In addition ignore top level imports, e.g. pandas, numpy in tests and dev
-"dev/*" = ["D", "TID253"]
+"dev/*" = ["D"]
+# In addition ignore top level imports, e.g. pandas, numpy in tests
+"dev/perf/*" = ["TID253"]
+"dev/breeze/tests/*" = ["TID253"]
 "tests/*" = ["D", "TID253"]
 "docker_tests/*" = ["D", "TID253"]
 "kubernetes_tests/*" = ["D", "TID253"]
+"helm_tests/*" = ["D", "TID253"]
 
 # All of the modules which have an extra license header (i.e. that we copy 
from another project) need to
 # ignore E402 -- module level import not at top level
@@ -1374,6 +1381,83 @@ combine-as-imports = true
 "tests/providers/qdrant/hooks/test_qdrant.py" = ["E402"]
 "tests/providers/qdrant/operators/test_qdrant.py" = ["E402"]
 
+# All the modules which do not follow D105 yet, please remove as soon as it 
becomes compatible
+"airflow/callbacks/callback_requests.py" = ["D105"]
+"airflow/cli/commands/task_command.py" = ["D105"]
+"airflow/configuration.py" = ["D105"]
+"airflow/datasets/__init__.py" = ["D105"]
+"airflow/decorators/base.py" = ["D105"]
+"airflow/decorators/setup_teardown.py" = ["D105"]
+"airflow/exceptions.py" = ["D105"]
+"airflow/executors/base_executor.py" = ["D105"]
+"airflow/io/path.py" = ["D105"]
+"airflow/io/store/__init__.py" = ["D105"]
+"airflow/kubernetes/pre_7_4_0_compatibility/secret.py" = ["D105"]
+"airflow/metrics/protocols.py" = ["D105"]
+"airflow/metrics/validators.py" = ["D105"]
+"airflow/models/abstractoperator.py" = ["D105"]
+"airflow/models/baseoperator.py" = ["D105"]
+"airflow/models/connection.py" = ["D105"]
+"airflow/models/dag.py" = ["D105"]
+"airflow/models/dagrun.py" = ["D105"]
+"airflow/models/dagwarning.py" = ["D105"]
+"airflow/models/dataset.py" = ["D105"]
+"airflow/models/expandinput.py" = ["D105"]
+"airflow/models/log.py" = ["D105"]
+"airflow/models/mappedoperator.py" = ["D105"]
+"airflow/models/param.py" = ["D105"]
+"airflow/models/pool.py" = ["D105"]
+"airflow/models/renderedtifields.py" = ["D105"]
+"airflow/models/serialized_dag.py" = ["D105"]
+"airflow/models/slamiss.py" = ["D105"]
+"airflow/models/taskfail.py" = ["D105"]
+"airflow/models/taskinstance.py" = ["D105"]
+"airflow/models/tasklog.py" = ["D105"]
+"airflow/models/taskmixin.py" = ["D105"]
+"airflow/models/variable.py" = ["D105"]
+"airflow/models/xcom.py" = ["D105"]
+"airflow/models/xcom_arg.py" = ["D105"]
+"airflow/plugins_manager.py" = ["D105"]
+"airflow/providers_manager.py" = ["D105"]
+"airflow/sensors/base.py" = ["D105"]
+"airflow/sensors/external_task.py" = ["D105"]
+"airflow/ti_deps/deps/base_ti_dep.py" = ["D105"]
+"airflow/ti_deps/deps/task_not_running_dep.py" = ["D105"]
+"airflow/ti_deps/deps/valid_state_dep.py" = ["D105"]
+"airflow/timetables/events.py" = ["D105"]
+"airflow/triggers/base.py" = ["D105"]
+"airflow/utils/context.py" = ["D105"]
+"airflow/utils/db.py" = ["D105"]
+"airflow/utils/log/secrets_masker.py" = ["D105"]
+"airflow/utils/operator_resources.py" = ["D105"]
+"airflow/utils/sqlalchemy.py" = ["D105"]
+"airflow/utils/state.py" = ["D105"]
+"airflow/utils/task_group.py" = ["D105"]
+"airflow/utils/timeout.py" = ["D105"]
+"airflow/utils/trigger_rule.py" = ["D105"]
+"airflow/utils/types.py" = ["D105"]
+"airflow/utils/weight_rule.py" = ["D105"]
+"airflow/providers/amazon/aws/exceptions.py" = ["D105"]
+"airflow/providers/amazon/aws/executors/ecs/utils.py" = ["D105"]
+"airflow/providers/amazon/aws/hooks/ecr.py" = ["D105"]
+"airflow/providers/amazon/aws/utils/connection_wrapper.py" = ["D105"]
+"airflow/providers/apache/hive/hooks/hive.py" = ["D105"]
+"airflow/providers/cncf/kubernetes/secret.py" = ["D105"]
+"airflow/providers/cncf/kubernetes/utils/delete_from.py" = ["D105"]

Re: [PR] Partially enable D105 rule [airflow]

[via GitHub]

potiuk merged PR #37462:
URL: https://github.com/apache/airflow/pull/37462


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Partially enable D105 rule [airflow]

[via GitHub]

Taragolis opened a new pull request, #37462:
URL: https://github.com/apache/airflow/pull/37462

   
   
   
   
   related: https://github.com/apache/airflow/issues/10742
   
   
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] Enable More PyDocStyle Checks [airflow]

[via GitHub]

Taragolis commented on issue #10742:
URL: https://github.com/apache/airflow/issues/10742#issuecomment-1947242983

   Just finish calculation other violations of D100-D107
   
   | **Rule** | **Total Errors** | **Uniq Core Modules** | **Uniq Providers 
Modules** | **Uniq Other Modules** |
   
|--|-:|--:|---:|---:|
   | _D100_   |  767 |   303 |  
  463 |  1 |
   | _D102_   | 2202 |   117 |  
  474 |  1 |
   | _D104_   |  104 |53 |  
  402 |  0 |
   | _D105_   |  237 |55 |  
   21 |  0 |
   | _D107_   | 1557 |   127 |  
  402 |  0 |
   
   > [!TIP]
   > D104 for providers packages should be resolved by change 
`PROVIDER__INIT__PY_TEMPLATE.py.jinja2` IN the 
`dev/breeze/src/airflow_breeze/templates` DIRECTORY
   
   If someone interested to look on possible exclusion lists, see attached 
archive 
[ruff-exclusions.zip](https://github.com/apache/airflow/files/14301973/ruff-exclusions.zip)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

[via GitHub]

potiuk commented on PR #36935:
URL: https://github.com/apache/airflow/pull/36935#issuecomment-1947220665

   This time I rebased it, but you can usually do it yourself without calling 
maintainers -> generally I recommend to rebase eaarly, often, and especially 
when you see you are behind and see unrelated errors.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Adding ability to automatically set DAG to off after X times it failed sequentially [airflow]

[via GitHub]

potiuk commented on PR #36935:
URL: https://github.com/apache/airflow/pull/36935#issuecomment-1947217035

   > The Test failures look like are coming from other changes cc. @eladkal 
@potiuk
   
   Yes. Usually rebasing helps. Those errors were found and fixed earlier today


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] add override option for app with the same name [airflow]

[via GitHub]

romsharon98 closed pull request #37447: add override option for app with the 
same name
URL: https://github.com/apache/airflow/pull/37447


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] add override option for app with the same name [airflow]

[via GitHub]

hussein-awala commented on code in PR #37447:
URL: https://github.com/apache/airflow/pull/37447#discussion_r1491561796


##
airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py:
##
@@ -255,6 +258,8 @@ def custom_obj_api(self) -> CustomObjectsApi:
 
 def execute(self, context: Context):
 self.log.info("Creating sparkApplication.")
+if self.override_existing:
+self.on_kill()

Review Comment:
   if you check the on_kill method, you find that it needs the launcher which 
is initiated just after this line; so your PR will not work.



##
airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py:
##
@@ -88,6 +89,7 @@ def __init__(
 reattach_on_restart: bool = True,
 delete_on_termination: bool = True,
 kubernetes_conn_id: str = "kubernetes_default",
+override_existing: bool = True,

Review Comment:
   The default behavior should stay the same:
   ```suggestion
   override_existing: bool = False,
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch main updated: Support type checking for third-party pytest's fixtures (#37457)

[husseinawala]

This is an automated email from the ASF dual-hosted git repository.

husseinawala pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new ef0ab856e4 Support type checking for third-party pytest's fixtures 
(#37457)
ef0ab856e4 is described below

commit ef0ab856e444f248c9304a26174310e09585b766
Author: Andrey Anshin 
AuthorDate: Thu Feb 15 23:46:01 2024 +0400

Support type checking for third-party pytest's fixtures (#37457)
---
 tests/conftest.py | 23 +++
 1 file changed, 23 insertions(+)

diff --git a/tests/conftest.py b/tests/conftest.py
index 3f7e640884..5e3a4115c4 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1291,3 +1291,26 @@ def configure_warning_output(config):
 
 
 # End of modified code from  
https://github.com/athinkingape/pytest-capture-warnings
+
+if TYPE_CHECKING:
+# Static checkers do not know about pytest fixtures' types and return,
+# In case if them distributed through third party packages.
+# This hack should help with autosuggestion in IDEs.
+from pytest_mock import MockerFixture
+from requests_mock.contrib.fixture import Fixture as RequestsMockFixture
+from time_machine import TimeMachineFixture
+
+# pytest-mock
+@pytest.fixture
+def mocker() -> MockerFixture:
+...
+
+# requests-mock
+@pytest.fixture
+def requests_mock() -> RequestsMockFixture:
+...
+
+# time-machine
+@pytest.fixture  # type: ignore[no-redef]
+def time_machine() -> TimeMachineFixture:
+...

Re: [PR] Support type checking for third-party pytest's fixtures [airflow]

[via GitHub]

hussein-awala merged PR #37457:
URL: https://github.com/apache/airflow/pull/37457


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Add Yandex Query support from Yandex.Cloud [airflow]

[via GitHub]

Taragolis commented on PR #37458:
URL: https://github.com/apache/airflow/pull/37458#issuecomment-1947076850

   And static checks and mypy should be fixed:
   - 
https://github.com/apache/airflow/actions/runs/7919541474/job/21623012828?pr=37458
   - 
https://github.com/apache/airflow/actions/runs/7919541474/job/21623011765?pr=37458
   
   See [Static Code 
Checks](https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst)
 for the reference 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Revert "KPO Maintain backward compatibility for execute_complete and … [airflow]

[via GitHub]

potiuk commented on PR #37446:
URL: https://github.com/apache/airflow/pull/37446#issuecomment-1946977154

   Thanks for super-quick diagnosis and fix @pankajastro  BTW ! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Revert "KPO Maintain backward compatibility for execute_complete and … [airflow]

[via GitHub]

potiuk commented on PR #37446:
URL: https://github.com/apache/airflow/pull/37446#issuecomment-1946967719

   Merged!
   
   That explains it indeed, It did not fail in your PR, because it would 
**generally** work without mocking (and worked for me as well locally) if run 
in isolation / on not so busy system. Generally PRs will only run a subset of 
tests that selective checks find relevant - so in the original PR only a small 
subsert of tests was run:
   
   ```
   Always
   Providers[amazon]
   
Providers[apache.beam,apache.cassandra,apache.flink,apache.spark,celery,cncf.kubernetes,common.sql,facebook,hashicorp,microsoft.azure,microsoft.mssql,mysql,openlineage,oracle,postgres,presto,sa
   Providers[google]
   ```
   
   In the new PR - I saw you added "full tests  needed" and the set of tests 
was the same as in `main`:
   
   ```
   Providers[-amazon,google]
   Core
   WWW
   CLI
   Other
   Serialization
   Always
   PythonVenv
   API
   BranchExternalPython
   BranchPythonVenv
   ExternalPython
   Operators
   PlainAsserts
   Providers[amazon]
   Providers[google]
   ```
   
   That's why original change was "green" because the set of tests run did not 
make the machine "busy enough" to trigger the flaky condition. In a way, our 
`main` (canary) tests are a good way to find such issues, because they run up 
to 8 parallel test suites at a time (out of the 16 above) in 8 complete 
docker-compose instances run on the same docker engine (each one with its own 
database). So we are REALLY stressing the poor AWS instance and that uncovers 
all kinds of races and flakiness (which as we saw here is a good idea).
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

(airflow) branch main updated: KPO Maintain backward compatibility for execute_complete and trigger run method (#37454)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
 new c84efe77a5 KPO Maintain backward compatibility for execute_complete 
and trigger run method (#37454)
c84efe77a5 is described below

commit c84efe77a5881d6bd554341b9bfc4712601051f2
Author: Pankaj Singh <98807258+pankajas...@users.noreply.github.com>
AuthorDate: Fri Feb 16 00:18:44 2024 +0530

KPO Maintain backward compatibility for execute_complete and trigger run 
method (#37454)

In #37279 I introduce periodic logging of the container.
During the process, I also changed a few event Dict key names
and that is problematic for someone extending the KPO trigger.
Also, the current execute_compelete method was unused in the KPO operator
and was problematic if someone using it in an extended class since
now the trigger can also emit an event even if the pod is in the pod 
intermediate state.
one reported issue: #37279 (comment)
In this PR I'm restoring the trigger event dict structure.
Also, deprecating the execute_complete method
---
 airflow/providers/cncf/kubernetes/operators/pod.py | 150 +
 airflow/providers/cncf/kubernetes/triggers/pod.py  |  70 +++---
 .../cncf/kubernetes/operators/test_pod.py  |  34 +++--
 .../providers/cncf/kubernetes/triggers/test_pod.py | 102 --
 .../cloud/triggers/test_kubernetes_engine.py   |  51 +++
 5 files changed, 215 insertions(+), 192 deletions(-)

diff --git a/airflow/providers/cncf/kubernetes/operators/pod.py 
b/airflow/providers/cncf/kubernetes/operators/pod.py
index 73389f4038..61442a6014 100644
--- a/airflow/providers/cncf/kubernetes/operators/pod.py
+++ b/airflow/providers/cncf/kubernetes/operators/pod.py
@@ -18,6 +18,7 @@
 
 from __future__ import annotations
 
+import datetime
 import json
 import logging
 import re
@@ -30,6 +31,7 @@ from functools import cached_property
 from typing import TYPE_CHECKING, Any, Callable, Iterable, Sequence
 
 import kubernetes
+from deprecated import deprecated
 from kubernetes.client import CoreV1Api, V1Pod, models as k8s
 from kubernetes.stream import stream
 from urllib3.exceptions import HTTPError
@@ -68,7 +70,6 @@ from airflow.providers.cncf.kubernetes.utils.pod_manager 
import (
 EMPTY_XCOM_RESULT,
 OnFinishAction,
 PodLaunchFailedException,
-PodLaunchTimeoutException,
 PodManager,
 PodNotFoundException,
 PodOperatorHookProtocol,
@@ -79,7 +80,6 @@ from airflow.providers.cncf.kubernetes.utils.pod_manager 
import (
 from airflow.settings import pod_mutation_hook
 from airflow.utils import yaml
 from airflow.utils.helpers import prune_dict, validate_key
-from airflow.utils.timezone import utcnow
 from airflow.version import version as airflow_version
 
 if TYPE_CHECKING:
@@ -656,7 +656,7 @@ class KubernetesPodOperator(BaseOperator):
 
 def invoke_defer_method(self, last_log_time: DateTime | None = None):
 """Redefine triggers which are being used in child classes."""
-trigger_start_time = utcnow()
+trigger_start_time = datetime.datetime.now(tz=datetime.timezone.utc)
 self.defer(
 trigger=KubernetesPodTrigger(
 pod_name=self.pod.metadata.name,  # type: ignore[union-attr]
@@ -678,117 +678,87 @@ class KubernetesPodOperator(BaseOperator):
 method_name="trigger_reentry",
 )
 
-@staticmethod
-def raise_for_trigger_status(event: dict[str, Any]) -> None:
-"""Raise exception if pod is not in expected state."""
-if event["status"] == "error":
-error_type = event["error_type"]
-description = event["description"]
-if error_type == "PodLaunchTimeoutException":
-raise PodLaunchTimeoutException(description)
-else:
-raise AirflowException(description)
-
 def trigger_reentry(self, context: Context, event: dict[str, Any]) -> Any:
 """
 Point of re-entry from trigger.
 
-If ``logging_interval`` is None, then at this point the pod should be 
done and we'll just fetch
+If ``logging_interval`` is None, then at this point, the pod should be 
done, and we'll just fetch
 the logs and exit.
 
-If ``logging_interval`` is not None, it could be that the pod is still 
running and we'll just
+If ``logging_interval`` is not None, it could be that the pod is still 
running, and we'll just
 grab the latest logs and defer back to the trigger again.
 """
-remote_pod = None
+self.pod = None
 try:
-self.pod_request_obj = self.build_pod_request_obj(context)
-self.pod = self.find_pod(
-namespace=self.namespace or 
self.pod_request_obj.metadata.namespace,
-

Re: [PR] KPO Maintain backward compatibility for execute_complete and trigger run method [airflow]

[via GitHub]

potiuk merged PR #37454:
URL: https://github.com/apache/airflow/pull/37454


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org