[PR] Refactor cloudpickle support in Python operators/decorators [airflow]

2024-04-25 Thread via GitHub
Taragolis opened a new pull request, #39270: URL: https://github.com/apache/airflow/pull/39270 Some refactor cloudpickle support which added into the https://github.com/apache/airflow/pull/38531 Use new `serializer` parameter for specify serialiser, this one for avoid

Re: [PR] Fix GCSObjectExistenceSensor xcom push [airflow]

2024-04-25 Thread via GitHub
github-actions[bot] commented on PR #37949: URL: https://github.com/apache/airflow/pull/37949#issuecomment-2078371960 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

Re: [PR] Refactor cloudpickle support in Python operators/decorators [airflow]

2024-04-25 Thread via GitHub
dstandish commented on code in PR #39270: URL: https://github.com/apache/airflow/pull/39270#discussion_r1580420376 ## airflow/operators/python.py: ## @@ -350,6 +343,36 @@ def get_tasks_to_skip(): return condition +def _load_pickle(): +import pickle + +

Re: [PR] Handle db isolation for mapped operators and task groups [airflow]

2024-04-25 Thread via GitHub
dstandish commented on code in PR #39259: URL: https://github.com/apache/airflow/pull/39259#discussion_r1580437417 ## airflow/models/taskinstance.py: ## @@ -3167,6 +3188,8 @@ def render_templates( # MappedOperator is useless for template rendering, and we need to be

Re: [PR] Handle db isolation for mapped operators and task groups [airflow]

2024-04-25 Thread via GitHub
dstandish commented on code in PR #39259: URL: https://github.com/apache/airflow/pull/39259#discussion_r1580450719 ## airflow/models/taskinstance.py: ## @@ -3167,6 +3188,8 @@ def render_templates( # MappedOperator is useless for template rendering, and we need to be

Re: [PR] Handle db isolation for mapped operators and task groups [airflow]

2024-04-25 Thread via GitHub
dstandish commented on code in PR #39259: URL: https://github.com/apache/airflow/pull/39259#discussion_r1580481269 ## airflow/models/taskinstance.py: ## @@ -3167,6 +3188,8 @@ def render_templates( # MappedOperator is useless for template rendering, and we need to be

Re: [I] SQLSensor does not time out [airflow]

2024-04-25 Thread via GitHub
funes79 commented on issue #39219: URL: https://github.com/apache/airflow/issues/39219#issuecomment-2078658780 Unfortunately it looks like it is a locked down parameter :-( Amazon MWAA reserves a set of critical environment variables. If you overwrite a reserved variable, Amazon

(airflow) branch main updated (e5bf91b7af -> 0759e6f891)

2024-04-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from e5bf91b7af Add stacklevel into the `suppress_and_warn` warning (#39263) add 0759e6f891 Remove deprecation

Re: [PR] Add tests for EmrServerlessJobSensor [airflow]

2024-04-25 Thread via GitHub
mateuslatrova commented on PR #39099: URL: https://github.com/apache/airflow/pull/39099#issuecomment-2078350187 Hi @vincbeck ! Thanks for your suggestion, the code became much more cleaner. I have just uploaded the changes, whenever you can review it again. Thanks! -- This is an

[I] Include `run_id` in TaskInstance Logging for Enhanced Traceability and Debugging [airflow]

2024-04-25 Thread via GitHub
RyuSA opened a new issue, #39272: URL: https://github.com/apache/airflow/issues/39272 ### Description Currently, the log outputs for TaskInstance state changes within `TaskInstance.py` do not include the `run_id`. i.e

Re: [I] Include `run_id` in TaskInstance Logging for Enhanced Traceability and Debugging [airflow]

2024-04-25 Thread via GitHub
boring-cyborg[bot] commented on issue #39272: URL: https://github.com/apache/airflow/issues/39272#issuecomment-2078642768 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

[PR] Test connection in Qdrant provider [airflow]

2024-04-25 Thread via GitHub
Anush008 opened a new pull request, #39273: URL: https://github.com/apache/airflow/pull/39273 ## Description This PR - Adds support for testing the connection to a Qdrant instance in the Qdrant provider. - Pins the version of the testing image to avoid unexpected test failures

Re: [PR] Add dag re-parsing request endpoint [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #39138: URL: https://github.com/apache/airflow/pull/39138#discussion_r1580493923 ## airflow/api_connexion/openapi/v1.yaml: ## @@ -3159,6 +3182,18 @@ components: *New in version 2.5.0* nullable: true +

Re: [PR] Expose AWS IAM missing param in Hashicorp secret [airflow]

2024-04-25 Thread via GitHub
pankajastro commented on code in PR #38536: URL: https://github.com/apache/airflow/pull/38536#discussion_r1580501806 ## airflow/providers/hashicorp/_internal_client/vault_client.py: ## @@ -318,15 +321,36 @@ def _auth_azure(self, _client: hvac.Client) -> None: )

Re: [I] Update Databricks provider to depend on databricks-sql-connector >= 3.1.0 [airflow]

2024-04-25 Thread via GitHub
boring-cyborg[bot] commented on issue #39274: URL: https://github.com/apache/airflow/issues/39274#issuecomment-2078677272 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

Re: [PR] Add dag re-parsing request endpoint [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #39138: URL: https://github.com/apache/airflow/pull/39138#discussion_r1580490108 ## airflow/api_connexion/openapi/v1.yaml: ## @@ -1091,6 +1091,29 @@ paths: "404": $ref: "#/components/responses/NotFound" +

Re: [PR] Handle db isolation for mapped operators and task groups [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #39259: URL: https://github.com/apache/airflow/pull/39259#discussion_r1580433108 ## airflow/models/taskinstance.py: ## @@ -3167,6 +3188,8 @@ def render_templates( # MappedOperator is useless for template rendering, and we need to be

Re: [PR] Allow Task Group Ids to be passed as branches in BranchMixIn [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #38883: URL: https://github.com/apache/airflow/pull/38883#discussion_r1580467975 ## airflow/operators/branch.py: ## @@ -34,8 +36,42 @@ class BranchMixIn(SkipMixin): def do_branch(self, context: Context, branches_to_execute: str |

Re: [I] Intermittent json errors from xcom sidecar [airflow]

2024-04-25 Thread via GitHub
paramjeet01 commented on issue #39267: URL: https://github.com/apache/airflow/issues/39267#issuecomment-2078254006 Also found this github thread, shall we implement this : https://github.com/kubernetes-client/python-base/issues/190#issuecomment-805073981 -- This is an automated message

Re: [PR] Add stacklevel into the `suppress_and_warn` warning [airflow]

2024-04-25 Thread via GitHub
potiuk merged PR #39263: URL: https://github.com/apache/airflow/pull/39263 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated (adeb7f7cba -> e5bf91b7af)

2024-04-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from adeb7f7cba Fix trigger kwarg encryption migration (#39246) add e5bf91b7af Add stacklevel into the

(airflow) branch constraints-main updated: Updating constraints. Github run id:8840676319

2024-04-25 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 39b49ff517 Updating

Re: [PR] Handle db isolation for mapped operators and task groups [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #39259: URL: https://github.com/apache/airflow/pull/39259#discussion_r1580465386 ## airflow/models/taskinstance.py: ## @@ -3167,6 +3188,8 @@ def render_templates( # MappedOperator is useless for template rendering, and we need to be

Re: [PR] Add dag re-parsing request endpoint [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #39138: URL: https://github.com/apache/airflow/pull/39138#discussion_r1580489823 ## airflow/api_connexion/endpoints/dag_parsing.py: ## @@ -0,0 +1,70 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Add tests for EmrServerlessJobSensor [airflow]

2024-04-25 Thread via GitHub
mateuslatrova commented on PR #39099: URL: https://github.com/apache/airflow/pull/39099#issuecomment-2078394365 Since this MR is still not merged, and both sensors (EmrServerlessJobSensor and EmrServerlessApplicationSensor) are very related, I have just added here a commit with the tests

Re: [PR] Refactor cloudpickle support in Python operators/decorators [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #39270: URL: https://github.com/apache/airflow/pull/39270#discussion_r1580419537 ## airflow/operators/python.py: ## @@ -350,6 +343,36 @@ def get_tasks_to_skip(): return condition +def _load_pickle(): +import pickle + +return

Re: [PR] Handle db isolation for mapped operators and task groups [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #39259: URL: https://github.com/apache/airflow/pull/39259#discussion_r1580432816 ## airflow/models/taskinstance.py: ## @@ -1003,25 +1009,40 @@ def _refresh_from_task( task_instance_mutation_hook(task_instance) +@internal_api_call

Re: [PR] Fix "matchExpressions" for "nodeAffinity" [airflow]

2024-04-25 Thread via GitHub
eladkal commented on code in PR #39255: URL: https://github.com/apache/airflow/pull/39255#discussion_r1580437965 ## docs/kubernetes.rst: ## Review Comment: Is this doc refrenced somewhere? It seems old and outdated (it even imports from contrib) cc @RNHTTR I assume we

Re: [PR] Handle db isolation for mapped operators and task groups [airflow]

2024-04-25 Thread via GitHub
dstandish commented on code in PR #39259: URL: https://github.com/apache/airflow/pull/39259#discussion_r1580450719 ## airflow/models/taskinstance.py: ## @@ -3167,6 +3188,8 @@ def render_templates( # MappedOperator is useless for template rendering, and we need to be

Re: [PR] Determine needs_expansion on client side when DB isolated [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on PR #39257: URL: https://github.com/apache/airflow/pull/39257#issuecomment-2078611765 > It doesn't (either completely or at all) include the dag object or its groups. And… we somehow don’t want it to do that? I feel we need to do a better job recreating the

Re: [PR] Allow Task Group Ids to be passed as branches in BranchMixIn [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #38883: URL: https://github.com/apache/airflow/pull/38883#discussion_r1580468363 ## airflow/operators/branch.py: ## @@ -34,8 +36,42 @@ class BranchMixIn(SkipMixin): def do_branch(self, context: Context, branches_to_execute: str |

(airflow) branch main updated: Better `db migrate` error messages (#39268)

2024-04-25 Thread uranusjr
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 8188da7663 Better `db migrate` error messages

Re: [PR] Better `db migrate` error messages [airflow]

2024-04-25 Thread via GitHub
uranusjr merged PR #39268: URL: https://github.com/apache/airflow/pull/39268 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Handle db isolation for mapped operators and task groups [airflow]

2024-04-25 Thread via GitHub
dstandish commented on code in PR #39259: URL: https://github.com/apache/airflow/pull/39259#discussion_r1580450719 ## airflow/models/taskinstance.py: ## @@ -3167,6 +3188,8 @@ def render_templates( # MappedOperator is useless for template rendering, and we need to be

Re: [PR] Fix "matchExpressions" for "nodeAffinity" [airflow]

2024-04-25 Thread via GitHub
amoghrajesh commented on PR #39255: URL: https://github.com/apache/airflow/pull/39255#issuecomment-2078601761 @chs2 can you please update the PR description? Its hard to tell what you are trying to achieve without a description -- This is an automated message from the Apache Git

Re: [PR] Determine needs_expansion on client side when DB isolated [airflow]

2024-04-25 Thread via GitHub
dstandish commented on PR #39257: URL: https://github.com/apache/airflow/pull/39257#issuecomment-2078635029 > > It doesn't (either completely or at all) include the dag object or its groups. > > And… we somehow don’t want it to do that? I feel we need to do a better job recreating

Re: [PR] Add dag re-parsing request endpoint [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #39138: URL: https://github.com/apache/airflow/pull/39138#discussion_r1580492653 ## airflow/api_connexion/openapi/v1.yaml: ## @@ -3159,6 +3182,18 @@ components: *New in version 2.5.0* nullable: true +

Re: [PR] Add dag re-parsing request endpoint [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #39138: URL: https://github.com/apache/airflow/pull/39138#discussion_r1580492845 ## airflow/models/dagbag.py: ## @@ -727,3 +734,41 @@ def _sync_perm_for_dag(cls, dag: DAG, session: Session = NEW_SESSION): security_manager =

[PR] fix fix reuse_existing_run wrong payload sent DbtCloudRunJobOperator [airflow]

2024-04-25 Thread via GitHub
Lee-W opened a new pull request, #39271: URL: https://github.com/apache/airflow/pull/39271 … --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] Remove deprecation warnings in `db migrate` tests [airflow]

2024-04-25 Thread via GitHub
potiuk merged PR #39269: URL: https://github.com/apache/airflow/pull/39269 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Determine needs_expansion on client side when DB isolated [airflow]

2024-04-25 Thread via GitHub
dstandish commented on PR #39257: URL: https://github.com/apache/airflow/pull/39257#issuecomment-2078641479 but like ... also... @uranusjr, using this example... there actually are some reasonable arguments for not serializing the complete task object. for one it's more efficient. why

Re: [PR] Add dag re-parsing request endpoint [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #39138: URL: https://github.com/apache/airflow/pull/39138#discussion_r1580506488 ## airflow/models/dagbag.py: ## @@ -727,3 +734,41 @@ def _sync_perm_for_dag(cls, dag: DAG, session: Session = NEW_SESSION): security_manager =

Re: [PR] fix: sqa deprecations for airflow core [airflow]

2024-04-25 Thread via GitHub
uranusjr merged PR #39211: URL: https://github.com/apache/airflow/pull/39211 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated: Fix SQLA deprecations in Airflow core (#39211)

2024-04-25 Thread uranusjr
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 6cb9deff97 Fix SQLA deprecations in Airflow core

[I] HTTPOperator should have the possibility to retry at the page level using the http hook run_with_advanced_retry [airflow]

2024-04-25 Thread via GitHub
lopezvit opened a new issue, #39247: URL: https://github.com/apache/airflow/issues/39247 ### Description I should be possible that the HTTPOperator retries some of the request without failing the whole task, since when it is using pagination, some pages might fail. ### Use

[PR] Feature: Support using content of kubeconfig to create KubernetesHook [airflow]

2024-04-25 Thread via GitHub
luoyuliuyin opened a new pull request, #39249: URL: https://github.com/apache/airflow/pull/39249 **PR of `Use kubeconfig as one of the optional parameters for creating KubernetesHook` https://github.com/apache/airflow/issues/39227** When we use `KubernetesPodOperator`, we need to

Re: [I] Getting 404 not found error when writing airflow logs to google cloud storage [airflow]

2024-04-25 Thread via GitHub
sai3563 closed issue #26448: Getting 404 not found error when writing airflow logs to google cloud storage URL: https://github.com/apache/airflow/issues/26448 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Fix DataprocSubmitJobOperator in deferrable mode=True when task is marked as failed. [airflow]

2024-04-25 Thread via GitHub
sunank200 commented on code in PR #39230: URL: https://github.com/apache/airflow/pull/39230#discussion_r1579122643 ## airflow/providers/google/cloud/triggers/dataproc.py: ## @@ -91,20 +93,28 @@ def serialize(self): "gcp_conn_id": self.gcp_conn_id,

Re: [PR] Fix trigger kwarg encryption migration [airflow]

2024-04-25 Thread via GitHub
jedcunningham commented on code in PR #39246: URL: https://github.com/apache/airflow/pull/39246#discussion_r1578968682 ## airflow/migrations/versions/0140_2_9_0_update_trigger_kwargs_type.py: ## @@ -38,13 +41,43 @@ airflow_version = "2.9.0" +def get_session() ->

Re: [PR] added a new condition before launching the self._scan_stale_dags() fu… [airflow]

2024-04-25 Thread via GitHub
tchakib commented on code in PR #39159: URL: https://github.com/apache/airflow/pull/39159#discussion_r1579012744 ## airflow/dag_processing/manager.py: ## @@ -599,7 +599,10 @@ def _run_parsing_loop(self): if self.standalone_dag_processor:

Re: [PR] Pinecone provider support for `pinecone-client`>=3 [airflow]

2024-04-25 Thread via GitHub
rawwar commented on code in PR #37307: URL: https://github.com/apache/airflow/pull/37307#discussion_r1579077234 ## airflow/providers/pinecone/provider.yaml: ## @@ -45,7 +45,7 @@ dependencies: # Pinecone Python SDK v3.0.0 was released at 2024-01-16 and introduce some

(airflow) branch constraints-main updated: Updating constraints. Github run id:8828523455

2024-04-25 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 4f0174f25a Updating

Re: [PR] Pinecone provider support for `pinecone-client`>=3 [airflow]

2024-04-25 Thread via GitHub
sunank200 commented on code in PR #37307: URL: https://github.com/apache/airflow/pull/37307#discussion_r1579126340 ## airflow/providers/pinecone/hooks/pinecone.py: ## @@ -64,43 +69,73 @@ def get_ui_field_behaviour(cls) -> dict[str, Any]: """Return custom field

Re: [PR] Add `DatabricksNotebookOperator` [airflow]

2024-04-25 Thread via GitHub
Lee-W commented on code in PR #39178: URL: https://github.com/apache/airflow/pull/39178#discussion_r1579117660 ## airflow/providers/databricks/operators/databricks.py: ## @@ -892,3 +892,171 @@ class DatabricksRunNowDeferrableOperator(DatabricksRunNowOperator): def

Re: [PR] Fix trigger kwarg encryption migration [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #39246: URL: https://github.com/apache/airflow/pull/39246#discussion_r1578966905 ## airflow/migrations/versions/0140_2_9_0_update_trigger_kwargs_type.py: ## @@ -38,13 +41,43 @@ airflow_version = "2.9.0" +def get_session() -> sa.orm.Session:

Re: [PR] Fix trigger kwarg encryption migration [airflow]

2024-04-25 Thread via GitHub
jedcunningham commented on code in PR #39246: URL: https://github.com/apache/airflow/pull/39246#discussion_r1578991552 ## airflow/models/trigger.py: ## @@ -116,7 +116,15 @@ def _decrypt_kwargs(encrypted_kwargs: str) -> dict[str, Any]: from airflow.models.crypto import

Re: [PR] Fix triggerer race condition in HA setting [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #38666: URL: https://github.com/apache/airflow/pull/38666#discussion_r1579075206 ## tests/jobs/test_triggerer_job.py: ## @@ -309,6 +309,88 @@ def test_update_trigger_with_triggerer_argument_change( assert "got an unexpected keyword

Re: [PR] Fix triggerer race condition in HA setting [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #38666: URL: https://github.com/apache/airflow/pull/38666#discussion_r1579076419 ## tests/jobs/test_triggerer_job.py: ## @@ -309,6 +309,88 @@ def test_update_trigger_with_triggerer_argument_change( assert "got an unexpected keyword

[PR] OpenAI Files & Vector Store Hooks [airflow]

2024-04-25 Thread via GitHub
nathadfield opened a new pull request, #39248: URL: https://github.com/apache/airflow/pull/39248 Adding a series of hook functions for OpenAI [File](https://platform.openai.com/docs/api-reference/files) and [Vector Store](https://platform.openai.com/docs/api-reference/vector-stores)

Re: [PR] Resolve deprecations in Python sensors/operators/decorators [airflow]

2024-04-25 Thread via GitHub
Taragolis commented on code in PR #39241: URL: https://github.com/apache/airflow/pull/39241#discussion_r1578961880 ## tests/deprecations_ignore.yml: ## @@ -349,23 +313,8 @@ -

Re: [PR] Add dag re-parsing request endpoint [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #39138: URL: https://github.com/apache/airflow/pull/39138#discussion_r1578976140 ## airflow/api_connexion/endpoints/dag_parsing.py: ## @@ -0,0 +1,70 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Add Grid button to Task Instance view [airflow]

2024-04-25 Thread via GitHub
csp33 commented on code in PR #39223: URL: https://github.com/apache/airflow/pull/39223#discussion_r1578974010 ## airflow/www/templates/airflow/task_instance.html: ## @@ -39,6 +39,9 @@ {{ url_for(endpoint, dag_id=dag.dag_id, task_id=task_id,

Re: [PR] Fix trigger kwarg encryption migration [airflow]

2024-04-25 Thread via GitHub
Lee-W commented on code in PR #39246: URL: https://github.com/apache/airflow/pull/39246#discussion_r1578993768 ## airflow/models/trigger.py: ## @@ -116,7 +116,15 @@ def _decrypt_kwargs(encrypted_kwargs: str) -> dict[str, Any]: from airflow.models.crypto import

Re: [PR] [FEAT] added notebook error in databricks deferrable handler [airflow]

2024-04-25 Thread via GitHub
pankajkoti commented on code in PR #39110: URL: https://github.com/apache/airflow/pull/39110#discussion_r1578987446 ## airflow/providers/databricks/triggers/databricks.py: ## @@ -84,13 +84,30 @@ async def run(self): async with self.hook: while True:

Re: [PR] Remove unnecessary validation from cncf provider. [airflow]

2024-04-25 Thread via GitHub
potiuk merged PR #39238: URL: https://github.com/apache/airflow/pull/39238 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Fix DataprocSubmitJobOperator in deferrable mode=True when task is marked as failed. [airflow]

2024-04-25 Thread via GitHub
pankajastro commented on code in PR #39230: URL: https://github.com/apache/airflow/pull/39230#discussion_r1578998258 ## airflow/providers/google/cloud/triggers/dataproc.py: ## @@ -91,20 +93,28 @@ def serialize(self): "gcp_conn_id": self.gcp_conn_id,

Re: [PR] Fix trigger kwarg encryption migration [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #39246: URL: https://github.com/apache/airflow/pull/39246#discussion_r1579040923 ## airflow/models/trigger.py: ## @@ -116,7 +116,15 @@ def _decrypt_kwargs(encrypted_kwargs: str) -> dict[str, Any]: from airflow.models.crypto import

Re: [PR] Fix trigger kwarg encryption migration [airflow]

2024-04-25 Thread via GitHub
jedcunningham commented on code in PR #39246: URL: https://github.com/apache/airflow/pull/39246#discussion_r1578971118 ## airflow/migrations/versions/0140_2_9_0_update_trigger_kwargs_type.py: ## @@ -38,13 +41,43 @@ airflow_version = "2.9.0" +def get_session() ->

Re: [PR] fix: sqa deprecations for airflow core [airflow]

2024-04-25 Thread via GitHub
Taragolis commented on code in PR #39211: URL: https://github.com/apache/airflow/pull/39211#discussion_r1578979638 ## airflow/www/views.py: ## @@ -3294,7 +3292,7 @@ def next_run_datasets(self, dag_id): latest_run = dag_model.get_last_dagrun(session=session)

Re: [PR] Add dag re-parsing request endpoint [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #39138: URL: https://github.com/apache/airflow/pull/39138#discussion_r1578976762 ## airflow/api_connexion/endpoints/dag_parsing.py: ## @@ -0,0 +1,70 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] [FEAT] added notebook error in databricks deferrable handler [airflow]

2024-04-25 Thread via GitHub
pankajkoti commented on code in PR #39110: URL: https://github.com/apache/airflow/pull/39110#discussion_r1578991339 ## airflow/providers/databricks/triggers/databricks.py: ## @@ -84,13 +84,30 @@ async def run(self): async with self.hook: while True:

Re: [PR] added a new condition before launching the self._scan_stale_dags() fu… [airflow]

2024-04-25 Thread via GitHub
tchakib commented on code in PR #39159: URL: https://github.com/apache/airflow/pull/39159#discussion_r1579051279 ## airflow/dag_processing/manager.py: ## @@ -599,7 +599,10 @@ def _run_parsing_loop(self): if self.standalone_dag_processor:

Re: [PR] added a new condition before launching the self._scan_stale_dags() fu… [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #39159: URL: https://github.com/apache/airflow/pull/39159#discussion_r1579059967 ## airflow/dag_processing/manager.py: ## @@ -599,7 +599,10 @@ def _run_parsing_loop(self): if self.standalone_dag_processor:

Re: [PR] Fix triggerer race condition in HA setting [airflow]

2024-04-25 Thread via GitHub
uranusjr commented on code in PR #38666: URL: https://github.com/apache/airflow/pull/38666#discussion_r1579074035 ## tests/jobs/test_triggerer_job.py: ## @@ -309,6 +309,88 @@ def test_update_trigger_with_triggerer_argument_change( assert "got an unexpected keyword

Re: [PR] Fix deferrable mode for DataflowTemplatedJobStartOperator and DataflowStartFlexTemplateOperator [airflow]

2024-04-25 Thread via GitHub
e-galan commented on code in PR #39018: URL: https://github.com/apache/airflow/pull/39018#discussion_r1579071214 ## airflow/providers/google/cloud/hooks/dataflow.py: ## @@ -736,6 +736,69 @@ def start_template_dataflow( jobs_controller.wait_for_done() return

Re: [PR] Fix trigger kwarg encryption migration [airflow]

2024-04-25 Thread via GitHub
Lee-W commented on code in PR #39246: URL: https://github.com/apache/airflow/pull/39246#discussion_r1578985358 ## airflow/models/trigger.py: ## @@ -116,7 +116,15 @@ def _decrypt_kwargs(encrypted_kwargs: str) -> dict[str, Any]: from airflow.models.crypto import

Re: [PR] Remove unnecessary validation from cncf provider. [airflow]

2024-04-25 Thread via GitHub
boring-cyborg[bot] commented on PR #39238: URL: https://github.com/apache/airflow/pull/39238#issuecomment-2076528004 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional

(airflow) branch main updated: Remove unnecessary validation from cncf provider. (#39238)

2024-04-25 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 4920ab25b3 Remove unnecessary validation from cncf

Re: [PR] Remove unnecessary validation from cncf provider. [airflow]

2024-04-25 Thread via GitHub
potiuk commented on PR #39238: URL: https://github.com/apache/airflow/pull/39238#issuecomment-2076527341 > LGTM. Do you want to make the minimum mandatory parameter? It looks like it's been there and it's a good idea in general to keep those to limit the jobs -- This is an

Re: [PR] Improve DataprocCreateClusterOperator in Triggers for Enhanced Error Handling and Resource Cleanup [airflow]

2024-04-25 Thread via GitHub
pankajastro commented on code in PR #39130: URL: https://github.com/apache/airflow/pull/39130#discussion_r1579015319 ## airflow/providers/google/cloud/triggers/dataproc.py: ## @@ -140,24 +153,74 @@ def serialize(self) -> tuple[str, dict[str, Any]]:

Re: [I] ShieldedInstanceConfig not being accepted by Dataproc operators ClusterGenerator [airflow]

2024-04-25 Thread via GitHub
sai3563 commented on issue #31058: URL: https://github.com/apache/airflow/issues/31058#issuecomment-2076657554 @eladkal Please assign this to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Use kubeconfig as one of the optional parameters for creating KubernetesHook [airflow]

2024-04-25 Thread via GitHub
luoyuliuyin commented on issue #39227: URL: https://github.com/apache/airflow/issues/39227#issuecomment-2076655845 I submitted a PR to support this feature, https://github.com/apache/airflow/pull/39249 -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] Getting 404 not found error when writing airflow logs to google cloud storage [airflow]

2024-04-25 Thread via GitHub
sai3563 commented on issue #26448: URL: https://github.com/apache/airflow/issues/26448#issuecomment-2076676153 @eladkal @potiuk I believe this is resolved now. I'm not seeing this error in my prod instance. I checked code of the latest version. ``` def gcs_write(self,

Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

2024-04-25 Thread via GitHub
pruoff commented on PR #37436: URL: https://github.com/apache/airflow/pull/37436#issuecomment-2076827199 > Let's rebase. Quite some effort this time. Included the display_dag_name logic -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Improve DataprocCreateClusterOperator in Triggers for Enhanced Error Handling and Resource Cleanup [airflow]

2024-04-25 Thread via GitHub
sunank200 commented on code in PR #39130: URL: https://github.com/apache/airflow/pull/39130#discussion_r1579266460 ## airflow/providers/google/cloud/triggers/dataproc.py: ## @@ -140,24 +153,74 @@ def serialize(self) -> tuple[str, dict[str, Any]]: "gcp_conn_id":

Re: [I] Edit connection fields dissapearing from web interface [airflow]

2024-04-25 Thread via GitHub
boring-cyborg[bot] commented on issue #39251: URL: https://github.com/apache/airflow/issues/39251#issuecomment-2076987348 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

Re: [I] The role role probably shouldn't have menu access to Plugins [airflow]

2024-04-25 Thread via GitHub
RocFang commented on issue #39218: URL: https://github.com/apache/airflow/issues/39218#issuecomment-2077058664 @potiuk Greate expanatation, thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Update Teradata Provider [airflow]

2024-04-25 Thread via GitHub
SatishChGit commented on code in PR #39208: URL: https://github.com/apache/airflow/pull/39208#discussion_r1579237191 ## airflow/providers/teradata/hooks/teradata.py: ## @@ -187,3 +198,59 @@ def get_ui_field_behaviour() -> dict: "password": "dbc",

Re: [PR] Improve DataprocCreateClusterOperator in Triggers for Enhanced Error Handling and Resource Cleanup [airflow]

2024-04-25 Thread via GitHub
sunank200 commented on code in PR #39130: URL: https://github.com/apache/airflow/pull/39130#discussion_r1579277263 ## airflow/providers/google/cloud/triggers/dataproc.py: ## @@ -140,24 +153,74 @@ def serialize(self) -> tuple[str, dict[str, Any]]: "gcp_conn_id":

Re: [PR] Pass SSL arg to all requests in DruidOperator [airflow]

2024-04-25 Thread via GitHub
danielbe11 commented on code in PR #39066: URL: https://github.com/apache/airflow/pull/39066#discussion_r1579313516 ## airflow/providers/apache/druid/hooks/druid.py: ## @@ -144,13 +144,15 @@ def submit_indexing_job( sec = 0 while running: -

Re: [PR] Pinecone provider support for `pinecone-client`>=3 [airflow]

2024-04-25 Thread via GitHub
sunank200 commented on code in PR #37307: URL: https://github.com/apache/airflow/pull/37307#discussion_r1579128144 ## airflow/providers/pinecone/CHANGELOG.rst: ## @@ -20,6 +20,36 @@ Changelog - +2.0.0 +. + +Breaking changes + + +.. warning:: +

Re: [PR] Fix DataprocSubmitJobOperator in deferrable mode=True when task is marked as failed. [airflow]

2024-04-25 Thread via GitHub
Lee-W commented on code in PR #39230: URL: https://github.com/apache/airflow/pull/39230#discussion_r1579150485 ## airflow/providers/google/cloud/triggers/dataproc.py: ## @@ -91,20 +93,28 @@ def serialize(self): "gcp_conn_id": self.gcp_conn_id,

Re: [PR] Add Grid button to Task Instance view [airflow]

2024-04-25 Thread via GitHub
csp33 commented on code in PR #39223: URL: https://github.com/apache/airflow/pull/39223#discussion_r1579197259 ## airflow/www/templates/airflow/task_instance.html: ## @@ -39,6 +39,9 @@ {{ url_for(endpoint, dag_id=dag.dag_id, task_id=task_id,

Re: [PR] Fix DataprocSubmitJobOperator in deferrable mode=True when task is marked as failed. [airflow]

2024-04-25 Thread via GitHub
sunank200 commented on code in PR #39230: URL: https://github.com/apache/airflow/pull/39230#discussion_r1579218640 ## airflow/providers/google/cloud/triggers/dataproc.py: ## @@ -91,20 +93,28 @@ def serialize(self): "gcp_conn_id": self.gcp_conn_id,

Re: [PR] Improve DataprocCreateClusterOperator in Triggers for Enhanced Error Handling and Resource Cleanup [airflow]

2024-04-25 Thread via GitHub
sunank200 commented on code in PR #39130: URL: https://github.com/apache/airflow/pull/39130#discussion_r1579226470 ## airflow/providers/google/cloud/triggers/dataproc.py: ## @@ -140,24 +153,74 @@ def serialize(self) -> tuple[str, dict[str, Any]]: "gcp_conn_id":

Re: [I] openlineage, celery: scheduler hanging when emitting lots of OL events via HTTP [airflow]

2024-04-25 Thread via GitHub
mobuchowski commented on issue #39232: URL: https://github.com/apache/airflow/issues/39232#issuecomment-2076951904 Hey @RNHTTR - this is the result of our internal investigation. We think someone knowledgeable with Celery might be able to help - if not, then we'd love to go with #39235 as

[I] Edit connection fields dissapearing from web interface [airflow]

2024-04-25 Thread via GitHub
suhren opened a new issue, #39251: URL: https://github.com/apache/airflow/issues/39251 ### Apache Airflow version 2.9.0 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? We have recently upgraded from Airflow 2.4.0 to

Re: [PR] Power BI Provider [airflow]

2024-04-25 Thread via GitHub
potiuk commented on PR #39243: URL: https://github.com/apache/airflow/pull/39243#issuecomment-2076988457 Just a kind reminder that proposal to add a new provider should be announced with justification - why you think the provider cannot be released and maintained outside of the community

Re: [PR] Deprecation of AutoML services: Add deprecation warnings and raise exceptions for already deprecated ones [airflow]

2024-04-25 Thread via GitHub
potiuk commented on PR #38673: URL: https://github.com/apache/airflow/pull/38673#issuecomment-2076993546 @molcay -> can you please review and address/mark as resolved if they are - the comments made by @shahar1 ? -- This is an automated message from the Apache Git Service. To respond to

Re: [I] The role role probably shouldn't have menu access to Plugins [airflow]

2024-04-25 Thread via GitHub
potiuk commented on issue #39218: URL: https://github.com/apache/airflow/issues/39218#issuecomment-2077032883 > Here i use the link of 2.8.4 because this page not available for 2.9.0 The doc links are now a bit wrong but will be fixed in 2.9.1 - and should point to FAB provider where

  1   2   3   >