Re: [PR] Add `retry_from_failure` parameter to DbtCloudRunJobOperator [airflow]

2024-04-26 Thread via GitHub
Lee-W commented on PR #38868: URL: https://github.com/apache/airflow/pull/38868#issuecomment-2078862568 > rerun endpoint does not accept body, which means parameters like steps_override, schema_override, threads_override, cause cannot be passed. Current implementation always uses rerun

Re: [PR] Add triggered_by field to DAG Run model to distinguish the source of a trigger [airflow]

2024-04-26 Thread via GitHub
molcay commented on code in PR #39165: URL: https://github.com/apache/airflow/pull/39165#discussion_r1580666420 ## airflow/api/common/trigger_dag.py: ## @@ -39,6 +39,7 @@ def _trigger_dag( conf: dict | str | None = None, execution_date: datetime | None = None,

Re: [PR] Add triggered_by field to DAG Run model to distinguish the source of a trigger [airflow]

2024-04-26 Thread via GitHub
molcay commented on code in PR #39165: URL: https://github.com/apache/airflow/pull/39165#discussion_r1580666420 ## airflow/api/common/trigger_dag.py: ## @@ -39,6 +39,7 @@ def _trigger_dag( conf: dict | str | None = None, execution_date: datetime | None = None,

Re: [PR] Refactor cloudpickle support in Python operators/decorators [airflow]

2024-04-26 Thread via GitHub
uranusjr commented on code in PR #39270: URL: https://github.com/apache/airflow/pull/39270#discussion_r1580686884 ## airflow/operators/python.py: ## @@ -350,6 +343,36 @@ def get_tasks_to_skip(): return condition +def _load_pickle(): +import pickle + +return

(airflow) branch main updated (bea1b7f70c -> 7683344c9c)

2024-04-26 Thread weilee
This is an automated email from the ASF dual-hosted git repository. weilee pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from bea1b7f70c Improve `DataprocCreateClusterOperator` Triggers for Better Error Handling and Resource Cleanup (#39130)

Re: [PR] Fix deferrable mode for DataflowTemplatedJobStartOperator and DataflowStartFlexTemplateOperator [airflow]

2024-04-26 Thread via GitHub
Lee-W commented on code in PR #39018: URL: https://github.com/apache/airflow/pull/39018#discussion_r1580693521 ## tests/providers/google/cloud/operators/test_dataflow.py: ## @@ -540,8 +559,9 @@ def test_validation_deferrable_params_raises_error(self):

Re: [PR] Refactor cloudpickle support in Python operators/decorators [airflow]

2024-04-26 Thread via GitHub
Taragolis commented on code in PR #39270: URL: https://github.com/apache/airflow/pull/39270#discussion_r1580742279 ## airflow/operators/python.py: ## @@ -350,6 +343,36 @@ def get_tasks_to_skip(): return condition +def _load_pickle(): +import pickle + +

[PR] Remove unused index `idx_last_scheduling_decision` on `dag_run` table [airflow]

2024-04-26 Thread via GitHub
pankajkoti opened a new pull request, #39275: URL: https://github.com/apache/airflow/pull/39275 We added `idx_last_scheduling_decision` on the `last_scheduling_decision` column in the `dag_run` table with Airflow 2.0.0. However, this index seems to have been unused with 0

Re: [PR] Scan stale dags [airflow]

2024-04-26 Thread via GitHub
tchakib closed pull request #39276: Scan stale dags URL: https://github.com/apache/airflow/pull/39276 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Refactor cloudpickle support in Python operators/decorators [airflow]

2024-04-26 Thread via GitHub
Taragolis commented on code in PR #39270: URL: https://github.com/apache/airflow/pull/39270#discussion_r1580837257 ## airflow/operators/python.py: ## @@ -350,6 +343,36 @@ def get_tasks_to_skip(): return condition +def _load_pickle(): +import pickle + +

Re: [I] KubernetesPodOperator fails due to getting logs [airflow]

2024-04-26 Thread via GitHub
antoniocorralsierra commented on issue #39239: URL: https://github.com/apache/airflow/issues/39239#issuecomment-2078841626 Yes, I would like it but It's my first time so I reading how do it -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Scan stale dags [airflow]

2024-04-26 Thread via GitHub
tchakib commented on PR #39276: URL: https://github.com/apache/airflow/pull/39276#issuecomment-2079054098 I took the wrong branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Edit connection fields dissapearing from web interface [airflow]

2024-04-26 Thread via GitHub
suhren commented on issue #39251: URL: https://github.com/apache/airflow/issues/39251#issuecomment-2079108205 Hi @RNHTTR. I am using apache-airflow-providers-apache-spark 4.7.2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] fix: sqa deprecations for airflow task cmd [airflow]

2024-04-26 Thread via GitHub
dondaum commented on code in PR #39244: URL: https://github.com/apache/airflow/pull/39244#discussion_r1580726639 ## airflow/cli/commands/task_command.py: ## @@ -199,6 +199,8 @@ def _get_ti_db_access( ) # TODO: Validate map_index is in range? ti =

Re: [PR] Fix deferrable mode for DataflowTemplatedJobStartOperator and DataflowStartFlexTemplateOperator [airflow]

2024-04-26 Thread via GitHub
e-galan commented on code in PR #39018: URL: https://github.com/apache/airflow/pull/39018#discussion_r1580752703 ## tests/providers/google/cloud/operators/test_dataflow.py: ## @@ -540,8 +559,9 @@ def test_validation_deferrable_params_raises_error(self):

[PR] Scan stale dags [airflow]

2024-04-26 Thread via GitHub
tchakib opened a new pull request, #39276: URL: https://github.com/apache/airflow/pull/39276 --- **^ Add meaningful description above** Read the **[Pull Request

Re: [I] KubernetesPodOperator fails due to getting logs [airflow]

2024-04-26 Thread via GitHub
pankajastro commented on issue #39239: URL: https://github.com/apache/airflow/issues/39239#issuecomment-2078830822 would you like to create a PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Feature: Support using content of kubeconfig to create KubernetesHook [airflow]

2024-04-26 Thread via GitHub
luoyuliuyin commented on code in PR #39249: URL: https://github.com/apache/airflow/pull/39249#discussion_r1580747809 ## tests/providers/cncf/kubernetes/hooks/test_kubernetes.py: ## @@ -279,6 +279,55 @@ def test_kube_config_path(

Re: [PR] Add method to get metadata from GCS blob in GCSHook [airflow]

2024-04-26 Thread via GitHub
molcay commented on PR #38398: URL: https://github.com/apache/airflow/pull/38398#issuecomment-2079000366 LGTM > Just a small thing: it can be good to squash the commits -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Pinecone provider support for `pinecone-client`>=3 [airflow]

2024-04-26 Thread via GitHub
Lee-W commented on code in PR #37307: URL: https://github.com/apache/airflow/pull/37307#discussion_r1580751492 ## airflow/providers/pinecone/hooks/pinecone.py: ## @@ -64,43 +69,73 @@ def get_ui_field_behaviour(cls) -> dict[str, Any]: """Return custom field

(airflow) branch constraints-main updated: Updating constraints. Github run id:8845555229

2024-04-26 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new d8194a3d29 Updating

Re: [PR] OpenAI Files & Vector Store Hooks [airflow]

2024-04-26 Thread via GitHub
nathadfield commented on code in PR #39248: URL: https://github.com/apache/airflow/pull/39248#discussion_r1580850336 ## airflow/providers/openai/hooks/openai.py: ## @@ -133,26 +143,26 @@ def get_assistants(self, **kwargs: Any) -> list[Assistant]: return

Re: [PR] OpenAI Files & Vector Store Hooks [airflow]

2024-04-26 Thread via GitHub
nathadfield commented on code in PR #39248: URL: https://github.com/apache/airflow/pull/39248#discussion_r1580858126 ## airflow/providers/openai/hooks/openai.py: ## @@ -133,26 +143,26 @@ def get_assistants(self, **kwargs: Any) -> list[Assistant]: return

Re: [PR] Pinecone provider support for `pinecone-client`>=3 [airflow]

2024-04-26 Thread via GitHub
rawwar commented on code in PR #37307: URL: https://github.com/apache/airflow/pull/37307#discussion_r1580869413 ## airflow/providers/pinecone/hooks/pinecone.py: ## @@ -64,43 +69,73 @@ def get_ui_field_behaviour(cls) -> dict[str, Any]: """Return custom field

(airflow) branch constraints-main updated: Updating constraints. Github run id:8844295044

2024-04-26 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 3e230e9a2b Updating

Re: [PR] Improve DataprocCreateClusterOperator in Triggers for Enhanced Error Handling and Resource Cleanup [airflow]

2024-04-26 Thread via GitHub
phanikumv merged PR #39130: URL: https://github.com/apache/airflow/pull/39130 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add dag re-parsing request endpoint [airflow]

2024-04-26 Thread via GitHub
Lee-W commented on code in PR #39138: URL: https://github.com/apache/airflow/pull/39138#discussion_r1580712852 ## airflow/models/dagbag.py: ## @@ -727,3 +734,41 @@ def _sync_perm_for_dag(cls, dag: DAG, session: Session = NEW_SESSION): security_manager =

Re: [PR] OpenAI Files & Vector Store Hooks [airflow]

2024-04-26 Thread via GitHub
Lee-W commented on code in PR #39248: URL: https://github.com/apache/airflow/pull/39248#discussion_r1580846341 ## airflow/providers/openai/hooks/openai.py: ## @@ -257,11 +287,137 @@ def create_embeddings( model: str = "text-embedding-ada-002", **kwargs: Any,

Re: [I] KubernetesPodOperator fails due to getting logs [airflow]

2024-04-26 Thread via GitHub
antoniocorralsierra commented on issue #39239: URL: https://github.com/apache/airflow/issues/39239#issuecomment-2078828703 Hi @pankajastro , Of course. I have added the following in the **_clean**: ```python def _clean(self, event: dict[str, Any]): if

Re: [PR] Fix DataprocSubmitJobOperator in deferrable mode=True when task is marked as failed. [airflow]

2024-04-26 Thread via GitHub
Lee-W commented on code in PR #39230: URL: https://github.com/apache/airflow/pull/39230#discussion_r1580633626 ## airflow/providers/google/cloud/triggers/dataproc.py: ## @@ -91,20 +104,41 @@ def serialize(self): "gcp_conn_id": self.gcp_conn_id,

Re: [PR] Add `retry_from_failure` parameter to DbtCloudRunJobOperator [airflow]

2024-04-26 Thread via GitHub
Lee-W commented on code in PR #38868: URL: https://github.com/apache/airflow/pull/38868#discussion_r1580638087 ## airflow/providers/dbt/cloud/hooks/dbt.py: ## @@ -424,11 +427,14 @@ def trigger_job_run( } payload.update(additional_run_config) -return

(airflow) branch main updated: Improve `DataprocCreateClusterOperator` Triggers for Better Error Handling and Resource Cleanup (#39130)

2024-04-26 Thread phanikumv
This is an automated email from the ASF dual-hosted git repository. phanikumv pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new bea1b7f70c Improve

Re: [PR] Feature: Support using content of kubeconfig to create KubernetesHook [airflow]

2024-04-26 Thread via GitHub
romsharon98 commented on code in PR #39249: URL: https://github.com/apache/airflow/pull/39249#discussion_r1580705916 ## tests/providers/cncf/kubernetes/hooks/test_kubernetes.py: ## @@ -279,6 +279,55 @@ def test_kube_config_path(

Re: [PR] [FEAT] raise exception with main notebook error in DatabricksRunNowDeferrableOperator [airflow]

2024-04-26 Thread via GitHub
Lee-W commented on code in PR #39110: URL: https://github.com/apache/airflow/pull/39110#discussion_r1580744308 ## airflow/providers/databricks/triggers/databricks.py: ## @@ -84,13 +84,30 @@ async def run(self): async with self.hook: while True:

Re: [PR] Pinecone provider support for `pinecone-client`>=3 [airflow]

2024-04-26 Thread via GitHub
rawwar commented on code in PR #37307: URL: https://github.com/apache/airflow/pull/37307#discussion_r1580860624 ## airflow/providers/pinecone/hooks/pinecone.py: ## @@ -64,43 +69,73 @@ def get_ui_field_behaviour(cls) -> dict[str, Any]: """Return custom field

Re: [PR] Pinecone provider support for `pinecone-client`>=3 [airflow]

2024-04-26 Thread via GitHub
rawwar commented on code in PR #37307: URL: https://github.com/apache/airflow/pull/37307#discussion_r1580864264 ## airflow/providers/pinecone/CHANGELOG.rst: ## @@ -20,6 +20,36 @@ Changelog - +2.0.0 +. + +Breaking changes + + +.. warning:: +

Re: [PR] Pinecone provider support for `pinecone-client`>=3 [airflow]

2024-04-26 Thread via GitHub
rawwar commented on code in PR #37307: URL: https://github.com/apache/airflow/pull/37307#discussion_r1580864264 ## airflow/providers/pinecone/CHANGELOG.rst: ## @@ -20,6 +20,36 @@ Changelog - +2.0.0 +. + +Breaking changes + + +.. warning:: +

Re: [PR] Deprecation of AutoML services: Add deprecation warnings and raise exceptions for already deprecated ones [airflow]

2024-04-26 Thread via GitHub
molcay commented on PR #38673: URL: https://github.com/apache/airflow/pull/38673#issuecomment-2078885594 @potiuk, Ohh I see, there are some hidden conversations. I totally missed them. Sorry for the inconvenience. I will try to address and resolve them as soon as possible. -- This is

Re: [PR] Add `DatabricksNotebookOperator` [airflow]

2024-04-26 Thread via GitHub
Lee-W merged PR #39178: URL: https://github.com/apache/airflow/pull/39178 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] fix: sqa deprecations for airflow task cmd [airflow]

2024-04-26 Thread via GitHub
dondaum commented on PR #39244: URL: https://github.com/apache/airflow/pull/39244#issuecomment-2078992353 > is there any way test the same? > > is there any way test the same? > > Yeah it would be great to have a test for it. > > A good test should test the

Re: [PR] Add dag re-parsing request endpoint [airflow]

2024-04-26 Thread via GitHub
uranusjr commented on code in PR #39138: URL: https://github.com/apache/airflow/pull/39138#discussion_r1580516110 ## airflow/models/dagbag.py: ## @@ -727,3 +734,41 @@ def _sync_perm_for_dag(cls, dag: DAG, session: Session = NEW_SESSION): security_manager =

Re: [PR] Add dag re-parsing request endpoint [airflow]

2024-04-26 Thread via GitHub
uranusjr commented on code in PR #39138: URL: https://github.com/apache/airflow/pull/39138#discussion_r1580516357 ## airflow/models/dagbag.py: ## @@ -727,3 +734,41 @@ def _sync_perm_for_dag(cls, dag: DAG, session: Session = NEW_SESSION): security_manager =

Re: [PR] Refactor cloudpickle support in Python operators/decorators [airflow]

2024-04-26 Thread via GitHub
Taragolis commented on code in PR #39270: URL: https://github.com/apache/airflow/pull/39270#discussion_r1580529944 ## airflow/operators/python.py: ## @@ -350,6 +343,36 @@ def get_tasks_to_skip(): return condition +def _load_pickle(): +import pickle + +

Re: [PR] added a new condition before launching the self._scan_stale_dags() fu… [airflow]

2024-04-26 Thread via GitHub
tchakib commented on code in PR #39159: URL: https://github.com/apache/airflow/pull/39159#discussion_r1580580045 ## airflow/config_templates/config.yml: ## @@ -500,6 +500,14 @@ core: type: integer example: ~ default: "4096" +enable_purging_stale_dags:

(airflow) branch main updated: Test connection to Qdrant (#39273)

2024-04-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 0c96b06204 Test connection to Qdrant (#39273)

Re: [PR] added a new condition before launching the self._scan_stale_dags() fu… [airflow]

2024-04-26 Thread via GitHub
uranusjr commented on code in PR #39159: URL: https://github.com/apache/airflow/pull/39159#discussion_r1580525996 ## airflow/config_templates/config.yml: ## @@ -500,6 +500,14 @@ core: type: integer example: ~ default: "4096" +enable_purging_stale_dags:

Re: [PR] fix: sqa deprecations for airflow task cmd [airflow]

2024-04-26 Thread via GitHub
uranusjr commented on code in PR #39244: URL: https://github.com/apache/airflow/pull/39244#discussion_r1580524472 ## airflow/cli/commands/task_command.py: ## @@ -199,6 +199,8 @@ def _get_ti_db_access( ) # TODO: Validate map_index is in range? ti =

Re: [PR] Refactor cloudpickle support in Python operators/decorators [airflow]

2024-04-26 Thread via GitHub
Taragolis commented on code in PR #39270: URL: https://github.com/apache/airflow/pull/39270#discussion_r1580560702 ## airflow/operators/python.py: ## @@ -350,6 +343,36 @@ def get_tasks_to_skip(): return condition +def _load_pickle(): +import pickle + +

Re: [PR] Refactor cloudpickle support in Python operators/decorators [airflow]

2024-04-26 Thread via GitHub
Taragolis commented on code in PR #39270: URL: https://github.com/apache/airflow/pull/39270#discussion_r1580569241 ## airflow/operators/python.py: ## @@ -350,6 +343,36 @@ def get_tasks_to_skip(): return condition +def _load_pickle(): +import pickle + +

Re: [PR] Add dag re-parsing request endpoint [airflow]

2024-04-26 Thread via GitHub
uranusjr commented on code in PR #39138: URL: https://github.com/apache/airflow/pull/39138#discussion_r1580515745 ## airflow/models/dagbag.py: ## @@ -727,3 +734,41 @@ def _sync_perm_for_dag(cls, dag: DAG, session: Session = NEW_SESSION): security_manager =

Re: [PR] Add dag re-parsing request endpoint [airflow]

2024-04-26 Thread via GitHub
uranusjr commented on code in PR #39138: URL: https://github.com/apache/airflow/pull/39138#discussion_r1580515385 ## airflow/models/dagbag.py: ## @@ -727,3 +734,41 @@ def _sync_perm_for_dag(cls, dag: DAG, session: Session = NEW_SESSION): security_manager =

Re: [PR] added a new condition before launching the self._scan_stale_dags() fu… [airflow]

2024-04-26 Thread via GitHub
uranusjr commented on code in PR #39159: URL: https://github.com/apache/airflow/pull/39159#discussion_r1580525790 ## airflow/config_templates/config.yml: ## @@ -500,6 +500,14 @@ core: type: integer example: ~ default: "4096" +enable_purging_stale_dags:

Re: [PR] Refactor cloudpickle support in Python operators/decorators [airflow]

2024-04-26 Thread via GitHub
Taragolis commented on code in PR #39270: URL: https://github.com/apache/airflow/pull/39270#discussion_r1580556503 ## airflow/operators/python.py: ## @@ -350,6 +343,36 @@ def get_tasks_to_skip(): return condition +def _load_pickle(): +import pickle + +

Re: [PR] Refactor cloudpickle support in Python operators/decorators [airflow]

2024-04-26 Thread via GitHub
Taragolis commented on code in PR #39270: URL: https://github.com/apache/airflow/pull/39270#discussion_r1580581663 ## tests/operators/test_python.py: ## @@ -1159,64 +1201,36 @@ def f(a): # context using cloudpickle (which is slow apparently).

Re: [PR] Refactor cloudpickle support in Python operators/decorators [airflow]

2024-04-26 Thread via GitHub
uranusjr commented on code in PR #39270: URL: https://github.com/apache/airflow/pull/39270#discussion_r1580542424 ## airflow/operators/python.py: ## @@ -350,6 +343,36 @@ def get_tasks_to_skip(): return condition +def _load_pickle(): +import pickle + +return

Re: [PR] Add triggered_by field to DAG Run model to distinguish the source of a trigger [airflow]

2024-04-26 Thread via GitHub
uranusjr commented on code in PR #39165: URL: https://github.com/apache/airflow/pull/39165#discussion_r1580539589 ## airflow/api/common/trigger_dag.py: ## @@ -39,6 +39,7 @@ def _trigger_dag( conf: dict | str | None = None, execution_date: datetime | None = None,

Re: [PR] Refactor cloudpickle support in Python operators/decorators [airflow]

2024-04-26 Thread via GitHub
Taragolis commented on code in PR #39270: URL: https://github.com/apache/airflow/pull/39270#discussion_r1580570825 ## airflow/operators/python.py: ## @@ -350,6 +343,36 @@ def get_tasks_to_skip(): return condition +def _load_pickle(): +import pickle + +

Re: [PR] Refactor cloudpickle support in Python operators/decorators [airflow]

2024-04-26 Thread via GitHub
Taragolis commented on code in PR #39270: URL: https://github.com/apache/airflow/pull/39270#discussion_r1580569241 ## airflow/operators/python.py: ## @@ -350,6 +343,36 @@ def get_tasks_to_skip(): return condition +def _load_pickle(): +import pickle + +

Re: [PR] Refactor cloudpickle support in Python operators/decorators [airflow]

2024-04-26 Thread via GitHub
Taragolis commented on code in PR #39270: URL: https://github.com/apache/airflow/pull/39270#discussion_r1580581663 ## tests/operators/test_python.py: ## @@ -1159,64 +1201,36 @@ def f(a): # context using cloudpickle (which is slow apparently).

Re: [PR] Test connection in Qdrant provider [airflow]

2024-04-26 Thread via GitHub
potiuk merged PR #39273: URL: https://github.com/apache/airflow/pull/39273 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] added a new condition before launching the self._scan_stale_dags() fu… [airflow]

2024-04-26 Thread via GitHub
uranusjr commented on code in PR #39159: URL: https://github.com/apache/airflow/pull/39159#discussion_r1580528227 ## docs/apache-airflow/core-concepts/dags.rst: ## @@ -924,3 +924,10 @@ if it fails for ``N`` number of times consecutively. we can also provide and override these

Re: [PR] Refactor cloudpickle support in Python operators/decorators [airflow]

2024-04-26 Thread via GitHub
Taragolis commented on code in PR #39270: URL: https://github.com/apache/airflow/pull/39270#discussion_r1580529944 ## airflow/operators/python.py: ## @@ -350,6 +343,36 @@ def get_tasks_to_skip(): return condition +def _load_pickle(): +import pickle + +

Re: [I] Decision for broken task log prefix feature [airflow]

2024-04-26 Thread via GitHub
dstandish commented on issue #39019: URL: https://github.com/apache/airflow/issues/39019#issuecomment-2079696340 I think we probably could just remove it. It would be sorta not too bad if someone was actually using it because the cluster would no run I think so hopefully they'd discover

Re: [PR] Pinecone provider support for `pinecone-client`>=3 [airflow]

2024-04-26 Thread via GitHub
sunank200 commented on code in PR #37307: URL: https://github.com/apache/airflow/pull/37307#discussion_r1581266062 ## airflow/providers/pinecone/hooks/pinecone.py: ## @@ -135,119 +170,106 @@ def upsert( **kwargs, ) -@staticmethod +def

Re: [PR] Fix visible DagRun doesn't point to the same dag run id [airflow]

2024-04-26 Thread via GitHub
jedcunningham commented on PR #38365: URL: https://github.com/apache/airflow/pull/38365#issuecomment-2079806249 Gentle bump @bbovenzi. I do wonder if both places should use logical_date instead though. -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Make audit log before/after filterable [airflow]

2024-04-26 Thread via GitHub
bbovenzi merged PR #39120: URL: https://github.com/apache/airflow/pull/39120 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated (6d09adf992 -> a7e5510dbb)

2024-04-26 Thread bbovenzi
This is an automated email from the ASF dual-hosted git repository. bbovenzi pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 6d09adf992 Add tests for `EmrServerlessJobSensor` and `EmrServerlessApplicationSensor` (#39099) add a7e5510dbb

Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

2024-04-26 Thread via GitHub
jscheffl commented on code in PR #37436: URL: https://github.com/apache/airflow/pull/37436#discussion_r1581455829 ## airflow/www/views.py: ## @@ -5415,51 +5453,78 @@ class AutocompleteView(AirflowBaseView): def autocomplete(self, session: Session = NEW_SESSION):

Re: [PR] Fix broken task_log_prefix_template by adding it to FileTaskHandler itself. [airflow]

2024-04-26 Thread via GitHub
abhishekbhakat commented on PR #38709: URL: https://github.com/apache/airflow/pull/38709#issuecomment-2079673916 I tried, but couldn't get it working without just using an entirely different Handler. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] Decision for broken task log prefix feature [airflow]

2024-04-26 Thread via GitHub
jedcunningham commented on issue #39019: URL: https://github.com/apache/airflow/issues/39019#issuecomment-2079679978 Yep exactly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] kubernetes_executor does not properly fail task-instances in cleanup_stuck_queued_tasks [airflow]

2024-04-26 Thread via GitHub
waldoppper commented on issue #39078: URL: https://github.com/apache/airflow/issues/39078#issuecomment-2079738158 We've tried manually deleting the task instance with no luck. (this makes me think that addressing this particular issue may not help us) The only workaround I'm aware of

Re: [PR] Fix "matchExpressions" for "nodeAffinity" [airflow]

2024-04-26 Thread via GitHub
RNHTTR commented on code in PR #39255: URL: https://github.com/apache/airflow/pull/39255#discussion_r1581305660 ## docs/kubernetes.rst: ## Review Comment: I don't think so. It seems correct in the [docs for the current version of the

Re: [I] Rendering custom map index before task is run. [airflow]

2024-04-26 Thread via GitHub
RNHTTR commented on issue #39118: URL: https://github.com/apache/airflow/issues/39118#issuecomment-2079818129 @tomrutter Would you like to be assigned? Given the current design (using a task's `context` object) I think this might be trickier than meets the eye. -- This is an

Re: [PR] Fix visible DagRun doesn't point to the same dag run id [airflow]

2024-04-26 Thread via GitHub
bbovenzi commented on PR #38365: URL: https://github.com/apache/airflow/pull/38365#issuecomment-2079949143 Let's go for logical date then -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

2024-04-26 Thread via GitHub
bbovenzi commented on code in PR #37436: URL: https://github.com/apache/airflow/pull/37436#discussion_r1581435304 ## airflow/www/views.py: ## @@ -5415,51 +5453,78 @@ class AutocompleteView(AirflowBaseView): def autocomplete(self, session: Session = NEW_SESSION):

Re: [PR] Amazon Bedrock - Knowledge Bases and Data Sources [airflow]

2024-04-26 Thread via GitHub
ferruzzi commented on code in PR #39245: URL: https://github.com/apache/airflow/pull/39245#discussion_r1581460944 ## airflow/providers/amazon/aws/sensors/bedrock.py: ## @@ -68,7 +70,56 @@ def __init__( super().__init__(**kwargs) self.deferrable = deferrable

Re: [PR] Fix broken task_log_prefix_template by adding it to FileTaskHandler itself. [airflow]

2024-04-26 Thread via GitHub
abhishekbhakat closed pull request #38709: Fix broken task_log_prefix_template by adding it to FileTaskHandler itself. URL: https://github.com/apache/airflow/pull/38709 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Fix broken task_log_prefix_template by adding it to FileTaskHandler itself. [airflow]

2024-04-26 Thread via GitHub
abhishekbhakat commented on PR #38709: URL: https://github.com/apache/airflow/pull/38709#issuecomment-2079701806 Closing this PR in favor of deprecating this and more discussions at #39019. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Remove unused index `idx_last_scheduling_decision` on `dag_run` table [airflow]

2024-04-26 Thread via GitHub
utkarsharma2 commented on code in PR #39275: URL: https://github.com/apache/airflow/pull/39275#discussion_r1580976245 ## airflow/migrations/versions/0069_2_0_0_add_scheduling_decision_to_dagrun_and_.py: ## @@ -48,7 +48,9 @@ def upgrade(): with

Re: [I] Edit connection fields dissapearing from web interface [airflow]

2024-04-26 Thread via GitHub
eladkal commented on issue #39251: URL: https://github.com/apache/airflow/issues/39251#issuecomment-2079313744 I think this was fixed in https://github.com/apache/airflow/pull/39229 ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] Partially resolve builtin variable shadowing (`A001`) [airflow]

2024-04-26 Thread via GitHub
Taragolis opened a new pull request, #39278: URL: https://github.com/apache/airflow/pull/39278 We have plenty of the builtins variables/function with the same name as a builtins functions or constants . This PR rename it into the non production codebase (dev, tests) and

Re: [PR] openlineage: use `ProcessPoolExecutor` over `ThreadPoolExecutor`. [airflow]

2024-04-26 Thread via GitHub
JDarDagran commented on code in PR #39235: URL: https://github.com/apache/airflow/pull/39235#discussion_r1581034466 ## airflow/providers/openlineage/plugins/listener.py: ## @@ -250,7 +250,7 @@ def on_failure(): @property def executor(self): if not

Re: [PR] Expose AWS IAM missing param in Hashicorp secret [airflow]

2024-04-26 Thread via GitHub
pankajastro commented on code in PR #38536: URL: https://github.com/apache/airflow/pull/38536#discussion_r1581163636 ## airflow/providers/hashicorp/_internal_client/vault_client.py: ## @@ -318,15 +321,36 @@ def _auth_azure(self, _client: hvac.Client) -> None: )

Re: [PR] Fix broken task_log_prefix_template by adding it to FileTaskHandler itself. [airflow]

2024-04-26 Thread via GitHub
jedcunningham commented on PR #38709: URL: https://github.com/apache/airflow/pull/38709#issuecomment-2079655572 Sorry, didn't mean to split the conversation, but left some more thoughts on #39019. I've removed this from 2.9.1 since we don't know how we will proceed yet. -- This is

Re: [I] Decision for broken task log prefix feature [airflow]

2024-04-26 Thread via GitHub
dstandish commented on issue #39019: URL: https://github.com/apache/airflow/issues/39019#issuecomment-2079658176 > So the question is whether we can cut it now or not. It'd be possible for a user to swap in TaskHandlerWithCustomFormatter in airflow_local_settings, right? sorry

Re: [I] ShieldedInstanceConfig not being accepted by Dataproc operators ClusterGenerator [airflow]

2024-04-26 Thread via GitHub
sai3563 commented on issue #31058: URL: https://github.com/apache/airflow/issues/31058#issuecomment-2080009300 @eladkal @shivannakarthik It seems no modifications in the `DataprocCreateClusterOperator` are needed. While testing with the latest dev branch i.e. `2.10.0.dev0`, I was able to

Re: [PR] openlineage: use `ProcessPoolExecutor` over `ThreadPoolExecutor`. [airflow]

2024-04-26 Thread via GitHub
JDarDagran commented on code in PR #39235: URL: https://github.com/apache/airflow/pull/39235#discussion_r1581254797 ## airflow/providers/openlineage/plugins/listener.py: ## @@ -250,7 +250,7 @@ def on_failure(): @property def executor(self): if not

Re: [I] Rendering custom map index before task is run. [airflow]

2024-04-26 Thread via GitHub
raphaelauv commented on issue #39118: URL: https://github.com/apache/airflow/issues/39118#issuecomment-2079827043 since we can define at operator definition the field `map_index_template="{{ task.op_kwargs['date'] }}",`, it can be render by the webserver -- This is an automated message

Re: [PR] Fix DataprocSubmitJobOperator in deferrable mode=True when task is marked as failed. [airflow]

2024-04-26 Thread via GitHub
sunank200 commented on code in PR #39230: URL: https://github.com/apache/airflow/pull/39230#discussion_r1580936977 ## airflow/providers/google/cloud/triggers/dataproc.py: ## @@ -91,20 +104,41 @@ def serialize(self): "gcp_conn_id": self.gcp_conn_id,

[PR] Make task log messages include run_id [airflow]

2024-04-26 Thread via GitHub
RyuSA opened a new pull request, #39280: URL: https://github.com/apache/airflow/pull/39280 closes: #39272 This PR changes the TaskInstance log format which includes `run_id`. - Pausing task as DEFERRED: dag_id=Test1, task_id=task1, execution_date=...,

Re: [PR] Fix pre-mature evaluation of tasks in mapped task group [airflow]

2024-04-26 Thread via GitHub
eladkal commented on PR #36462: URL: https://github.com/apache/airflow/pull/36462#issuecomment-2079343702 @avkirilishin are you still working on this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Feature: Support using content of kubeconfig to create KubernetesHook [airflow]

2024-04-26 Thread via GitHub
luoyuliuyin commented on PR #39249: URL: https://github.com/apache/airflow/pull/39249#issuecomment-2079363137 > I believe this is a one time thing, do we need to expose this? kube config text may contain tokens. So, we need to secure them. It is indeed one-time for the airflow user,

Re: [PR] Expose AWS IAM missing param in Hashicorp secret [airflow]

2024-04-26 Thread via GitHub
ReadytoRocc commented on code in PR #38536: URL: https://github.com/apache/airflow/pull/38536#discussion_r1581083971 ## airflow/providers/hashicorp/_internal_client/vault_client.py: ## @@ -318,15 +321,36 @@ def _auth_azure(self, _client: hvac.Client) -> None: )

Re: [PR] Remove unused index `idx_last_scheduling_decision` on `dag_run` table [airflow]

2024-04-26 Thread via GitHub
vincbeck commented on code in PR #39275: URL: https://github.com/apache/airflow/pull/39275#discussion_r1581084139 ## airflow/migrations/versions/0069_2_0_0_add_scheduling_decision_to_dagrun_and_.py: ## @@ -48,7 +48,9 @@ def upgrade(): with op.batch_alter_table("dag_run",

Re: [PR] Remove unused index `idx_last_scheduling_decision` on `dag_run` table [airflow]

2024-04-26 Thread via GitHub
jedcunningham commented on code in PR #39275: URL: https://github.com/apache/airflow/pull/39275#discussion_r1581105685 ## airflow/migrations/versions/0142_2_9_1_remove_idx_last_scheduling_decision_.py: ## @@ -0,0 +1,50 @@ +# +# Licensed to the Apache Software Foundation (ASF)

Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

2024-04-26 Thread via GitHub
pruoff commented on PR #37436: URL: https://github.com/apache/airflow/pull/37436#issuecomment-2079612257 > Do we want to include searching by task_display_name in this PR or in a follow up? Searching by task_display_name is already included and the PR does not change this. --

Re: [I] Decision for broken task log prefix feature [airflow]

2024-04-26 Thread via GitHub
abhishekbhakat commented on issue #39019: URL: https://github.com/apache/airflow/issues/39019#issuecomment-2079660933 > It'd be possible for a user to swap in TaskHandlerWithCustomFormatter in airflow_local_settings, right? Nah, it's inheriting StreamHandler which is not at used for

Re: [I] Decision for broken task log prefix feature [airflow]

2024-04-26 Thread via GitHub
jedcunningham commented on issue #39019: URL: https://github.com/apache/airflow/issues/39019#issuecomment-2079662405 > > So the question is whether we can cut it now or not. It'd be possible for a user to swap in TaskHandlerWithCustomFormatter in airflow_local_settings, right? > >

Re: [PR] Remove unused index `idx_last_scheduling_decision` on `dag_run` table [airflow]

2024-04-26 Thread via GitHub
pankajkoti commented on code in PR #39275: URL: https://github.com/apache/airflow/pull/39275#discussion_r1581018600 ## docs/apache-airflow/migrations-ref.rst: ## @@ -39,7 +39,10 @@ Here's the list of all the Database Migrations that are executed via when you ru

[PR] Add assume_role_kwargs in hashicorp backend config [airflow]

2024-04-26 Thread via GitHub
pankajastro opened a new pull request, #39279: URL: https://github.com/apache/airflow/pull/39279 --- **^ Add meaningful description above** Read the **[Pull Request

  1   2   3   >