Re: [I] Unable to open AirFlow webserver UI [airflow]

2024-05-17 Thread via GitHub
Taragolis closed issue #39697: Unable to open AirFlow webserver UI URL: https://github.com/apache/airflow/issues/39697 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Add rpc server to helm chart [airflow]

2024-05-17 Thread via GitHub
dstandish commented on PR #38549: URL: https://github.com/apache/airflow/pull/38549#issuecomment-2118632135 > Let's prefix with `_`, or maybe `_aip44RpcServer`? I'm hesitant of a docs only change. Did it. Went with `_rpcServer`. I think that's definitely clear enough.

Re: [PR] Add rpc server to helm chart [airflow]

2024-05-17 Thread via GitHub
dstandish commented on code in PR #38549: URL: https://github.com/apache/airflow/pull/38549#discussion_r1605669686 ## helm_tests/webserver/test_rpc_server.py: ## @@ -0,0 +1,923 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

[PR] Print stderr from helm test failures [airflow]

2024-05-17 Thread via GitHub
dstandish opened a new pull request, #39698: URL: https://github.com/apache/airflow/pull/39698 Previously you would not see the error message; you'd only see that the command failed. Now you'll see it. Here's an example of what it looks like now: ```

Re: [PR] fix: delete pod when base container completes [airflow]

2024-05-17 Thread via GitHub
dirrao commented on code in PR #39694: URL: https://github.com/apache/airflow/pull/39694#discussion_r1605659696 ## airflow/providers/cncf/kubernetes/operators/pod.py: ## @@ -842,16 +838,14 @@ def cleanup(self, pod: k8s.V1Pod, remote_pod: k8s.V1Pod): if self._killed or

Re: [I] DataprocCreateClusterOperator doesn't read softwareConfig properties as it should [airflow]

2024-05-17 Thread via GitHub
github-actions[bot] commented on issue #38127: URL: https://github.com/apache/airflow/issues/38127#issuecomment-2118514106 This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further

Re: [PR] changing dag_processing.processes from UpDownCounter to guage [airflow]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #38400: URL: https://github.com/apache/airflow/pull/38400#issuecomment-2118514080 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

Re: [PR] Fix CronTriggerTimetable candidates to reflect doc [airflow]

2024-05-17 Thread via GitHub
github-actions[bot] commented on PR #38669: URL: https://github.com/apache/airflow/pull/38669#issuecomment-2118514058 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

Re: [I] Unable to open AirFlow webserver UI [airflow]

2024-05-17 Thread via GitHub
boring-cyborg[bot] commented on issue #39697: URL: https://github.com/apache/airflow/issues/39697#issuecomment-2118485888 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

[I] Unable to open AirFlow webserver UI [airflow]

2024-05-17 Thread via GitHub
MattSalminen opened a new issue, #39697: URL: https://github.com/apache/airflow/issues/39697 ### Apache Airflow version 2.9.1 ### If "Other Airflow 2 version" selected, which one? 2.9.0+astro.1 ### What happened? When the docker container is running and I

Re: [PR] More typing in TimeSensor and TimeSensorAsync [airflow]

2024-05-17 Thread via GitHub
Taragolis commented on code in PR #39696: URL: https://github.com/apache/airflow/pull/39696#discussion_r1605597936 ## airflow/sensors/time_sensor.py: ## @@ -72,13 +72,13 @@ def __init__(self, *, target_time, **kwargs): self.target_datetime =

(airflow) branch main updated: Add args to docker service ContainerSpec (#39464)

2024-05-17 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 77a6b4f419 Add args to docker service

Re: [PR] Add args to docker service ContainerSpec [airflow]

2024-05-17 Thread via GitHub
Taragolis merged PR #39464: URL: https://github.com/apache/airflow/pull/39464 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] fix: delete pod when base container completes [airflow]

2024-05-17 Thread via GitHub
jonathan-ostrander commented on PR #39694: URL: https://github.com/apache/airflow/pull/39694#issuecomment-2118361921 @jedcunningham I'm not sure I understand your point about `OnFinishAction.KEEP_POD` or why the istio side car needs to be treated as a special case. Shouldn't that finish

[PR] More typing in TimeSensor and TimeSensorAsync [airflow]

2024-05-17 Thread via GitHub
jedcunningham opened a new pull request, #39696: URL: https://github.com/apache/airflow/pull/39696 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[PR] Introduce anonymous credentials in GCP base hook [airflow]

2024-05-17 Thread via GitHub
shahar1 opened a new pull request, #39695: URL: https://github.com/apache/airflow/pull/39695 closes: #35755 --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] fix: delete pod when base container completes [airflow]

2024-05-17 Thread via GitHub
jedcunningham commented on PR #39694: URL: https://github.com/apache/airflow/pull/39694#issuecomment-2118342696 Yeah, it's tough to 1) do it generally 2) not delete pods and 3) not leave the pods running. That's why istio has (or tried to have it turns out) special handling. I'm

Re: [PR] fix: delete pod when base container completes [airflow]

2024-05-17 Thread via GitHub
jedcunningham commented on PR #39694: URL: https://github.com/apache/airflow/pull/39694#issuecomment-2118321909 I did a little digging, and that scenario was never covered when the feature was added in #33306 . I don't think we want to leave these pods still running. -- This is

Re: [PR] fix: delete pod when base container completes [airflow]

2024-05-17 Thread via GitHub
jedcunningham commented on PR #39694: URL: https://github.com/apache/airflow/pull/39694#issuecomment-2118312136 Did you test `on_finish_action="keep_pod"`? We don't always delete the pods. I doubt we can do away with our istio handling like this, and I suspect that got broken somehow

Re: [PR] feat: implement amazon s3 to dynamodb transfer operator [airflow]

2024-05-17 Thread via GitHub
dondaum commented on code in PR #39654: URL: https://github.com/apache/airflow/pull/39654#discussion_r1605483077 ## airflow/providers/amazon/aws/transfers/s3_to_dynamodb.py: ## @@ -0,0 +1,260 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] feat: implement amazon s3 to dynamodb transfer operator [airflow]

2024-05-17 Thread via GitHub
vincbeck commented on code in PR #39654: URL: https://github.com/apache/airflow/pull/39654#discussion_r1605455677 ## airflow/providers/amazon/aws/transfers/s3_to_dynamodb.py: ## @@ -0,0 +1,260 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] feat: implement amazon s3 to dynamodb transfer operator [airflow]

2024-05-17 Thread via GitHub
dondaum commented on code in PR #39654: URL: https://github.com/apache/airflow/pull/39654#discussion_r1605451998 ## airflow/providers/amazon/aws/transfers/s3_to_dynamodb.py: ## @@ -0,0 +1,261 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] fix: delete pod when base container completes [airflow]

2024-05-17 Thread via GitHub
boring-cyborg[bot] commented on PR #39694: URL: https://github.com/apache/airflow/pull/39694#issuecomment-2118226821 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

[PR] fix: delete pod when base container completes [airflow]

2024-05-17 Thread via GitHub
jonathan-ostrander opened a new pull request, #39694: URL: https://github.com/apache/airflow/pull/39694 Closes #39693 Special logic was added to `KubernetesPodOperator`'s lifecycle to handle the case where an istio proxy sidecar is running and preventing the pod from completing, but

Re: [I] KubernetesPodOperator with multiple containers hangs if container other than base container is still running [airflow]

2024-05-17 Thread via GitHub
Taragolis commented on issue #39693: URL: https://github.com/apache/airflow/issues/39693#issuecomment-2118108799 Feel free to fix this behaiviour -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] DbAPiHook: Don't log a warning message if placeholder is None and make sure warning message is correct if invalid [airflow]

2024-05-17 Thread via GitHub
Taragolis commented on code in PR #39690: URL: https://github.com/apache/airflow/pull/39690#discussion_r1605365262 ## tests/providers/common/sql/hooks/test_dbapi.py: ## @@ -421,6 +421,56 @@ def test_get_uri_without_auth_and_empty_host(self): ) assert

Re: [I] Difference between logs which stores in files and logs in UI [airflow]

2024-05-17 Thread via GitHub
Zoynels commented on issue #39686: URL: https://github.com/apache/airflow/issues/39686#issuecomment-2118031758 As I understood, the main problem is in log.splitlines(), which split log-string by simple lines and not by log-messages. Then function analyzes line by line and deduplicates

(airflow) branch main updated: Fix the argument type of input_vectors in pinecone upsert (#39688)

2024-05-17 Thread utkarsharma
This is an automated email from the ASF dual-hosted git repository. utkarsharma pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 9ff245591e Fix the argument type of

Re: [PR] Fix the argument type of input_vectors in pinecone upsert [airflow]

2024-05-17 Thread via GitHub
utkarsharma2 merged PR #39688: URL: https://github.com/apache/airflow/pull/39688 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] KubernetesPodOperator with multiple containers hangs if container other than base container is still running [airflow]

2024-05-17 Thread via GitHub
boring-cyborg[bot] commented on issue #39693: URL: https://github.com/apache/airflow/issues/39693#issuecomment-2118005449 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

[I] KubernetesPodOperator with multiple containers hangs if container other than base container is still running [airflow]

2024-05-17 Thread via GitHub
jonathan-ostrander opened a new issue, #39693: URL: https://github.com/apache/airflow/issues/39693 ### Apache Airflow version main (development) ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? A `KubernetesPodOperator`

Re: [PR] Add metrics about task CPU and memory usage [airflow]

2024-05-17 Thread via GitHub
vincbeck commented on code in PR #39650: URL: https://github.com/apache/airflow/pull/39650#discussion_r1605316385 ## airflow/task/task_runner/standard_task_runner.py: ## @@ -186,3 +193,12 @@ def get_process_pid(self) -> int: if self.process is None: raise

Re: [PR] Update providers metadata 2024-05-17 [airflow]

2024-05-17 Thread via GitHub
eladkal commented on PR #39691: URL: https://github.com/apache/airflow/pull/39691#issuecomment-2117977840 Not sure why it didn't pick the new apache-iceberg provider -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Add rpc server to helm chart [airflow]

2024-05-17 Thread via GitHub
dstandish commented on code in PR #38549: URL: https://github.com/apache/airflow/pull/38549#discussion_r1605285048 ## helm_tests/webserver/test_rpc_server.py: ## @@ -0,0 +1,923 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] DbAPiHook: Don't log a warning message if placeholder is None and make sure warning message is correct if invalid [airflow]

2024-05-17 Thread via GitHub
dabla commented on code in PR #39690: URL: https://github.com/apache/airflow/pull/39690#discussion_r1605280393 ## tests/providers/common/sql/hooks/test_dbapi.py: ## @@ -421,6 +421,56 @@ def test_get_uri_without_auth_and_empty_host(self): ) assert

Re: [PR] Fix the argument type of input_vectors in pinecone upsert [airflow]

2024-05-17 Thread via GitHub
sunank200 commented on code in PR #39688: URL: https://github.com/apache/airflow/pull/39688#discussion_r1605271881 ## airflow/providers/pinecone/hooks/pinecone.py: ## @@ -24,7 +24,7 @@ from functools import cached_property from typing import TYPE_CHECKING, Any -from

Re: [PR] Fix the argument type of input_vectors in pinecone upsert [airflow]

2024-05-17 Thread via GitHub
sunank200 commented on code in PR #39688: URL: https://github.com/apache/airflow/pull/39688#discussion_r1605271233 ## airflow/providers/pinecone/operators/pinecone.py: ## @@ -52,7 +52,7 @@ def __init__( *, conn_id: str = PineconeHook.default_conn_name,

Re: [PR] DbAPiHook: Don't log a warning message if placeholder is None and make sure warning message is correct if invalid [airflow]

2024-05-17 Thread via GitHub
dabla commented on code in PR #39690: URL: https://github.com/apache/airflow/pull/39690#discussion_r1605260210 ## tests/providers/common/sql/hooks/test_dbapi.py: ## @@ -421,6 +421,56 @@ def test_get_uri_without_auth_and_empty_host(self): ) assert

Re: [PR] Fix the argument type of input_vectors in pinecone upsert [airflow]

2024-05-17 Thread via GitHub
utkarsharma2 commented on code in PR #39688: URL: https://github.com/apache/airflow/pull/39688#discussion_r1605261683 ## airflow/providers/pinecone/operators/pinecone.py: ## @@ -21,7 +21,7 @@ from typing import TYPE_CHECKING, Any, Sequence from airflow.models import

Re: [I] Params Fail to Evaluate In Dag Body [airflow]

2024-05-17 Thread via GitHub
josh-fell closed issue #39651: Params Fail to Evaluate In Dag Body URL: https://github.com/apache/airflow/issues/39651 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] DbAPiHook: Don't log a warning message if placeholder is None and make sure warning message is correct if invalid [airflow]

2024-05-17 Thread via GitHub
dabla commented on code in PR #39690: URL: https://github.com/apache/airflow/pull/39690#discussion_r1605260210 ## tests/providers/common/sql/hooks/test_dbapi.py: ## @@ -421,6 +421,56 @@ def test_get_uri_without_auth_and_empty_host(self): ) assert

Re: [PR] Fix the argument type of input_vectors in pinecone upsert [airflow]

2024-05-17 Thread via GitHub
utkarsharma2 commented on code in PR #39688: URL: https://github.com/apache/airflow/pull/39688#discussion_r1605261158 ## airflow/providers/pinecone/hooks/pinecone.py: ## @@ -24,7 +24,7 @@ from functools import cached_property from typing import TYPE_CHECKING, Any -from

Re: [PR] DbAPiHook: Don't log a warning message if placeholder is None and make sure warning message is correct if invalid [airflow]

2024-05-17 Thread via GitHub
dabla commented on code in PR #39690: URL: https://github.com/apache/airflow/pull/39690#discussion_r1605260210 ## tests/providers/common/sql/hooks/test_dbapi.py: ## @@ -421,6 +421,56 @@ def test_get_uri_without_auth_and_empty_host(self): ) assert

Re: [I] Status of testing Providers that were prepared on May 12, 2024 [airflow]

2024-05-17 Thread via GitHub
eladkal commented on issue #39578: URL: https://github.com/apache/airflow/issues/39578#issuecomment-2117934716 Thank you everyone. Providers are released. pinecone provider is excluded and will have rc3 I invite everyone to help improve providers for the next release, a list of

Re: [I] Status of testing Providers that were prepared on May 12, 2024 [airflow]

2024-05-17 Thread via GitHub
eladkal closed issue #39578: Status of testing Providers that were prepared on May 12, 2024 URL: https://github.com/apache/airflow/issues/39578 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Fix the argument type of input_vectors in pinecone upsert [airflow]

2024-05-17 Thread via GitHub
utkarsharma2 commented on code in PR #39688: URL: https://github.com/apache/airflow/pull/39688#discussion_r1605252501 ## airflow/providers/pinecone/operators/pinecone.py: ## @@ -52,7 +52,7 @@ def __init__( *, conn_id: str = PineconeHook.default_conn_name,

[PR] Update providers metadata 2024-05-17 [airflow]

2024-05-17 Thread via GitHub
eladkal opened a new pull request, #39691: URL: https://github.com/apache/airflow/pull/39691 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

(airflow-site) branch add-documentation-2024-05-12 deleted (was a12dcc8818)

2024-05-17 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch add-documentation-2024-05-12 in repository https://gitbox.apache.org/repos/asf/airflow-site.git was a12dcc8818 exclude pinecone The revisions that were on this branch are still contained

Re: [PR] Add documentation for packages - 2024-05-12 [airflow-site]

2024-05-17 Thread via GitHub
eladkal merged PR #1014: URL: https://github.com/apache/airflow-site/pull/1014 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

svn commit: r69257 - /dev/airflow/providers/ /release/airflow/providers/

2024-05-17 Thread eladkal
Author: eladkal Date: Fri May 17 16:04:31 2024 New Revision: 69257 Log: Release Airflow Providers on 2024-05-17 Added: release/airflow/providers/apache_airflow_providers_amazon-8.22.0-py3-none-any.whl - copied unchanged from r69256,

Re: [I] let BigQueryGetData operator take a list of fields for the "order by" clause [airflow]

2024-05-17 Thread via GitHub
eladkal commented on issue #39127: URL: https://github.com/apache/airflow/issues/39127#issuecomment-2117907515 Fixed in https://github.com/apache/airflow/pull/39315 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] let BigQueryGetData operator take a list of fields for the "order by" clause [airflow]

2024-05-17 Thread via GitHub
eladkal closed issue #39127: let BigQueryGetData operator take a list of fields for the "order by" clause URL: https://github.com/apache/airflow/issues/39127 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] DbAPiHook: Don't log a warning message if placeholder is None and make sure warning message is correct if invalid [airflow]

2024-05-17 Thread via GitHub
Taragolis commented on code in PR #39690: URL: https://github.com/apache/airflow/pull/39690#discussion_r1605218668 ## tests/providers/common/sql/hooks/test_dbapi.py: ## @@ -421,6 +421,56 @@ def test_get_uri_without_auth_and_empty_host(self): ) assert

Re: [I] GenericTransfer - Warning: "Placeholder defined in Connection..." [airflow]

2024-05-17 Thread via GitHub
dabla commented on issue #39678: URL: https://github.com/apache/airflow/issues/39678#issuecomment-2117866993 I've created a [PR](https://github.com/apache/airflow/pull/39690) for this which fixes the issue. -- This is an automated message from the Apache Git Service. To respond to the

[PR] DbAPiHook: Don't log a warning message if placeholder is None and make sure warning message is correct if invalid [airflow]

2024-05-17 Thread via GitHub
dabla opened a new pull request, #39690: URL: https://github.com/apache/airflow/pull/39690 This PR fixes an invalid warning message being logged when placeholder is None (and thus not defined in the connection extra field) and also makes sure that when the placeholder isn't

Re: [PR] Make _run_raw_task AIP-44 compatible [airflow]

2024-05-17 Thread via GitHub
dstandish commented on code in PR #38992: URL: https://github.com/apache/airflow/pull/38992#discussion_r1605193809 ## airflow/models/taskinstance.py: ## @@ -2709,43 +3004,12 @@ def _execute_task(self, context: Context, task_orig: Operator): return _execute_task(self,

Re: [PR] Make _run_raw_task AIP-44 compatible [airflow]

2024-05-17 Thread via GitHub
dstandish commented on code in PR #38992: URL: https://github.com/apache/airflow/pull/38992#discussion_r1605193809 ## airflow/models/taskinstance.py: ## @@ -2709,43 +3004,12 @@ def _execute_task(self, context: Context, task_orig: Operator): return _execute_task(self,

Re: [PR] feat: implement amazon s3 to dynamodb transfer operator [airflow]

2024-05-17 Thread via GitHub
vincbeck commented on code in PR #39654: URL: https://github.com/apache/airflow/pull/39654#discussion_r1605201021 ## airflow/providers/amazon/aws/transfers/s3_to_dynamodb.py: ## @@ -0,0 +1,261 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] Make _run_raw_task AIP-44 compatible [airflow]

2024-05-17 Thread via GitHub
dstandish commented on code in PR #38992: URL: https://github.com/apache/airflow/pull/38992#discussion_r1605204033 ## airflow/models/taskinstance.py: ## @@ -1265,6 +1558,130 @@ def _update_rtif(ti, rendered_fields, session: Session | None = None):

Re: [PR] feat: implement amazon s3 to dynamodb transfer operator [airflow]

2024-05-17 Thread via GitHub
vincbeck commented on code in PR #39654: URL: https://github.com/apache/airflow/pull/39654#discussion_r1605199584 ## airflow/providers/amazon/aws/transfers/s3_to_dynamodb.py: ## @@ -0,0 +1,261 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] Make _run_raw_task AIP-44 compatible [airflow]

2024-05-17 Thread via GitHub
dstandish commented on code in PR #38992: URL: https://github.com/apache/airflow/pull/38992#discussion_r1605195033 ## airflow/datasets/manager.py: ## @@ -71,13 +76,14 @@ def register_dataset_change( For local datasets, look them up, record the dataset event, queue

Re: [PR] Make _run_raw_task AIP-44 compatible [airflow]

2024-05-17 Thread via GitHub
dstandish commented on code in PR #38992: URL: https://github.com/apache/airflow/pull/38992#discussion_r1605193809 ## airflow/models/taskinstance.py: ## @@ -2709,43 +3004,12 @@ def _execute_task(self, context: Context, task_orig: Operator): return _execute_task(self,

Re: [PR] feat: implement amazon s3 to dynamodb transfer operator [airflow]

2024-05-17 Thread via GitHub
dondaum commented on code in PR #39654: URL: https://github.com/apache/airflow/pull/39654#discussion_r1605193195 ## airflow/providers/amazon/aws/transfers/s3_to_dynamodb.py: ## @@ -0,0 +1,261 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

[PR] Fix the argument type of input_vectors in pinecone upsert [airflow]

2024-05-17 Thread via GitHub
sunank200 opened a new pull request, #39688: URL: https://github.com/apache/airflow/pull/39688 The upstream `pinecone-client>=3.0.0` accepts vector as `Union[List[Vector], List[tuple], List[dict]]` as per

Re: [PR] Add instruction: where to put image pull secret for private registry [airflow]

2024-05-17 Thread via GitHub
boring-cyborg[bot] commented on PR #39687: URL: https://github.com/apache/airflow/pull/39687#issuecomment-2117824225 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

Re: [I] GenericTransfer - Warning: "Placeholder defined in Connection..." [airflow]

2024-05-17 Thread via GitHub
dabla commented on issue #39678: URL: https://github.com/apache/airflow/issues/39678#issuecomment-2117807835 I'm on it, will write a test to reproduce but I suppose a None check is missing hence why the warning message. -- This is an automated message from the Apache Git Service. To

Re: [I] KPO - get_logs - logs missing [airflow]

2024-05-17 Thread via GitHub
Taragolis commented on issue #39682: URL: https://github.com/apache/airflow/issues/39682#issuecomment-2117764089 What happen if you change your script to? ```python import time if __name__ == '__main__': for i in range(10): time.sleep(2)

Re: [PR] Add metrics about task CPU and memory usage [airflow]

2024-05-17 Thread via GitHub
vincbeck commented on code in PR #39650: URL: https://github.com/apache/airflow/pull/39650#discussion_r1605099377 ## airflow/task/task_runner/standard_task_runner.py: ## @@ -186,3 +193,12 @@ def get_process_pid(self) -> int: if self.process is None: raise

Re: [I] GenericTransfer - Warning: "Placeholder defined in Connection..." [airflow]

2024-05-17 Thread via GitHub
Taragolis commented on issue #39678: URL: https://github.com/apache/airflow/issues/39678#issuecomment-2117729384 I see two problem here: 1. An error in a warning message, instead of connection id use placeholder 2. The warning raised in case if placeholder not defined in connection

Re: [PR] Add metrics about task CPU and memory usage [airflow]

2024-05-17 Thread via GitHub
vincbeck commented on code in PR #39650: URL: https://github.com/apache/airflow/pull/39650#discussion_r1605099377 ## airflow/task/task_runner/standard_task_runner.py: ## @@ -186,3 +193,12 @@ def get_process_pid(self) -> int: if self.process is None: raise

Re: [I] GenericTransfer - Warning: "Placeholder defined in Connection..." [airflow]

2024-05-17 Thread via GitHub
Taragolis commented on issue #39678: URL: https://github.com/apache/airflow/issues/39678#issuecomment-2117722075 cc: @dabla https://github.com/apache/airflow/pull/38528/files#r1551219388 -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] Difference between logs which stores in files and logs in UI [airflow]

2024-05-17 Thread via GitHub
boring-cyborg[bot] commented on issue #39686: URL: https://github.com/apache/airflow/issues/39686#issuecomment-2117715320 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

[I] Difference between logs which stores in files and logs in UI [airflow]

2024-05-17 Thread via GitHub
Zoynels opened a new issue, #39686: URL: https://github.com/apache/airflow/issues/39686 ### Apache Airflow version 2.9.1 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? We print to log some information and saw that

Re: [PR] Add metrics about task CPU and memory usage [airflow]

2024-05-17 Thread via GitHub
Taragolis commented on code in PR #39650: URL: https://github.com/apache/airflow/pull/39650#discussion_r1605080305 ## airflow/task/task_runner/standard_task_runner.py: ## @@ -186,3 +193,12 @@ def get_process_pid(self) -> int: if self.process is None: raise

Re: [PR] Add metrics about task CPU and memory usage [airflow]

2024-05-17 Thread via GitHub
vincbeck commented on code in PR #39650: URL: https://github.com/apache/airflow/pull/39650#discussion_r1605078994 ## docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst: ## @@ -242,6 +242,8 @@ Name

Re: [PR] feat: implement amazon s3 to dynamodb transfer operator [airflow]

2024-05-17 Thread via GitHub
vincbeck commented on code in PR #39654: URL: https://github.com/apache/airflow/pull/39654#discussion_r1605076612 ## airflow/providers/amazon/aws/transfers/s3_to_dynamodb.py: ## @@ -0,0 +1,256 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] feat: implement amazon s3 to dynamodb transfer operator [airflow]

2024-05-17 Thread via GitHub
vincbeck commented on code in PR #39654: URL: https://github.com/apache/airflow/pull/39654#discussion_r1605073951 ## airflow/providers/amazon/aws/transfers/s3_to_dynamodb.py: ## @@ -0,0 +1,261 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

Re: [PR] Add metrics about task CPU and memory usage [airflow]

2024-05-17 Thread via GitHub
Taragolis commented on code in PR #39650: URL: https://github.com/apache/airflow/pull/39650#discussion_r1605075721 ## docs/apache-airflow/administration-and-deployment/logging-monitoring/metrics.rst: ## @@ -242,6 +242,8 @@ Name

Re: [PR] [#38284] Update Confirmation Logic for Config Changes on Sensitive Environments Like Production [airflow]

2024-05-17 Thread via GitHub
maycuatroi commented on PR #38299: URL: https://github.com/apache/airflow/pull/38299#issuecomment-2117698205 > May I suggest that this be added to `dags.html` as well? `dags.html` is the Dags list screen. I believe this change is necessary, and I'll create a PR to update it. --

Re: [PR] Format DAG run count [airflow]

2024-05-17 Thread via GitHub
raphaelauv commented on PR #39684: URL: https://github.com/apache/airflow/pull/39684#issuecomment-2117693566 then I think we need to add an overlay to show the real value -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Make _run_raw_task AIP-44 compatible [airflow]

2024-05-17 Thread via GitHub
vincbeck commented on code in PR #38992: URL: https://github.com/apache/airflow/pull/38992#discussion_r1605048275 ## airflow/datasets/manager.py: ## @@ -71,13 +76,14 @@ def register_dataset_change( For local datasets, look them up, record the dataset event, queue

Re: [PR] Add metrics about task CPU and memory usage [airflow]

2024-05-17 Thread via GitHub
vincbeck commented on code in PR #39650: URL: https://github.com/apache/airflow/pull/39650#discussion_r1605041519 ## airflow/task/task_runner/standard_task_runner.py: ## @@ -186,3 +193,20 @@ def get_process_pid(self) -> int: if self.process is None: raise

[I] EksPodOperator error with deferrable=True [airflow]

2024-05-17 Thread via GitHub
-date;x-amz-security-token;x-amz-target ed0a66facc5d11f21f1673d559ae47610eccc44488ef4bce5816471fe2712851 [2024-05-17, 10:03:47 UTC] {{auth.py:428}} DEBUG - StringToSign: AWS4-HMAC-SHA256 20240517T100347Z 20240517/eu-west-1/logs/aws4_request dbd39dc01e3f6b1a289cc0cc9c53a767cbbe72eecb64

Re: [I] EksPodOperator error with deferrable=True [airflow]

2024-05-17 Thread via GitHub
boring-cyborg[bot] commented on issue #39685: URL: https://github.com/apache/airflow/issues/39685#issuecomment-2117645713 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

Re: [PR] Add metrics about task CPU and memory usage [airflow]

2024-05-17 Thread via GitHub
vincbeck commented on code in PR #39650: URL: https://github.com/apache/airflow/pull/39650#discussion_r1605032756 ## airflow/task/task_runner/standard_task_runner.py: ## @@ -186,3 +193,20 @@ def get_process_pid(self) -> int: if self.process is None: raise

Re: [PR] Format DAG run count [airflow]

2024-05-17 Thread via GitHub
boring-cyborg[bot] commented on PR #39684: URL: https://github.com/apache/airflow/pull/39684#issuecomment-2117635900 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

[PR] Format DAG run count [airflow]

2024-05-17 Thread via GitHub
rafaelleinio opened a new pull request, #39684: URL: https://github.com/apache/airflow/pull/39684 DAGs with many runs are badly formatted inside the circle. Current:

Re: [I] Resolve `AirflowProviderDeprecationWarning` in providers system tests [airflow]

2024-05-17 Thread via GitHub
dondaum commented on issue #39485: URL: https://github.com/apache/airflow/issues/39485#issuecomment-2117622894 I can take on the task or at least support it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] ERROR: could not queue task TaskInstanceKey [airflow]

2024-05-17 Thread via GitHub
Karpisek commented on issue #20047: URL: https://github.com/apache/airflow/issues/20047#issuecomment-2117614006 @harshith-bolar-rapido hi, have you been able to resolve the issue? (we are facing it too on AF version 2.5.3) -- This is an automated message from the Apache Git Service. To

(airflow) branch main updated: Update plugins.rst examples to use pyproject.toml over setup.py (#39665)

2024-05-17 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 8b19b78ba5 Update plugins.rst examples to use

Re: [PR] Update plugins.rst examples to use pyproject.toml over setup.py [airflow]

2024-05-17 Thread via GitHub
Taragolis merged PR #39665: URL: https://github.com/apache/airflow/pull/39665 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Dynamically Mapped Tasks: DB performance issues [airflow]

2024-05-17 Thread via GitHub
Taragolis commented on issue #39680: URL: https://github.com/apache/airflow/issues/39680#issuecomment-2117588770 > yeah, would be great to avoid DB internal error Feel free to fix it. Apache Airflow it is open source project, and every one could contribute changes (fixes/features)

Re: [I] Dynamically Mapped Tasks: DB performance issues [airflow]

2024-05-17 Thread via GitHub
VladimirYushkevich commented on issue #39680: URL: https://github.com/apache/airflow/issues/39680#issuecomment-2117568358 yeah, > As measure might be use `skip_locked` instead of `no_wait` both of them non blocking, with differences how it handled internally in DB backend: >

Re: [PR] Prevent start trigger initialization in scheduler [airflow]

2024-05-17 Thread via GitHub
Lee-W commented on code in PR #39585: URL: https://github.com/apache/airflow/pull/39585#discussion_r1604969594 ## docs/apache-airflow/authoring-and-scheduling/deferring.rst: ## @@ -143,10 +143,12 @@ The ``self.defer`` call raises the ``TaskDeferred`` exception, so it can work

Re: [PR] Starts execution directly from triggerer without going to worker [airflow]

2024-05-17 Thread via GitHub
Lee-W commented on code in PR #38674: URL: https://github.com/apache/airflow/pull/38674#discussion_r1604969145 ## docs/apache-airflow/authoring-and-scheduling/deferring.rst: ## @@ -141,6 +141,60 @@ The ``self.defer`` call raises the ``TaskDeferred`` exception, so it can work

Re: [I] Dynamically Mapped Tasks: DB performance issues [airflow]

2024-05-17 Thread via GitHub
Taragolis commented on issue #39680: URL: https://github.com/apache/airflow/issues/39680#issuecomment-2117554228 As measure might be use `skip_locked` instead of `no_wait` both of them non blocking, with differences how it handled internally in DB backend: - SKIP LOCKED: return non

Re: [I] Dynamically Mapped Tasks: DB performance issues [airflow]

2024-05-17 Thread via GitHub
Taragolis commented on issue #39680: URL: https://github.com/apache/airflow/issues/39680#issuecomment-2117549402 This one use in mini scheduler mechanism, which might fail and it is fine and by design, this just an optimisation mechanism. It might also tried to work concurrently, so the

Re: [PR] UI changes for DAG Reparsing feature [airflow]

2024-05-17 Thread via GitHub
utkarsharma2 commented on code in PR #39636: URL: https://github.com/apache/airflow/pull/39636#discussion_r1604943555 ## airflow/www/static/js/main.js: ## @@ -287,3 +287,18 @@ $(document).ready(() => { // Global Tooltip selector $(".js-tooltip").tooltip(); }); +

Re: [PR] UI changes for DAG Reparsing feature [airflow]

2024-05-17 Thread via GitHub
utkarsharma2 commented on code in PR #39636: URL: https://github.com/apache/airflow/pull/39636#discussion_r1604941192 ## airflow/www/templates/airflow/dags.html: ## @@ -398,6 +398,10 @@ {{ page_title }} class="btn btn-sm btn-default btn-icon-only{{ '

Re: [I] Dynamically Mapped Tasks: DB performance issues [airflow]

2024-05-17 Thread via GitHub
VladimirYushkevich commented on issue #39680: URL: https://github.com/apache/airflow/issues/39680#issuecomment-2117518597 I'm not sure. But then should be kind of retry mechanism(I'm also not sure if we have it). Anyway I just see a lot of errors in DB and my first guess: something is not

  1   2   >