Re: [PR] Add method to get metadata from GCS blob in GCSHook [airflow]

2024-03-22 Thread via GitHub
jalengg commented on code in PR #38398: URL: https://github.com/apache/airflow/pull/38398#discussion_r1536565254 ## airflow/providers/google/cloud/hooks/gcs.py: ## @@ -1006,6 +1006,27 @@ def get_md5hash(self, bucket_name: str, object_name: str) -> str:

Re: [I] SparkKubernetesOperator not retrieves logs from the driver pod and displays them in the Airflow UI. [airflow]

2024-03-22 Thread via GitHub
sudohainguyen commented on issue #37681: URL: https://github.com/apache/airflow/issues/37681#issuecomment-2016329205 hey can we expect this will be fixed in the next release? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] Add method to get metadata from GCS blob in GCSHook [airflow]

2024-03-22 Thread via GitHub
dirrao commented on code in PR #38398: URL: https://github.com/apache/airflow/pull/38398#discussion_r1536551567 ## airflow/providers/google/cloud/hooks/gcs.py: ## @@ -1006,6 +1006,27 @@ def get_md5hash(self, bucket_name: str, object_name: str) -> str:

Re: [PR] fix: BigQueryCheckOperator skip value and error check in deferrable mode [airflow]

2024-03-22 Thread via GitHub
dirrao commented on code in PR #38408: URL: https://github.com/apache/airflow/pull/38408#discussion_r1536548308 ## airflow/providers/google/cloud/operators/bigquery.py: ## @@ -322,8 +322,26 @@ def execute(self, context: Context): ),

(airflow) branch modify-docker-tests updated (ff0fc7568a -> a36fb6eb82)

2024-03-22 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch modify-docker-tests in repository https://gitbox.apache.org/repos/asf/airflow.git from ff0fc7568a Respect docker-tests requirements (use venv) add a36fb6eb82 Create temporary

(airflow) branch modify-docker-tests updated (fdbb7e3d37 -> ff0fc7568a)

2024-03-22 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch modify-docker-tests in repository https://gitbox.apache.org/repos/asf/airflow.git from fdbb7e3d37 Use `python-on-whales` in docker tests add dec65d5400 Use image fixture for

Re: [PR] Improve logging readability with DockerOperator [airflow]

2024-03-22 Thread via GitHub
lyuyunbo commented on PR #38008: URL: https://github.com/apache/airflow/pull/38008#issuecomment-2016229746 > ``` > change() > try: >do stuff > finally: >restore() > ``` Ha! I misunderstood which try-catch block you were mentioning before. Updated, this is

Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

2024-03-22 Thread via GitHub
kaxil commented on issue #34013: URL: https://github.com/apache/airflow/issues/34013#issuecomment-2016216787 >Can everyone in this thread make an experiment, disable it and report back here - at least this will give some clue on where the problems might happen. Yes please,

Re: [PR] add "return" statement to "yield" within a while loop in core triggers [airflow]

2024-03-22 Thread via GitHub
Lee-W commented on PR #38389: URL: https://github.com/apache/airflow/pull/38389#issuecomment-2016215062 > related to that #38353 ? I think it's somewhat related but not 100% sure this is the root cause -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Support configuring multiple namespace for multiNamespaceMode [airflow]

2024-03-22 Thread via GitHub
github-actions[bot] commented on PR #35639: URL: https://github.com/apache/airflow/pull/35639#issuecomment-2016205670 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

Re: [PR] Add type hints for AWS provider [airflow]

2024-03-22 Thread via GitHub
github-actions[bot] commented on PR #35966: URL: https://github.com/apache/airflow/pull/35966#issuecomment-2016205635 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

Re: [PR] [WIP] Mapping tasks from ObjectStoragePath [airflow]

2024-03-22 Thread via GitHub
github-actions[bot] commented on PR #36319: URL: https://github.com/apache/airflow/pull/36319#issuecomment-2016205587 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

[PR] Use `python-on-whales` in docker tests [airflow]

2024-03-22 Thread via GitHub
Taragolis opened a new pull request, #38421: URL: https://github.com/apache/airflow/pull/38421 This attempt to reduce number of subprocess usage and switch to [`python-on-whales`](https://github.com/gabrieldemarmiesse/python-on-whales) where it possible. The main

(airflow) 01/01: Use `python-on-whales` in docker tests

2024-03-22 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch modify-docker-tests in repository https://gitbox.apache.org/repos/asf/airflow.git commit fdbb7e3d37fad146a4dc16f007dde58d0fbeae69 Author: Andrey Anshin AuthorDate: Sat Mar 23 03:40:48 2024

(airflow) branch modify-docker-tests updated (393af8e7bf -> fdbb7e3d37)

2024-03-22 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch modify-docker-tests in repository https://gitbox.apache.org/repos/asf/airflow.git discard 393af8e7bf Use `python-on-whales` in docker tests new fdbb7e3d37 Use `python-on-whales` in

(airflow) 01/01: Use `python-on-whales` in docker tests

2024-03-22 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch modify-docker-tests in repository https://gitbox.apache.org/repos/asf/airflow.git commit d9549bbadeee2193037e4a2ae88bf255a315bb09 Author: Andrey Anshin AuthorDate: Sat Mar 23 03:40:48 2024

(airflow) branch modify-docker-tests created (now d9549bbade)

2024-03-22 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch modify-docker-tests in repository https://gitbox.apache.org/repos/asf/airflow.git at d9549bbade Use `python-on-whales` in docker tests This branch includes the following new commits:

(airflow) 01/01: Use `python-on-whales` in docker tests

2024-03-22 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch modify-docker-tests in repository https://gitbox.apache.org/repos/asf/airflow.git commit 393af8e7bfcfe72d4500bb25153fe6228bf7d017 Author: Andrey Anshin AuthorDate: Sat Mar 23 03:40:48 2024

(airflow) branch modify-docker-tests updated (d9549bbade -> 393af8e7bf)

2024-03-22 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch modify-docker-tests in repository https://gitbox.apache.org/repos/asf/airflow.git discard d9549bbade Use `python-on-whales` in docker tests new 393af8e7bf Use `python-on-whales` in

Re: [PR] Improve logging readability with DockerOperator [airflow]

2024-03-22 Thread via GitHub
potiuk commented on PR #38008: URL: https://github.com/apache/airflow/pull/38008#issuecomment-2016154715 ``` change() try: do stuff finally: restore() ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] Mysql change xcom value col type for MySQL backend [airflow]

2024-03-22 Thread via GitHub
kaxil commented on PR #38401: URL: https://github.com/apache/airflow/pull/38401#issuecomment-2016148389 >If it does, and it results in a slow migration, I really wonder if we should not do this and instruct users to set up an object storage xcom backend instead. Thoughts? The

(airflow) branch constraints-main updated: Updating constraints. Github run id:8397236584

2024-03-22 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 3aeee405a2 Updating

Re: [PR] [AIP-49] Airflow OpenTelemetry Provider [airflow]

2024-03-22 Thread via GitHub
hussein-awala commented on PR #37989: URL: https://github.com/apache/airflow/pull/37989#issuecomment-2016109464 The PR was broken with around 100 unrelated commits, I just squashed all related commits into a single commit and rebased `main` as it was impossible to review the PR. Could you

Re: [PR] Improve logging readability with DockerOperator [airflow]

2024-03-22 Thread via GitHub
lyuyunbo commented on PR #38008: URL: https://github.com/apache/airflow/pull/38008#issuecomment-2016091345 > Still try/finally not solved Can you be more specific please @potiuk? Because we indeed added `_restore_log_formatter` into the except clause, which will revert the changes

Re: [PR] Migrate to connexion v3 [airflow]

2024-03-22 Thread via GitHub
Satoshi-Sh commented on PR #37638: URL: https://github.com/apache/airflow/pull/37638#issuecomment-2016090236 I've been working on `tests/api_connexion/test_cors.py`. It would be great if you could give me some inputs on this. You can find the details

Re: [PR] feat: Add DbtCloudRetryJobOperator to retry failed dbt job [airflow]

2024-03-22 Thread via GitHub
andyguwc commented on PR #38001: URL: https://github.com/apache/airflow/pull/38001#issuecomment-2016065152 @josh-fell As I think about adding a flag for retry from failure. I can't wrap my head around the logic for retry in a deferrable operator. Mind share your thoughts on how this could

Re: [PR] Improve logging readability with DockerOperator [airflow]

2024-03-22 Thread via GitHub
potiuk commented on PR #38008: URL: https://github.com/apache/airflow/pull/38008#issuecomment-2016055030 Still try/finally not solved -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[I] Add legend for average task runtime graph [airflow]

2024-03-22 Thread via GitHub
simond opened a new issue, #38420: URL: https://github.com/apache/airflow/issues/38420 ### Description When clicking on a task in the grid view, a newly-created chart that shows the duration of the tasks shows two reference lines. It is not clear what these reference lines mean and

Re: [I] Add declarative configuration to define custom roles [airflow]

2024-03-22 Thread via GitHub
potiuk commented on issue #38347: URL: https://github.com/apache/airflow/issues/38347#issuecomment-2016047870 > What's currently being done for connections and variables to solve that issue? They both already have declarative equivalents. No - they don't - same problem. When you

Re: [I] Add declarative configuration to define custom roles [airflow]

2024-03-22 Thread via GitHub
potiuk commented on issue #38347: URL: https://github.com/apache/airflow/issues/38347#issuecomment-2016046849 > How have they failed? To solve all the edge cases and the fact that users can modify the DB after it has been created and how to consistently mantain source of the truth.t

Re: [PR] refactor: Refactored __new__ magic method of BaseOperatorMeta to avoid bad mixing classic and decorated operators [airflow]

2024-03-22 Thread via GitHub
potiuk commented on PR #37937: URL: https://github.com/apache/airflow/pull/37937#issuecomment-2016043868 @ashb ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

(airflow) branch main updated (a7d7ef6433 -> 88ea87d3d0)

2024-03-22 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from a7d7ef6433 Don't allow defaults other than None in context parameters, and improve error message (#38015) add

Re: [PR] Update UV to latest version released [airflow]

2024-03-22 Thread via GitHub
potiuk merged PR #38419: URL: https://github.com/apache/airflow/pull/38419 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Improve logging readability with DockerOperator [airflow]

2024-03-22 Thread via GitHub
lyuyunbo commented on PR #38008: URL: https://github.com/apache/airflow/pull/38008#issuecomment-2015985430 > conflicts to resolve when rebasing. Resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Merge kubernetes pod objects inplace [airflow]

2024-03-22 Thread via GitHub
potiuk commented on PR #38259: URL: https://github.com/apache/airflow/pull/38259#issuecomment-2015984516 > what do you mean by breaking IMHO it's just broken, not breaking - because it's impossible to foresee how one pod executing will impact another one, and since you have no influence on

Re: [PR] Improve logging readability with DockerOperator [airflow]

2024-03-22 Thread via GitHub
potiuk commented on PR #38008: URL: https://github.com/apache/airflow/pull/38008#issuecomment-2015980402 conflicts to resolve when rebasing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Don't allow defaults other than None in context parameters, and improve error message [airflow]

2024-03-22 Thread via GitHub
boring-cyborg[bot] commented on PR #38015: URL: https://github.com/apache/airflow/pull/38015#issuecomment-2015977471 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional

(airflow) branch main updated: Don't allow defaults other than None in context parameters, and improve error message (#38015)

2024-03-22 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new a7d7ef6433 Don't allow defaults other than None in

Re: [I] Context variable defaults break tasks where these parameters don't go at the end of the task fuction definition [airflow]

2024-03-22 Thread via GitHub
potiuk closed issue #38006: Context variable defaults break tasks where these parameters don't go at the end of the task fuction definition URL: https://github.com/apache/airflow/issues/38006 -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Don't allow defaults other than None in context parameters, and improve error message [airflow]

2024-03-22 Thread via GitHub
potiuk merged PR #38015: URL: https://github.com/apache/airflow/pull/38015 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated: support iam token from metadata, simplify code (#38411)

2024-03-22 Thread husseinawala
This is an automated email from the ASF dual-hosted git repository. husseinawala pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 30817a5c6d support iam token from metadata,

Re: [PR] [yandexcloud/yq] support iam token from metadata, simplify code [airflow]

2024-03-22 Thread via GitHub
hussein-awala merged PR #38411: URL: https://github.com/apache/airflow/pull/38411 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Delete deprecated AutoML operators and hook [airflow]

2024-03-22 Thread via GitHub
shahar1 commented on code in PR #38418: URL: https://github.com/apache/airflow/pull/38418#discussion_r1536239715 ## airflow/providers/google/cloud/hooks/automl.py: ## @@ -1,642 +0,0 @@ -# Review Comment: Technically speaking - we didn't actually deprecate this hook, but as

Re: [PR] Delete deprecated AutoML operators and hook [airflow]

2024-03-22 Thread via GitHub
shahar1 commented on code in PR #38418: URL: https://github.com/apache/airflow/pull/38418#discussion_r1536239715 ## airflow/providers/google/cloud/hooks/automl.py: ## @@ -1,642 +0,0 @@ -# Review Comment: Technically speaking - we didn't actually deprecate this hook, but as

[PR] Update UV to latest version released [airflow]

2024-03-22 Thread via GitHub
potiuk opened a new pull request, #38419: URL: https://github.com/apache/airflow/pull/38419 --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] Delete deprecated AutoML operators and hook [airflow]

2024-03-22 Thread via GitHub
shahar1 commented on code in PR #38418: URL: https://github.com/apache/airflow/pull/38418#discussion_r1536239715 ## airflow/providers/google/cloud/hooks/automl.py: ## @@ -1,642 +0,0 @@ -# Review Comment: Technically speaking - we didn't actually deprecate this hook, but as

Re: [PR] Revert ObjectStorage config variables name [airflow]

2024-03-22 Thread via GitHub
potiuk merged PR #38415: URL: https://github.com/apache/airflow/pull/38415 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated: Revert ObjectStorage config variables name (#38415)

2024-03-22 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new fd5fe8d2c6 Revert ObjectStorage config variables

[PR] Delete deprecated AutoML operators and hook [airflow]

2024-03-22 Thread via GitHub
shahar1 opened a new pull request, #38418: URL: https://github.com/apache/airflow/pull/38418 Starting March 31, 2024, GCP will sunset the old AutoML API, thus making the deprecated operators and hook effectively unusable. This PR removes all related logic (operators, hook

(airflow) branch main updated (9b2d8e0fcd -> bcd7d35ff8)

2024-03-22 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 9b2d8e0fcd Upgrade to latest chart dependencies detected by canary run (#38416) add bcd7d35ff8 Update Airflow

Re: [PR] Update Airflow decorators stub to reflect changes in #38387 [airflow]

2024-03-22 Thread via GitHub
potiuk merged PR #38406: URL: https://github.com/apache/airflow/pull/38406 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] Fix `parent_model` parameter in GCP Vertex AI AutoML and Custom Job operators [airflow]

2024-03-22 Thread via GitHub
shahar1 opened a new pull request, #38417: URL: https://github.com/apache/airflow/pull/38417 closes: #37400 The following line wrongly parsed the `parent_model` param in all Vertex AI AutoML "Create" operators: ```python self.parent_model =

Re: [I] Enable kubernetes_pod_operator to reattach_on_restart when the worker dies [airflow]

2024-03-22 Thread via GitHub
wircho commented on issue #21900: URL: https://github.com/apache/airflow/issues/21900#issuecomment-2015828679 @yeachan153 Did you ever solve this problem? We would love to be able to keep pods running during environment restarts, and it looks like your idea might work. -- This is an

Re: [PR] New generaic SQL transfer operator proposal [airflow]

2024-03-22 Thread via GitHub
potiuk commented on PR #38281: URL: https://github.com/apache/airflow/pull/38281#issuecomment-2015813508 Probably yes. How to perform that? > Just the usual "Refactor -> Extract" - both are in common.sql provider, so adding `common.sql.util` package with `provider_compatibility.py`

(airflow) branch main updated: Upgrade to latest chart dependencies detected by canary run (#38416)

2024-03-22 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 9b2d8e0fcd Upgrade to latest chart dependencies

Re: [PR] Upgrade to latest chart dependencies detected by canary run [airflow]

2024-03-22 Thread via GitHub
potiuk merged PR #38416: URL: https://github.com/apache/airflow/pull/38416 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Mysql change xcom value col type for MySQL backend [airflow]

2024-03-22 Thread via GitHub
potiuk commented on PR #38401: URL: https://github.com/apache/airflow/pull/38401#issuecomment-2015810027 > > I think, if that's the case then we will likely have to at least warn the users in significant note that it might happen and that they should likely run db clean on xcom table for

(airflow) branch main updated: Add experimental warning in AWS auth manager documentation (#38414)

2024-03-22 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 0d11f3ca2f Add experimental warning in AWS auth

Re: [PR] Add experimental warning in AWS auth manager documentation [airflow]

2024-03-22 Thread via GitHub
Taragolis merged PR #38414: URL: https://github.com/apache/airflow/pull/38414 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] New generaic SQL transfer operator proposal [airflow]

2024-03-22 Thread via GitHub
frodo2000 commented on PR #38281: URL: https://github.com/apache/airflow/pull/38281#issuecomment-2015774537 > How about extracting the common code for version check? Sounds reasonable to do it now? Probably yes. How to perform that? -- This is an automated message from the Apache

(airflow) branch main updated: Fix release docs for the helm chart (#38413)

2024-03-22 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 36704fb9a7 Fix release docs for the helm chart

Re: [PR] Fix release docs for the helm chart [airflow]

2024-03-22 Thread via GitHub
potiuk merged PR #38413: URL: https://github.com/apache/airflow/pull/38413 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] Upgrade to latest chart dependencies detected by canary run [airflow]

2024-03-22 Thread via GitHub
potiuk opened a new pull request, #38416: URL: https://github.com/apache/airflow/pull/38416 --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] Fix XComObjectStoreBackend config var in docs and bug when xcom_objectstore_compression is empty [airflow]

2024-03-22 Thread via GitHub
pankajastro commented on PR #38142: URL: https://github.com/apache/airflow/pull/38142#issuecomment-2015737146 @bolkedebruin create a PR https://github.com/apache/airflow/pull/38415 PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[PR] Revert ObjectStorage config variables name [airflow]

2024-03-22 Thread via GitHub
pankajastro opened a new pull request, #38415: URL: https://github.com/apache/airflow/pull/38415 Address https://github.com/apache/airflow/pull/38142#issuecomment-2007166415 --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] Migrate to connexion v3 [airflow]

2024-03-22 Thread via GitHub
potiuk commented on PR #37638: URL: https://github.com/apache/airflow/pull/37638#issuecomment-2015695912 > ### Does not raise Valueerror When I run it in my unit test, I got Internal Server error - that's why it does not return ValueError, because it converts ValueError into 500

Re: [PR] Prevent large objects from being stored in the RTIF [airflow]

2024-03-22 Thread via GitHub
ephraimbuddy commented on PR #38094: URL: https://github.com/apache/airflow/pull/38094#issuecomment-2015691919 > > Nice. The prefix is cool. > > There is one worry with that U just wanted to mention it. > > I understand that we are masking secreates before serialization ? Just wanted

Re: [PR] Prevent large objects from being stored in the RTIF [airflow]

2024-03-22 Thread via GitHub
ephraimbuddy commented on PR #38094: URL: https://github.com/apache/airflow/pull/38094#issuecomment-2015670623 > Nice. The prefix is cool. > > There is one worry with that U just wanted to mention it. > > I understand that we are masking secreates before serialization ? Just

Re: [I] Add declarative configuration to define custom roles [airflow]

2024-03-22 Thread via GitHub
ketozhang commented on issue #38347: URL: https://github.com/apache/airflow/issues/38347#issuecomment-2015651861 Thanks and yep, it's a classic problem most devops folks have to deal with. It's heavily dependent if you want to lock down configs post-deploy and if DB should be overwritten

Re: [I] Task is not able to be run - k8s Celery workers with stale resources [airflow]

2024-03-22 Thread via GitHub
potiuk commented on issue #34013: URL: https://github.com/apache/airflow/issues/34013#issuecomment-2015639112 No. I do not see anyone working on it nor providing a solution. But the log above suggests that this is caused by so called "mini scheduler" which is enabled by this configuration

Re: [PR] to avoid type's mismatch durind concatenation modified to f-strings [airflow]

2024-03-22 Thread via GitHub
potiuk merged PR #38412: URL: https://github.com/apache/airflow/pull/38412 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Make postgresql default engine args comply with SA 2.0 [airflow]

2024-03-22 Thread via GitHub
potiuk merged PR #38362: URL: https://github.com/apache/airflow/pull/38362 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Migrate to connexion v3 [airflow]

2024-03-22 Thread via GitHub
sudiptob2 commented on PR #37638: URL: https://github.com/apache/airflow/pull/37638#issuecomment-2015632855 > Since is it API experimental (which is deprecated), I would not spend too much time on it. Maybe change the expected exception? Change it to expect `ValueError` from `connexion`

(airflow) branch main updated: to avoid type's mismatch durind concatenation modified to f-strings (#38412)

2024-03-22 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 6a225ccb2a to avoid type's mismatch durind

Re: [PR] to avoid type's mismatch durind concatenation modified to f-strings [airflow]

2024-03-22 Thread via GitHub
boring-cyborg[bot] commented on PR #38412: URL: https://github.com/apache/airflow/pull/38412#issuecomment-2015621212 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional

(airflow) branch main updated: Make postgresql default engine args comply with SA 2.0 (#38362)

2024-03-22 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 5344881715 Make postgresql default engine args

Re: [I] Airflow postgres executemany_mode': 'values' error [airflow]

2024-03-22 Thread via GitHub
potiuk closed issue #38064: Airflow postgres executemany_mode': 'values' error URL: https://github.com/apache/airflow/issues/38064 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Fix release docs for the helm chart [airflow]

2024-03-22 Thread via GitHub
potiuk commented on code in PR #38413: URL: https://github.com/apache/airflow/pull/38413#discussion_r1535975928 ## dev/chart/build_changelog_annotations.py: ## @@ -98,7 +98,7 @@ def print_entry(section: str, description: str, pr_number: int | None):

Re: [PR] Fix release docs for the helm chart [airflow]

2024-03-22 Thread via GitHub
enchant3dmango commented on code in PR #38413: URL: https://github.com/apache/airflow/pull/38413#discussion_r1535967153 ## dev/chart/build_changelog_annotations.py: ## @@ -98,7 +98,7 @@ def print_entry(section: str, description: str, pr_number: int | None):

Re: [I] Status of testing of Apache Airflow Helm Chart 1.13.1rc1 [airflow]

2024-03-22 Thread via GitHub
brokenjacobs commented on issue #38382: URL: https://github.com/apache/airflow/issues/38382#issuecomment-2015577073 LGTM!, keda bypass of pgbouncer is working as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

(airflow) branch main updated (947c48b2fb -> 40559a315e)

2024-03-22 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 947c48b2fb DockerOperator: use DOCKER_HOST as default for docker_url (#38387) add 40559a315e Fix typo in

Re: [PR] Fix typo in verification process for Helm Chart [airflow]

2024-03-22 Thread via GitHub
potiuk merged PR #38410: URL: https://github.com/apache/airflow/pull/38410 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Add declarative configuration to define custom roles [airflow]

2024-03-22 Thread via GitHub
potiuk commented on issue #38347: URL: https://github.com/apache/airflow/issues/38347#issuecomment-2015572014 Sure. If you have a good idea how to do it in a robust way - proposing a solution, especially one that will not overwrite existing DB and will allow to both declaratively upload

Re: [PR] Migrate to connexion v3 [airflow]

2024-03-22 Thread via GitHub
vincbeck commented on PR #37638: URL: https://github.com/apache/airflow/pull/37638#issuecomment-2015557108 > ### Does not raise Valueerror > This test seems not to raise `ValueError` anymore after migrating to connexion v3. See **screenshot 1**. But a ValueError is indeed raised from

Re: [I] Status of testing of Apache Airflow Helm Chart 1.13.1rc1 [airflow]

2024-03-22 Thread via GitHub
brokenjacobs commented on issue #38382: URL: https://github.com/apache/airflow/issues/38382#issuecomment-2015552750 quay.io outage causing issues with the statsd container: ![image](https://github.com/apache/airflow/assets/3159463/562b91ea-2813-435e-80fb-396f489c00f2) ``` Warning

[PR] Add experimental warning on AWS auth manager documentation [airflow]

2024-03-22 Thread via GitHub
vincbeck opened a new pull request, #38414: URL: https://github.com/apache/airflow/pull/38414 --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] Fix pre-mature evaluation of tasks in mapped task group [airflow]

2024-03-22 Thread via GitHub
ephraimbuddy commented on PR #36462: URL: https://github.com/apache/airflow/pull/36462#issuecomment-2015546025 Can you add this test: ``` def test_mapped_tasks_in_mapped_task_group_waits_for_upstreams_to_complete(dag_maker, session): """Test that one failed trigger rule works

[PR] Fix release docs for the helm chart [airflow]

2024-03-22 Thread via GitHub
jedcunningham opened a new pull request, #38413: URL: https://github.com/apache/airflow/pull/38413 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] New generaic SQL transfer operator proposal [airflow]

2024-03-22 Thread via GitHub
potiuk commented on PR #38281: URL: https://github.com/apache/airflow/pull/38281#issuecomment-2015543690 How about extracting the common code for version check? Sounds reasonable to do it now? -- This is an automated message from the Apache Git Service. To respond to the message, please

[PR] to avoid type's mismatch durind concatenation modified to f-strings [airflow]

2024-03-22 Thread via GitHub
igeni opened a new pull request, #38412: URL: https://github.com/apache/airflow/pull/38412 --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] to avoid type's mismatch durind concatenation modified to f-strings [airflow]

2024-03-22 Thread via GitHub
boring-cyborg[bot] commented on PR #38412: URL: https://github.com/apache/airflow/pull/38412#issuecomment-2015543056 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

Re: [PR] New generaic SQL transfer operator proposal [airflow]

2024-03-22 Thread via GitHub
potiuk commented on code in PR #38281: URL: https://github.com/apache/airflow/pull/38281#discussion_r1535923249 ## airflow/providers/common/sql/transfers/sql.py: ## @@ -0,0 +1,248 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Check task attribute before use in sentry.add_tagging() [airflow]

2024-03-22 Thread via GitHub
ephraimbuddy commented on code in PR #37143: URL: https://github.com/apache/airflow/pull/37143#discussion_r1535920258 ## airflow/sentry/__init__.py: ## @@ -0,0 +1,30 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[PR] [yandexcloud/yq] support iam token from metadata, simplify code [airflow]

2024-03-22 Thread via GitHub
uzhastik opened a new pull request, #38411: URL: https://github.com/apache/airflow/pull/38411 Add ability to use IAM token from metadata service in YQ operator --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] Prevent large objects from being stored in the RTIF [airflow]

2024-03-22 Thread via GitHub
jedcunningham commented on code in PR #38094: URL: https://github.com/apache/airflow/pull/38094#discussion_r1535918174 ## airflow/serialization/helpers.py: ## @@ -38,6 +39,13 @@ def is_jsonable(x): else: return True +max_length = conf.getint("core",

Re: [PR] New generaic SQL transfer operator proposal [airflow]

2024-03-22 Thread via GitHub
frodo2000 commented on code in PR #38281: URL: https://github.com/apache/airflow/pull/38281#discussion_r1535918264 ## airflow/providers/common/sql/transfers/sql.py: ## @@ -0,0 +1,248 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[PR] Fix typo in verification process for Helm Chart [airflow]

2024-03-22 Thread via GitHub
potiuk opened a new pull request, #38410: URL: https://github.com/apache/airflow/pull/38410 --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] Check task attribute before use in sentry.add_tagging() [airflow]

2024-03-22 Thread via GitHub
LipuFei commented on PR #37143: URL: https://github.com/apache/airflow/pull/37143#issuecomment-2015525725 @potiuk I will split the commits so they are easier to review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Migrate to connexion v3 [airflow]

2024-03-22 Thread via GitHub
sudiptob2 commented on PR #37638: URL: https://github.com/apache/airflow/pull/37638#issuecomment-2015521624 ### Does not raise Valueerror This test seems not to raise `ValueError` anymore after migrating to connexion v3. See **screenshot 1**. But a ValueError is indeed raised from deep

Re: [PR] feat: Add DbtCloudRetryJobOperator to retry failed dbt job [airflow]

2024-03-22 Thread via GitHub
andyguwc commented on PR #38001: URL: https://github.com/apache/airflow/pull/38001#issuecomment-2015519361 > Why not create a flag on the DbtCloudRunJobOperator to retry from failure rather than create a new operator that reuses a lot of the same code? @josh-fell thanks. I was

  1   2   3   >