Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
potiuk commented on PR #39336: URL: https://github.com/apache/airflow/pull/39336#issuecomment-2101991720 Maybe just a bit more manual test scrutiny for 2.9.2 around try_num will be `enough` of stability ? I think the sooner we get it out, the easier will be to fix any potential problems

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
dstandish commented on PR #39336: URL: https://github.com/apache/airflow/pull/39336#issuecomment-2101987113 I feel like it is not unreasonable to think of it as bug fix. But I think the point of that convention is that patch releases should get more stable not less. So since this has some

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
potiuk commented on PR #39336: URL: https://github.com/apache/airflow/pull/39336#issuecomment-2101983614 > A few nits, otherwise it looks good, and I don't know why we (I) did it this terrible hacky way in the first place! I think we will find out when it is released :). But I also

Re: [PR] Adding note for provider dependencies [airflow]

2024-05-08 Thread via GitHub
potiuk commented on PR #39512: URL: https://github.com/apache/airflow/pull/39512#issuecomment-2101980429 Nice! thanks @amoghrajesh ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] Adding note for provider dependencies [airflow]

2024-05-08 Thread via GitHub
amoghrajesh opened a new pull request, #39512: URL: https://github.com/apache/airflow/pull/39512 Based on the slack discussion https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1715038775776789 there was no explicit note for dependencies in providers, adding it.

Re: [I] Deferrable operator tasks do not call `on_kill` method when fail or restarted [airflow]

2024-05-08 Thread via GitHub
thesuperzapper commented on issue #36090: URL: https://github.com/apache/airflow/issues/36090#issuecomment-2101966602 @sunank200 also we will need to update all the other providers that set an `on_kill` and can be deferred (unless you are already doing this). This search might help

Re: [I] Add a new ExternalAPITaskSensor to monitor external DAGs via Airflow REST API [airflow]

2024-05-08 Thread via GitHub
cherrera20 commented on issue #39463: URL: https://github.com/apache/airflow/issues/39463#issuecomment-2101960623 OK thanks. Just to illustrate how the sensor would look like I leave you a code snippet of the hypothetical call to the sensor. If it makes sense to you we can move on :-)

Re: [PR] Fetch served logs also when task attempt is up for retry and no remote logs available [airflow]

2024-05-08 Thread via GitHub
kahlstrm commented on code in PR #39496: URL: https://github.com/apache/airflow/pull/39496#discussion_r1594971540 ## tests/utils/test_log_handlers.py: ## @@ -317,43 +317,52 @@ def test__read_for_celery_executor_fallbacks_to_worker(self, create_task_instanc """Test for

Re: [I] Deferrable operator tasks do not call `on_kill` method when fail or restarted [airflow]

2024-05-08 Thread via GitHub
thesuperzapper commented on issue #36090: URL: https://github.com/apache/airflow/issues/36090#issuecomment-2101920452 @sunank200 also note that using the `self.task_instance` of the BaseTrigger will require __at least Airflow 2.6.0__, it was added by this PR

Re: [I] Deferrable operator tasks do not call `on_kill` method when fail or restarted [airflow]

2024-05-08 Thread via GitHub
thesuperzapper commented on issue #36090: URL: https://github.com/apache/airflow/issues/36090#issuecomment-2101913601 @sunank200 I see you have removed the check for a RUNNING state, and now only check if it's not in a DEFERRED state, for example:

Re: [PR] Warn user about unused replace parameter in insert_rows (OracleHook) [airflow]

2024-05-08 Thread via GitHub
RichieSK commented on code in PR #39408: URL: https://github.com/apache/airflow/pull/39408#discussion_r1594944686 ## airflow/providers/oracle/hooks/oracle.py: ## @@ -272,8 +272,14 @@ def insert_rows( :param commit_every: the maximum number of rows to insert in one

(airflow) branch main updated: Scheduler to handle incrementing of try_number (#39336)

2024-05-08 Thread dstandish
This is an automated email from the ASF dual-hosted git repository. dstandish pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 3938f71dfa Scheduler to handle incrementing of

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
dstandish merged PR #39336: URL: https://github.com/apache/airflow/pull/39336 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
dstandish commented on PR #39336: URL: https://github.com/apache/airflow/pull/39336#issuecomment-2101842090 alright, yolo, here we go thanks @ashb @hussein-awala @jedcunningham -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[PR] Fix side effects in DBT Cloud tests [airflow]

2024-05-08 Thread via GitHub
Taragolis opened a new pull request, #39511: URL: https://github.com/apache/airflow/pull/39511 Fixup flakey test `tests/providers/dbt/cloud/operators/test_dbt.py::TestDbtCloudRunJobOperator::test_execute_wait_for_termination` which depends on a system time. As a bonus test

(airflow) branch main updated: Implement slicing in lazy sequence (#39483)

2024-05-08 Thread uranusjr
This is an automated email from the ASF dual-hosted git repository. uranusjr pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new f36f9f4dc1 Implement slicing in lazy sequence

Re: [PR] Implement slicing in lazy sequence [airflow]

2024-05-08 Thread via GitHub
uranusjr merged PR #39483: URL: https://github.com/apache/airflow/pull/39483 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated: Update docker-compose command (#39504)

2024-05-08 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 3401ab1f57 Update docker-compose command

Re: [PR] Update docker-compose command to docker compose in tutorial [airflow]

2024-05-08 Thread via GitHub
boring-cyborg[bot] commented on PR #39504: URL: https://github.com/apache/airflow/pull/39504#issuecomment-2101822880 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional

Re: [PR] Update docker-compose command to docker compose in tutorial [airflow]

2024-05-08 Thread via GitHub
Taragolis merged PR #39504: URL: https://github.com/apache/airflow/pull/39504 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Add Scarf based telemetry [airflow]

2024-05-08 Thread via GitHub
kaxil commented on code in PR #39510: URL: https://github.com/apache/airflow/pull/39510#discussion_r1594883962 ## airflow/utils/scarf.py: ## @@ -0,0 +1,56 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the

[PR] Add Scarf based telemetry [airflow]

2024-05-08 Thread via GitHub
kaxil opened a new pull request, #39510: URL: https://github.com/apache/airflow/pull/39510 To take [the discussion about adding Telemetry](https://lists.apache.org/thread/7f6qyr8w2n8w34g63s7ybhzphgt8h43m) forward, I am creating a draft PR that adds some basic telemetry to send to Scarf. I

Re: [PR] enh(amazon_hook): in file download, change type of raised exception due to 404 [airflow]

2024-05-08 Thread via GitHub
boring-cyborg[bot] commented on PR #39509: URL: https://github.com/apache/airflow/pull/39509#issuecomment-2101769925 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

Re: [PR] Add unit tests for EksCreateClusterTrigger and EksDeleteClusterTrigger [airflow]

2024-05-08 Thread via GitHub
mateuslatrova commented on PR #39349: URL: https://github.com/apache/airflow/pull/39349#issuecomment-2101757016 I made a mistake because I thought this branch was already merged, so I had to reopen the PR. Here it is: https://github.com/apache/airflow/pull/39508 Now, it also has unit

[PR] Add unit tests for EksCreateClusterTrigger and EksDeleteClusterTrigger [airflow]

2024-05-08 Thread via GitHub
mateuslatrova opened a new pull request, #39508: URL: https://github.com/apache/airflow/pull/39508 --- Related: https://github.com/apache/airflow/issues/35442 This PR adds unit tests for EksCreateClusterTrigger and EksDeleteClusterTrigger triggers

Re: [PR] Add unit tests for EksCreateClusterTrigger [airflow]

2024-05-08 Thread via GitHub
mateuslatrova closed pull request #39349: Add unit tests for EksCreateClusterTrigger URL: https://github.com/apache/airflow/pull/39349 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Add a new ExternalAPITaskSensor to monitor external DAGs via Airflow REST API [airflow]

2024-05-08 Thread via GitHub
Taragolis commented on issue #39463: URL: https://github.com/apache/airflow/issues/39463#issuecomment-2101749986 IMHO, technically it could be added as a separate Provider with access through [apache-airflow-client](https://pypi.org/project/apache-airflow-client/), but as every new

Re: [PR] Add MySQL LTS 8.4 [airflow]

2024-05-08 Thread via GitHub
Taragolis commented on code in PR #39488: URL: https://github.com/apache/airflow/pull/39488#discussion_r1594845455 ## dev/breeze/src/airflow_breeze/global_constants.py: ## @@ -110,9 +110,9 @@ # the first LTS release should be in summer/fall 2024. # - Innovations: Shot living

Re: [PR] Improve logging readability with DockerOperator [airflow]

2024-05-08 Thread via GitHub
github-actions[bot] commented on PR #38008: URL: https://github.com/apache/airflow/pull/38008#issuecomment-2101697150 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

Re: [PR] [WIP] Support Flask Session 0.7 [airflow]

2024-05-08 Thread via GitHub
github-actions[bot] closed pull request #38072: [WIP] Support Flask Session 0.7 URL: https://github.com/apache/airflow/pull/38072 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] Resolve deprecations in Other tests [airflow]

2024-05-08 Thread via GitHub
Taragolis opened a new pull request, #39507: URL: https://github.com/apache/airflow/pull/39507 related: https://github.com/apache/airflow/issues/38642 --- **^ Add meaningful description above** Read the **[Pull Request

[PR] Resolve deprecations in always connections tests [airflow]

2024-05-08 Thread via GitHub
Taragolis opened a new pull request, #39506: URL: https://github.com/apache/airflow/pull/39506 related: https://github.com/apache/airflow/issues/38642 --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] Render map_index_template when the task status is set manually [airflow]

2024-05-08 Thread via GitHub
boring-cyborg[bot] commented on PR #39505: URL: https://github.com/apache/airflow/pull/39505#issuecomment-2101681621 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

[PR] Render map_index_template when the task status is set manually [airflow]

2024-05-08 Thread via GitHub
karenbraganz opened a new pull request, #39505: URL: https://github.com/apache/airflow/pull/39505 Render map_index_template when task status is manually set I made render_map_index a class method in task_instance.py and called it in the set_task_instance_state method in

(airflow) branch main updated: Fix generation temp filename in `DataprocSubmitPySparkJobOperator` (#39498)

2024-05-08 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 5abe7acf66 Fix generation temp filename in

Re: [I] Error when _upload_file_temp in DataprocSubmitPySparkJobOperator [airflow]

2024-05-08 Thread via GitHub
Taragolis closed issue #39478: Error when _upload_file_temp in DataprocSubmitPySparkJobOperator URL: https://github.com/apache/airflow/issues/39478 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Fix generation temp filename in `DataprocSubmitPySparkJobOperator` [airflow]

2024-05-08 Thread via GitHub
Taragolis merged PR #39498: URL: https://github.com/apache/airflow/pull/39498 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Update docker-compose command to docker compose [airflow]

2024-05-08 Thread via GitHub
boring-cyborg[bot] commented on PR #39504: URL: https://github.com/apache/airflow/pull/39504#issuecomment-2101665056 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

[PR] Update docker-compose command to docker compose [airflow]

2024-05-08 Thread via GitHub
Zorgosto opened a new pull request, #39504: URL: https://github.com/apache/airflow/pull/39504 Update docker-compose command to docker compose as the previous version is outdated. --- **^ Add meaningful description above** Read the **[Pull Request

(airflow) branch constraints-main updated: Updating constraints. Github run id:9009303509

2024-05-08 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 7168238f16 Updating

Re: [I] SageMakerTransformOperator does not deduplicate model name [airflow]

2024-05-08 Thread via GitHub
mateuslatrova commented on issue #39502: URL: https://github.com/apache/airflow/issues/39502#issuecomment-2101646400 Hi! I would like to work on this issue after triage! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Add MySQL LTS 8.4 [airflow]

2024-05-08 Thread via GitHub
ferruzzi commented on code in PR #39488: URL: https://github.com/apache/airflow/pull/39488#discussion_r1594787607 ## dev/breeze/src/airflow_breeze/global_constants.py: ## @@ -110,9 +110,9 @@ # the first LTS release should be in summer/fall 2024. # - Innovations: Shot living

Re: [PR] Fetch served logs also when task attempt is up for retry and no remote logs available [airflow]

2024-05-08 Thread via GitHub
ferruzzi commented on code in PR #39496: URL: https://github.com/apache/airflow/pull/39496#discussion_r1594784899 ## tests/utils/test_log_handlers.py: ## @@ -317,43 +317,52 @@ def test__read_for_celery_executor_fallbacks_to_worker(self, create_task_instanc """Test for

Re: [PR] Fetch served logs also when task attempt is up for retry and no remote logs available [airflow]

2024-05-08 Thread via GitHub
ferruzzi commented on code in PR #39496: URL: https://github.com/apache/airflow/pull/39496#discussion_r1594784899 ## tests/utils/test_log_handlers.py: ## @@ -317,43 +317,52 @@ def test__read_for_celery_executor_fallbacks_to_worker(self, create_task_instanc """Test for

Re: [PR] Fetch served logs also when task attempt is up for retry and no remote logs available [airflow]

2024-05-08 Thread via GitHub
ferruzzi commented on code in PR #39496: URL: https://github.com/apache/airflow/pull/39496#discussion_r1594784899 ## tests/utils/test_log_handlers.py: ## @@ -317,43 +317,52 @@ def test__read_for_celery_executor_fallbacks_to_worker(self, create_task_instanc """Test for

Re: [PR] Add unit tests for EksCreateClusterTrigger [airflow]

2024-05-08 Thread via GitHub
mateuslatrova commented on PR #39349: URL: https://github.com/apache/airflow/pull/39349#issuecomment-2101611197 > @mateuslatrova can you handle the above comment? Sorry I'm late! Just pushed the asked change. -- This is an automated message from the Apache Git Service. To respond

(airflow) branch main updated: Clarify comments in test_example_dags.py (#39501)

2024-05-08 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 0be44788e4 Clarify comments in

Re: [PR] Clarify comments in test_example_dags.py [airflow]

2024-05-08 Thread via GitHub
Taragolis merged PR #39501: URL: https://github.com/apache/airflow/pull/39501 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated: add doc about Yandex Query operator (#39445)

2024-05-08 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 05945a47f3 add doc about Yandex Query operator

Re: [PR] add doc about Yandex Query operator [airflow]

2024-05-08 Thread via GitHub
Taragolis merged PR #39445: URL: https://github.com/apache/airflow/pull/39445 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[I] Context is not preserved after execution returns from deferred state [airflow]

2024-05-08 Thread via GitHub
mxuan0 opened a new issue, #39503: URL: https://github.com/apache/airflow/issues/39503 ### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.7.2 ### What happened? We are using

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594723182 ## airflow/jobs/backfill_job_runner.py: ## @@ -245,7 +245,16 @@ def _update_counters(self, ti_status: _DagRunTaskStatus, session: Session) -> No

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594723182 ## airflow/jobs/backfill_job_runner.py: ## @@ -245,7 +245,16 @@ def _update_counters(self, ti_status: _DagRunTaskStatus, session: Session) -> No

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594719121 ## airflow/providers/dbt/cloud/utils/openlineage.py: ## @@ -21,13 +21,25 @@ from contextlib import suppress from typing import TYPE_CHECKING +from airflow

Re: [I] SageMakerTransformOperator does not deduplicate model name [airflow]

2024-05-08 Thread via GitHub
boring-cyborg[bot] commented on issue #39502: URL: https://github.com/apache/airflow/issues/39502#issuecomment-2101521119 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

[I] SageMakerTransformOperator does not deduplicate model name [airflow]

2024-05-08 Thread via GitHub
mxuan0 opened a new issue, #39502: URL: https://github.com/apache/airflow/issues/39502 ### Apache Airflow Provider(s) amazon ### Versions of Apache Airflow Providers 8.7.1 ### Apache Airflow version 2.7.2 ### Operating System amazon linux AMI

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594717827 ## kubernetes_tests/test_kubernetes_pod_operator.py: ## @@ -112,7 +112,7 @@ def setup_tests(self, test_label): "run_id":

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594716629 ## airflow/models/dagrun.py: ## @@ -1567,7 +1568,16 @@ def schedule_tis( TI.run_id == self.run_id,

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594715786 ## airflow/jobs/backfill_job_runner.py: ## @@ -245,7 +245,16 @@ def _update_counters(self, ti_status: _DagRunTaskStatus, session: Session) -> No

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594715342 ## airflow/jobs/backfill_job_runner.py: ## @@ -245,7 +245,16 @@ def _update_counters(self, ti_status: _DagRunTaskStatus, session: Session) -> No

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594713811 ## airflow/models/dagrun.py: ## @@ -1567,7 +1568,16 @@ def schedule_tis( TI.run_id == self.run_id,

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594711540 ## airflow/providers/dbt/cloud/utils/openlineage.py: ## @@ -21,13 +21,25 @@ from contextlib import suppress from typing import TYPE_CHECKING +from airflow

[PR] Clarify comments in test_example_dags.py [airflow]

2024-05-08 Thread via GitHub
RNHTTR opened a new pull request, #39501: URL: https://github.com/apache/airflow/pull/39501 --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
hussein-awala commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594675865 ## airflow/jobs/backfill_job_runner.py: ## @@ -245,7 +245,16 @@ def _update_counters(self, ti_status: _DagRunTaskStatus, session: Session) -> No

(airflow) branch main updated (e794d6cce4 -> 73918925ed)

2024-05-08 Thread jedcunningham
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from e794d6cce4 Add pre-commit to correct mismatching revision IDs in migration file (#39480) add 73918925ed

Re: [PR] Simplify `airflow_version` imports [airflow]

2024-05-08 Thread via GitHub
jedcunningham merged PR #39497: URL: https://github.com/apache/airflow/pull/39497 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
ashb commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594684024 ## airflow/providers/dbt/cloud/utils/openlineage.py: ## @@ -21,13 +21,25 @@ from contextlib import suppress from typing import TYPE_CHECKING +from airflow import

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
jscheffl commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594677571 ## airflow/providers/dbt/cloud/utils/openlineage.py: ## @@ -21,13 +21,25 @@ from contextlib import suppress from typing import TYPE_CHECKING +from airflow

[PR] Amaazon Bedrock - Retrieve and RetrieveAndGenerate [airflow]

2024-05-08 Thread via GitHub
ferruzzi opened a new pull request, #39500: URL: https://github.com/apache/airflow/pull/39500 Both of these calls are super fast and neither has any kind of waiter or means of checking the status, so here can not be any sensor or trigger for them. They are simple client calls, but I think

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
ashb commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594663632 ## kubernetes_tests/test_kubernetes_pod_operator.py: ## @@ -112,7 +112,7 @@ def setup_tests(self, test_label): "run_id":

Re: [I] Error rendering template: '_GeneratorContextManager' object has no attribute 'query' [airflow]

2024-05-08 Thread via GitHub
RNHTTR commented on issue #39162: URL: https://github.com/apache/airflow/issues/39162#issuecomment-2101451643 @mro-dmoura is this happening consistently or intermittently? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Scheduler to handle incrementing of try_number [airflow]

2024-05-08 Thread via GitHub
ashb commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594657364 ## airflow/providers/dbt/cloud/utils/openlineage.py: ## @@ -21,13 +21,25 @@ from contextlib import suppress from typing import TYPE_CHECKING +from airflow import

Re: [PR] Fix: Reuse ProcessPoolExecutor in CeleryExecutor [airflow]

2024-05-08 Thread via GitHub
dstandish commented on PR #39484: URL: https://github.com/apache/airflow/pull/39484#issuecomment-2101387857 > I'm testing it through the logs. It takes a while. It usually takes about 2 days to see that the scheduling process slows down significantly. Understood. Did you see my

Re: [PR] Fix: Reuse ProcessPoolExecutor in CeleryExecutor [airflow]

2024-05-08 Thread via GitHub
luoyuliuyin commented on PR #39484: URL: https://github.com/apache/airflow/pull/39484#issuecomment-2101380558 > I am a little skeptical that the overhead for process pool creation is meaningful. Because i think the dominant performance consideration is the blocking IO sending the task to

Re: [PR] Fix: Reuse ProcessPoolExecutor in CeleryExecutor [airflow]

2024-05-08 Thread via GitHub
luoyuliuyin commented on PR #39484: URL: https://github.com/apache/airflow/pull/39484#issuecomment-2101370206 > What happens if you `kill -9` one (or all) of the ProcessPool processes and then try to schedule/enqueue new tasks? I'll test it later -- This is an automated

Re: [PR] Fix: Reuse ProcessPoolExecutor in CeleryExecutor [airflow]

2024-05-08 Thread via GitHub
luoyuliuyin commented on PR #39484: URL: https://github.com/apache/airflow/pull/39484#issuecomment-2101367892 > > The above data is my running results in the Linux environment. If ProcessPoolExecutor is created every time, no problem will be seen at the beginning, but over time the

[PR] Fix generation temp filename in `DataprocSubmitPySparkJobOperator` [airflow]

2024-05-08 Thread via GitHub
Taragolis opened a new pull request, #39498: URL: https://github.com/apache/airflow/pull/39498 closes: #39478 This one fix regression during https://github.com/apache/airflow/pull/33455 --- **^ Add meaningful description above** Read the **[Pull

Re: [PR] Simplify `airflow_version` imports [airflow]

2024-05-08 Thread via GitHub
jedcunningham commented on code in PR #39497: URL: https://github.com/apache/airflow/pull/39497#discussion_r1594570487 ## airflow/providers/alibaba/__init__.py: ## @@ -29,10 +29,7 @@ __version__ = "2.8.0" -try: -from airflow import __version__ as airflow_version

Re: [PR] Fix: Reuse ProcessPoolExecutor in CeleryExecutor [airflow]

2024-05-08 Thread via GitHub
luoyuliuyin commented on PR #39484: URL: https://github.com/apache/airflow/pull/39484#issuecomment-2101323443 > broker I use rabbitmq, AIRFLOW__CELERY__BROKER_URL=amqp://admin:ad...@xxx.xxx:5672 -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Configure HttpHook's auth_type from Connection [airflow]

2024-05-08 Thread via GitHub
dabla commented on PR #35591: URL: https://github.com/apache/airflow/pull/35591#issuecomment-2101312408 I think this one is finally ready -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Error when _upload_file_temp in DataprocSubmitPySparkJobOperator [airflow]

2024-05-08 Thread via GitHub
Taragolis commented on issue #39478: URL: https://github.com/apache/airflow/issues/39478#issuecomment-2101310734 > But even if it is deprecated, why in the version 10.7.0 has been modified and now dont work, if is deprecated, it should not be modified no? That is a good question, in

Re: [PR] Fix: Reuse ProcessPoolExecutor in CeleryExecutor [airflow]

2024-05-08 Thread via GitHub
dstandish commented on PR #39484: URL: https://github.com/apache/airflow/pull/39484#issuecomment-2101300305 I am skeptical that the overhead for process pool creation is meaningful. Because i think the dominant performance consideration is the blocking IO sending the task to redis and

Re: [PR] Simplify `airflow_version` imports [airflow]

2024-05-08 Thread via GitHub
jedcunningham commented on PR #39497: URL: https://github.com/apache/airflow/pull/39497#issuecomment-2101299301 > BTW. I think we can simplify it even further. I think this one should work: > > ``` > packaging.version.parse(airflow_version).base_version <

Re: [PR] Simplify `airflow_version` imports [airflow]

2024-05-08 Thread via GitHub
potiuk commented on code in PR #39497: URL: https://github.com/apache/airflow/pull/39497#discussion_r1594519939 ## airflow/providers/alibaba/__init__.py: ## @@ -29,10 +29,7 @@ __version__ = "2.8.0" -try: -from airflow import __version__ as airflow_version -except

Re: [PR] Simplify `airflow_version` imports [airflow]

2024-05-08 Thread via GitHub
potiuk commented on code in PR #39497: URL: https://github.com/apache/airflow/pull/39497#discussion_r1594517558 ## airflow/providers/alibaba/__init__.py: ## @@ -29,10 +29,7 @@ __version__ = "2.8.0" -try: -from airflow import __version__ as airflow_version -except

Re: [PR] Simplify `airflow_version` imports [airflow]

2024-05-08 Thread via GitHub
potiuk commented on code in PR #39497: URL: https://github.com/apache/airflow/pull/39497#discussion_r1594517558 ## airflow/providers/alibaba/__init__.py: ## @@ -29,10 +29,7 @@ __version__ = "2.8.0" -try: -from airflow import __version__ as airflow_version -except

Re: [PR] Remove try_number shenanigans [airflow]

2024-05-08 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594514841 ## tests/jobs/test_backfill_job.py: ## @@ -1525,7 +1544,7 @@ def test_update_counters(self, dag_maker, session): # match what's in the in-memory

Re: [PR] Remove try_number shenanigans [airflow]

2024-05-08 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594514256 ## tests/jobs/test_backfill_job.py: ## @@ -185,7 +185,7 @@ def test_trigger_controller_dag(self, session): assert task_instances_list -

Re: [PR] Remove try_number shenanigans [airflow]

2024-05-08 Thread via GitHub
jedcunningham commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594459245 ## newsfragments/39336.significant.rst: ## @@ -0,0 +1,5 @@ +``try_number`` is no longer incremented during task execution + +Previously, the try_number was

Re: [PR] Remove try_number shenanigans [airflow]

2024-05-08 Thread via GitHub
jedcunningham commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594459245 ## newsfragments/39336.significant.rst: ## @@ -0,0 +1,5 @@ +``try_number`` is no longer incremented during task execution + +Previously, the try_number was

Re: [PR] Simplify `airflow_version` imports [airflow]

2024-05-08 Thread via GitHub
potiuk commented on PR #39497: URL: https://github.com/apache/airflow/pull/39497#issuecomment-2101248064 BTW. I think we can simplify it even further. I think this one should work: ``` packaging.version.parse(airflow_version).base_version < packaging.version.parse("2.7.0") ```

Re: [PR] Simplify `airflow_version` imports [airflow]

2024-05-08 Thread via GitHub
jedcunningham commented on code in PR #39497: URL: https://github.com/apache/airflow/pull/39497#discussion_r1594509148 ## airflow/providers/airbyte/__init__.py: ## @@ -29,10 +29,7 @@ __version__ = "3.8.0" -try: Review Comment: > BTW. It could be done by changing

Re: [PR] Remove try_number shenanigans [airflow]

2024-05-08 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594508672 ## airflow/www/views.py: ## @@ -5196,7 +5196,7 @@ class TaskInstanceModelView(AirflowModelView): "pool", "queued_by_job_id", ] - +#

Re: [PR] Remove try_number shenanigans [airflow]

2024-05-08 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594508672 ## airflow/www/views.py: ## @@ -5196,7 +5196,7 @@ class TaskInstanceModelView(AirflowModelView): "pool", "queued_by_job_id", ] - +#

Re: [PR] Simplify `airflow_version` imports [airflow]

2024-05-08 Thread via GitHub
jedcunningham commented on code in PR #39497: URL: https://github.com/apache/airflow/pull/39497#discussion_r1594508448 ## airflow/providers/airbyte/__init__.py: ## @@ -29,10 +29,7 @@ __version__ = "3.8.0" -try: Review Comment: I did, it's the very last file :) --

Re: [PR] Simplify `airflow_version` imports [airflow]

2024-05-08 Thread via GitHub
potiuk commented on code in PR #39497: URL: https://github.com/apache/airflow/pull/39497#discussion_r1594508185 ## airflow/providers/airbyte/__init__.py: ## @@ -29,10 +29,7 @@ __version__ = "3.8.0" -try: Review Comment: Ups. I see it's been done :) -- This is an

Re: [PR] Remove try_number shenanigans [airflow]

2024-05-08 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594507755 ## airflow/providers/amazon/aws/executors/ecs/ecs_executor.py: ## @@ -527,7 +527,7 @@ def try_adopt_task_instances(self, tis: Sequence[TaskInstance]) ->

Re: [PR] Remove try_number shenanigans [airflow]

2024-05-08 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594507359 ## airflow/models/taskinstancekey.py: ## @@ -37,6 +37,7 @@ def primary(self) -> tuple[str, str, str, int]: @property def reduced(self) ->

Re: [PR] Simplify `airflow_version` imports [airflow]

2024-05-08 Thread via GitHub
potiuk commented on code in PR #39497: URL: https://github.com/apache/airflow/pull/39497#discussion_r1594507299 ## airflow/providers/airbyte/__init__.py: ## @@ -29,10 +29,7 @@ __version__ = "3.8.0" -try: Review Comment: See comment few lines above: ``` # IF

Re: [PR] Remove try_number shenanigans [airflow]

2024-05-08 Thread via GitHub
dstandish commented on code in PR #39336: URL: https://github.com/apache/airflow/pull/39336#discussion_r1594506487 ## airflow/models/taskinstance.py: ## @@ -1549,53 +1519,22 @@ def init_on_load(self) -> None: """Initialize the attributes that aren't stored in the

  1   2   3   >