Re: [PR] Add the deferrable mode to the Dataflow sensors [airflow]

2024-04-08 Thread via GitHub
Lee-W commented on code in PR #37693: URL: https://github.com/apache/airflow/pull/37693#discussion_r1556763479 ## airflow/providers/google/cloud/sensors/dataflow.py: ## @@ -151,12 +205,14 @@ def __init__( self, *, job_id: str, -callback:

Re: [PR] Add four unit tests for aws/utils [airflow]

2024-04-08 Thread via GitHub
slycyberguy commented on code in PR #38820: URL: https://github.com/apache/airflow/pull/38820#discussion_r1556846390 ## tests/providers/amazon/aws/utils/test_sagemaker.py: ## Review Comment: I removed those tests. I figured they were necessary because those services were

[I] Unable to find Spark Support in helm chart [airflow]

2024-04-08 Thread via GitHub
anandlingaraj opened a new issue, #38851: URL: https://github.com/apache/airflow/issues/38851 ### Apache Airflow version 2.9.0 ### If "Other Airflow 2 version" selected, which one? apache/airflow:2.8.3 ### What happened? When selecting connection type for

Re: [I] Unable to find Spark Support in helm chart [airflow]

2024-04-08 Thread via GitHub
boring-cyborg[bot] commented on issue #38851: URL: https://github.com/apache/airflow/issues/38851#issuecomment-2044123456 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

Re: [PR] Make _get_ti compatible with RPC [airflow]

2024-04-08 Thread via GitHub
dstandish commented on code in PR #38570: URL: https://github.com/apache/airflow/pull/38570#discussion_r1556918483 ## airflow/serialization/serialized_objects.py: ## @@ -1462,9 +1462,15 @@ def deserialize_dag(cls, encoded_dag: dict[str, Any]) -> SerializedDAG:

Re: [PR] Deferrable mode for Custom Training Job operators [airflow]

2024-04-08 Thread via GitHub
Lee-W commented on code in PR #38584: URL: https://github.com/apache/airflow/pull/38584#discussion_r1556745491 ## airflow/providers/google/cloud/operators/vertex_ai/custom_job.py: ## @@ -539,6 +564,94 @@ def on_kill(self) -> None: if self.hook:

Re: [PR] Add the deferrable mode to the Dataflow sensors [airflow]

2024-04-08 Thread via GitHub
Lee-W commented on code in PR #37693: URL: https://github.com/apache/airflow/pull/37693#discussion_r1556794107 ## airflow/providers/google/cloud/sensors/dataflow.py: ## @@ -194,27 +244,72 @@ def poke(self, context: Context) -> bool: project_id=self.project_id,

Re: [PR] Always use the executemany method when inserting rows in DbApiHook as it's way much faster [airflow]

2024-04-08 Thread via GitHub
uranusjr commented on code in PR #38715: URL: https://github.com/apache/airflow/pull/38715#discussion_r1556827871 ## airflow/providers/common/sql/hooks/sql.py: ## @@ -166,6 +175,7 @@ def __init__(self, *args, schema: str | None = None, log_sql: bool = True, **kwa #

Re: [I] White INFO log is hard to read [airflow]

2024-04-08 Thread via GitHub
tirkarthi commented on issue #38844: URL: https://github.com/apache/airflow/issues/38844#issuecomment-2044202208 Does the logs returned from API have ansi code that causes this? cc: @jscheffl -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Make _get_ti compatible with RPC [airflow]

2024-04-08 Thread via GitHub
dstandish commented on code in PR #38570: URL: https://github.com/apache/airflow/pull/38570#discussion_r1556922160 ## airflow/cli/commands/task_command.py: ## @@ -156,8 +156,10 @@ def _get_dag_run( raise ValueError(f"unknown create_if_necessary value:

Re: [PR] Make _get_ti compatible with RPC [airflow]

2024-04-08 Thread via GitHub
dstandish commented on code in PR #38570: URL: https://github.com/apache/airflow/pull/38570#discussion_r1556927388 ## airflow/serialization/pydantic/dag_run.py: ## @@ -78,7 +78,7 @@ def get_task_instances( def get_task_instance( self, task_id: str, -

Re: [I] clear a deferred task do not increment the tries [airflow]

2024-04-08 Thread via GitHub
tirkarthi commented on issue #38735: URL: https://github.com/apache/airflow/issues/38735#issuecomment-2044200109 When a task is deferred the `_try_number` is decremented so that when the trigger yields an event the task resumes execution where `_try_number` is incremented to use the same

Re: [I] timeout on Dynamic Task mapping : skipped inner task , task status is still success [airflow]

2024-04-08 Thread via GitHub
raphaelauv commented on issue #37332: URL: https://github.com/apache/airflow/issues/37332#issuecomment-2044184401 @ephraimbuddy could you please remo6the tag wait for response , thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Fix AttributeError: 'DagRunNote' object has no attribute 'dag_id' [airflow]

2024-04-08 Thread via GitHub
MyLong commented on code in PR #38843: URL: https://github.com/apache/airflow/pull/38843#discussion_r1556746373 ## airflow/models/dagrun.py: ## @@ -1648,7 +1648,5 @@ def __init__(self, content, user_id=None): self.user_id = user_id def __repr__(self): -

Re: [PR] Paginate Airflow task logs [airflow]

2024-04-08 Thread via GitHub
uranusjr commented on code in PR #38807: URL: https://github.com/apache/airflow/pull/38807#discussion_r1556841592 ## airflow/providers/amazon/aws/log/s3_task_handler.py: ## @@ -178,7 +180,13 @@ def s3_read(self, remote_log_location: str, return_error: bool = False) -> str:

Re: [PR] fix: try002 for provider cncf kubernetes [airflow]

2024-04-08 Thread via GitHub
eladkal commented on PR #38799: URL: https://github.com/apache/airflow/pull/38799#issuecomment-2044119678 There are conflics to resolve -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

(airflow-site) branch gh-pages updated (ecb04021c3 -> d51b51096d)

2024-04-08 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch gh-pages in repository https://gitbox.apache.org/repos/asf/airflow-site.git discard ecb04021c3 Rewritten history to remove past gh-pages deployments new d51b51096d Rewritten history to

Re: [PR] Bugfix to correct GCSHook being called even when not required with BeamRunPythonPipelineOperator [airflow]

2024-04-08 Thread via GitHub
e-galan commented on code in PR #38716: URL: https://github.com/apache/airflow/pull/38716#discussion_r1555810282 ## airflow/providers/apache/beam/operators/beam.py: ## @@ -364,11 +364,13 @@ def execute(self, context: Context): def execute_sync(self, context: Context):

Re: [PR] Resolve core unit test internal warning [airflow]

2024-04-08 Thread via GitHub
Taragolis commented on code in PR #38835: URL: https://github.com/apache/airflow/pull/38835#discussion_r1555814562 ## tests/deprecations_ignore.yml: ## @@ -70,9 +70,6 @@ # Core -- tests/core/test_core.py::TestCore::test_dag_params_and_task_params --

(airflow-site) branch gh-pages updated (d51b51096d -> 12ec5916dd)

2024-04-08 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch gh-pages in repository https://gitbox.apache.org/repos/asf/airflow-site.git discard d51b51096d Rewritten history to remove past gh-pages deployments new 12ec5916dd Rewritten history to

Re: [PR] Bugfix to correct GCSHook being called even when not required with BeamRunPythonPipelineOperator [airflow]

2024-04-08 Thread via GitHub
e-galan commented on code in PR #38716: URL: https://github.com/apache/airflow/pull/38716#discussion_r1555830278 ## airflow/providers/apache/beam/operators/beam.py: ## @@ -364,11 +364,13 @@ def execute(self, context: Context): def execute_sync(self, context: Context):

Re: [PR] Resolve core unit test internal warning [airflow]

2024-04-08 Thread via GitHub
Owen-CH-Leung commented on code in PR #38835: URL: https://github.com/apache/airflow/pull/38835#discussion_r1555842384 ## tests/deprecations_ignore.yml: ## @@ -70,9 +70,6 @@ # Core -- tests/core/test_core.py::TestCore::test_dag_params_and_task_params --

Re: [PR] OpenAI Chat & Assistant hook functions [airflow]

2024-04-08 Thread via GitHub
utkarsharma2 commented on code in PR #38736: URL: https://github.com/apache/airflow/pull/38736#discussion_r1555846716 ## airflow/providers/openai/hooks/openai.py: ## @@ -77,6 +89,165 @@ def get_conn(self) -> OpenAI: **openai_client_kwargs, ) +def

Re: [PR] OpenAI Chat & Assistant hook functions [airflow]

2024-04-08 Thread via GitHub
nathadfield commented on code in PR #38736: URL: https://github.com/apache/airflow/pull/38736#discussion_r1555850833 ## airflow/providers/openai/hooks/openai.py: ## @@ -77,6 +89,165 @@ def get_conn(self) -> OpenAI: **openai_client_kwargs, ) +def

Re: [PR] OpenAI Chat & Assistant hook functions [airflow]

2024-04-08 Thread via GitHub
utkarsharma2 commented on code in PR #38736: URL: https://github.com/apache/airflow/pull/38736#discussion_r1555852061 ## airflow/providers/openai/hooks/openai.py: ## @@ -77,6 +89,165 @@ def get_conn(self) -> OpenAI: **openai_client_kwargs, ) +def

Re: [PR] OpenAI Chat & Assistant hook functions [airflow]

2024-04-08 Thread via GitHub
nathadfield commented on code in PR #38736: URL: https://github.com/apache/airflow/pull/38736#discussion_r1555850833 ## airflow/providers/openai/hooks/openai.py: ## @@ -77,6 +89,165 @@ def get_conn(self) -> OpenAI: **openai_client_kwargs, ) +def

Re: [PR] OpenAI Chat & Assistant hook functions [airflow]

2024-04-08 Thread via GitHub
nathadfield commented on code in PR #38736: URL: https://github.com/apache/airflow/pull/38736#discussion_r1555854814 ## airflow/providers/openai/hooks/openai.py: ## @@ -77,6 +89,165 @@ def get_conn(self) -> OpenAI: **openai_client_kwargs, ) +def

Re: [PR] OpenAI Chat & Assistant hook functions [airflow]

2024-04-08 Thread via GitHub
nathadfield commented on code in PR #38736: URL: https://github.com/apache/airflow/pull/38736#discussion_r1555871245 ## airflow/providers/openai/hooks/openai.py: ## @@ -77,6 +89,165 @@ def get_conn(self) -> OpenAI: **openai_client_kwargs, ) +def

Re: [PR] OpenAI Chat & Assistant hook functions [airflow]

2024-04-08 Thread via GitHub
nathadfield commented on code in PR #38736: URL: https://github.com/apache/airflow/pull/38736#discussion_r1555871738 ## airflow/providers/openai/hooks/openai.py: ## @@ -77,6 +89,165 @@ def get_conn(self) -> OpenAI: **openai_client_kwargs, ) +def

Re: [PR] OpenAI Chat & Assistant hook functions [airflow]

2024-04-08 Thread via GitHub
nathadfield commented on code in PR #38736: URL: https://github.com/apache/airflow/pull/38736#discussion_r1555871738 ## airflow/providers/openai/hooks/openai.py: ## @@ -77,6 +89,165 @@ def get_conn(self) -> OpenAI: **openai_client_kwargs, ) +def

[I] DB migrate throws na error on encrypt_trigger_kwargs [airflow]

2024-04-08 Thread via GitHub
romanzdk opened a new issue, #38836: URL: https://github.com/apache/airflow/issues/38836 ### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.9.0 ### What happened? `airflow db

[PR] Airflow 2.9.0 has been released [airflow]

2024-04-08 Thread via GitHub
ephraimbuddy opened a new pull request, #38837: URL: https://github.com/apache/airflow/pull/38837 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Expose AWS IAM missing param in Hashicorp secret [airflow]

2024-04-08 Thread via GitHub
ReadytoRocc commented on code in PR #38536: URL: https://github.com/apache/airflow/pull/38536#discussion_r1555877716 ## airflow/providers/hashicorp/_internal_client/vault_client.py: ## @@ -318,15 +322,23 @@ def _auth_azure(self, _client: hvac.Client) -> None: )

Re: [PR] OpenAI Chat & Assistant hook functions [airflow]

2024-04-08 Thread via GitHub
nathadfield commented on code in PR #38736: URL: https://github.com/apache/airflow/pull/38736#discussion_r1555881195 ## airflow/providers/openai/hooks/openai.py: ## @@ -77,6 +89,165 @@ def get_conn(self) -> OpenAI: **openai_client_kwargs, ) +def

Re: [PR] Add four unit tests for aws/utils [airflow]

2024-04-08 Thread via GitHub
potiuk commented on code in PR #38820: URL: https://github.com/apache/airflow/pull/38820#discussion_r1555893343 ## tests/providers/amazon/aws/utils/test_sagemaker.py: ## Review Comment: Potential Value added vs. time needed to review/classify those tests do not seem to be

Re: [PR] Add four unit tests for aws/utils [airflow]

2024-04-08 Thread via GitHub
potiuk commented on code in PR #38820: URL: https://github.com/apache/airflow/pull/38820#discussion_r1555891937 ## tests/providers/amazon/aws/utils/test_sagemaker.py: ## Review Comment: I had the same impression that most of the tests produced / generated here are

[PR] Chart: Default Airflow version to 2.9.0 [airflow]

2024-04-08 Thread via GitHub
ephraimbuddy opened a new pull request, #38839: URL: https://github.com/apache/airflow/pull/38839 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

(airflow) branch main updated: Remove decorator from rendering fields example (#38827)

2024-04-08 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 285f037dbc Remove decorator from rendering fields

Re: [PR] Remove decorator from rendering fields example [airflow]

2024-04-08 Thread via GitHub
potiuk merged PR #38827: URL: https://github.com/apache/airflow/pull/38827 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated (285f037dbc -> 76477b0977)

2024-04-08 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 285f037dbc Remove decorator from rendering fields example (#38827) add 76477b0977 Update version added

Re: [I] Databricks provider does not support Azure managed identity if more than one identity exists (e.g. ADF, AKS) [airflow]

2024-04-08 Thread via GitHub
jtv8 commented on issue #38762: URL: https://github.com/apache/airflow/issues/38762#issuecomment-2043178028 Hi @ghjklw! I agree that it would be better to leverage the abstractions in the official Microsoft libraries - the current implementation is far too low level. However, that would

Re: [PR] Give `on_task_instance_failed` access to the error that caused the failure [airflow]

2024-04-08 Thread via GitHub
potiuk commented on code in PR #38155: URL: https://github.com/apache/airflow/pull/38155#discussion_r1556148549 ## airflow/models/taskinstance.py: ## @@ -1409,7 +1410,9 @@ class TaskInstance(Base, LoggingMixin): cascade="all, delete, delete-orphan", ) note =

Re: [PR] Setting use_beeline by default for hive cli connection [airflow]

2024-04-08 Thread via GitHub
potiuk commented on code in PR #38763: URL: https://github.com/apache/airflow/pull/38763#discussion_r1556155471 ## airflow/providers/apache/hive/provider.yaml: ## @@ -25,6 +25,7 @@ state: ready source-date-epoch: 1709554960 # note that those versions are maintained by release

Re: [PR] Amazon Bedrock - Model Customization Jobs [airflow]

2024-04-08 Thread via GitHub
ferruzzi commented on code in PR #38693: URL: https://github.com/apache/airflow/pull/38693#discussion_r1556195811 ## airflow/providers/amazon/aws/hooks/bedrock.py: ## @@ -19,6 +19,37 @@ from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook +class

Re: [I] Change the UI color for ExternalSensor tasks [airflow]

2024-04-08 Thread via GitHub
idantepper commented on issue #38845: URL: https://github.com/apache/airflow/issues/38845#issuecomment-2043379278 Hi, I would like to take this issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Deferrable mode for Custom Training Job operators [airflow]

2024-04-08 Thread via GitHub
potiuk commented on code in PR #38584: URL: https://github.com/apache/airflow/pull/38584#discussion_r1556100397 ## airflow/providers/google/cloud/operators/vertex_ai/custom_job.py: ## @@ -539,6 +564,94 @@ def on_kill(self) -> None: if self.hook:

Re: [PR] Update version added field in config.yml [airflow]

2024-04-08 Thread via GitHub
ephraimbuddy merged PR #38840: URL: https://github.com/apache/airflow/pull/38840 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-08 Thread via GitHub
o-nikolas commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1556103085 ## airflow/jobs/job.py: ## @@ -104,12 +106,13 @@ class Job(Base, LoggingMixin): Only makes sense for SchedulerJob and BackfillJob instances. """ -

Re: [PR] Chart: Default Airflow version to 2.9.0 [airflow]

2024-04-08 Thread via GitHub
ephraimbuddy commented on code in PR #38839: URL: https://github.com/apache/airflow/pull/38839#discussion_r1556101539 ## chart/newsfragments/38478.significant.rst: ## Review Comment: Thanks. Updated ## chart/newsfragments/38478.significant.rst: ##

Re: [PR] Make _run_task_by_local_task_job compatible with internal API [airflow]

2024-04-08 Thread via GitHub
dstandish commented on code in PR #38563: URL: https://github.com/apache/airflow/pull/38563#discussion_r1556179515 ## airflow/cli/commands/task_command.py: ## @@ -278,7 +279,15 @@ def _run_task_by_local_task_job(args, ti: TaskInstance | TaskInstancePydantic) -

Re: [PR] Amazon Bedrock - Model Customization Jobs [airflow]

2024-04-08 Thread via GitHub
ferruzzi commented on code in PR #38693: URL: https://github.com/apache/airflow/pull/38693#discussion_r1556199352 ## airflow/providers/amazon/aws/sensors/bedrock.py: ## @@ -0,0 +1,111 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

Re: [PR] Deferrable mode for Custom Training Job operators [airflow]

2024-04-08 Thread via GitHub
potiuk commented on code in PR #38584: URL: https://github.com/apache/airflow/pull/38584#discussion_r1556098638 ## airflow/providers/google/cloud/operators/vertex_ai/custom_job.py: ## @@ -539,6 +564,94 @@ def on_kill(self) -> None: if self.hook:

Re: [I] Databricks provider does not support Azure managed identity if more than one identity exists (e.g. ADF, AKS) [airflow]

2024-04-08 Thread via GitHub
potiuk commented on issue #38762: URL: https://github.com/apache/airflow/issues/38762#issuecomment-2043197124 > Perhaps one of the project maintainers could kindly suggest it if they have open channels of communication with Microsoft about their commercial offerings? Not as far as I

Re: [PR] Setting use_beeline by default for hive cli connection [airflow]

2024-04-08 Thread via GitHub
potiuk commented on PR #38763: URL: https://github.com/apache/airflow/pull/38763#issuecomment-2043244944 Approved with NIT's -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Add lineage_job_namespace and lineage_job_name OpenLineage macros [airflow]

2024-04-08 Thread via GitHub
mobuchowski merged PR #38829: URL: https://github.com/apache/airflow/pull/38829 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated: Add lineage_job_namespace and lineage_job_name OpenLineage macros (#38829)

2024-04-08 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 093ab7e755 Add lineage_job_namespace and

Re: [PR] Setting use_beeline by default for hive cli connection [airflow]

2024-04-08 Thread via GitHub
eladkal commented on code in PR #38763: URL: https://github.com/apache/airflow/pull/38763#discussion_r1556176719 ## airflow/providers/apache/hive/CHANGELOG.rst: ## @@ -27,6 +27,19 @@ Changelog - +8.0.0 +. + + +Breaking changes + + +Changed the

Re: [PR] Amazon Bedrock - Model Customization Jobs [airflow]

2024-04-08 Thread via GitHub
ferruzzi commented on code in PR #38693: URL: https://github.com/apache/airflow/pull/38693#discussion_r1556206876 ## airflow/providers/amazon/aws/operators/bedrock.py: ## @@ -91,3 +98,155 @@ def execute(self, context: Context) -> dict[str, str | int]:

(airflow) branch main updated: Chart: Default Airflow version to 2.9.0 (#38839)

2024-04-08 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 1d70bbf683 Chart: Default Airflow version

Re: [PR] Chart: Default Airflow version to 2.9.0 [airflow]

2024-04-08 Thread via GitHub
ephraimbuddy merged PR #38839: URL: https://github.com/apache/airflow/pull/38839 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Replace dill package to use cloudpickle [airflow]

2024-04-08 Thread via GitHub
potiuk commented on code in PR #38531: URL: https://github.com/apache/airflow/pull/38531#discussion_r1556118149 ## airflow/models/taskinstance.py: ## @@ -1287,7 +1287,7 @@ class TaskInstance(Base, LoggingMixin): queued_dttm = Column(UtcDateTime) queued_by_job_id =

Re: [PR] Improve editable airflow installation by adding preinstalled deps [airflow]

2024-04-08 Thread via GitHub
potiuk merged PR #38764: URL: https://github.com/apache/airflow/pull/38764 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated: Improve editable airflow installation by adding preinstalled deps (#38764)

2024-04-08 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new d409b8be96 Improve editable airflow installation

Re: [PR] Setting use_beeline by default for hive cli connection [airflow]

2024-04-08 Thread via GitHub
potiuk commented on code in PR #38763: URL: https://github.com/apache/airflow/pull/38763#discussion_r1556154848 ## airflow/providers/apache/hive/provider.yaml: ## @@ -25,6 +25,7 @@ state: ready source-date-epoch: 1709554960 # note that those versions are maintained by release

Re: [PR] Amazon Bedrock - Model Customization Jobs [airflow]

2024-04-08 Thread via GitHub
ferruzzi commented on code in PR #38693: URL: https://github.com/apache/airflow/pull/38693#discussion_r1556206876 ## airflow/providers/amazon/aws/operators/bedrock.py: ## @@ -91,3 +98,155 @@ def execute(self, context: Context) -> dict[str, str | int]:

Re: [PR] Amazon Bedrock - Model Customization Jobs [airflow]

2024-04-08 Thread via GitHub
ferruzzi commented on code in PR #38693: URL: https://github.com/apache/airflow/pull/38693#discussion_r1556213468 ## airflow/providers/amazon/aws/operators/bedrock.py: ## @@ -91,3 +98,155 @@ def execute(self, context: Context) -> dict[str, str | int]:

Re: [PR] Expose AWS IAM missing param in Hashicorp secret [airflow]

2024-04-08 Thread via GitHub
pankajastro commented on code in PR #38536: URL: https://github.com/apache/airflow/pull/38536#discussion_r1555273271 ## airflow/providers/hashicorp/_internal_client/vault_client.py: ## @@ -318,15 +322,23 @@ def _auth_azure(self, _client: hvac.Client) -> None: )

Re: [PR] Added Feature: search dags by task id with suggestions [airflow]

2024-04-08 Thread via GitHub
pruoff commented on PR #37436: URL: https://github.com/apache/airflow/pull/37436#issuecomment-2041956580 > Thanks for the fix, I can confirm it works now as advertised. > > I approve from my side. I just kindly request to also alter the documentation a bit. Else the risk is high that

Re: [I] `AirbyteTriggerSyncOperator` does not work on deferrable mode [airflow]

2024-04-08 Thread via GitHub
josix commented on issue #37449: URL: https://github.com/apache/airflow/issues/37449#issuecomment-2041992100 I did some investigation, the root cause of this issue is that the url format is invalid for `aiohttp` so it would raise the following error. Note that I used the local airbyte to

Re: [PR] Always use the fast_executemany method when inserting rows in DbApiHook as it's way much faster [airflow]

2024-04-08 Thread via GitHub
dabla commented on PR #38715: URL: https://github.com/apache/airflow/pull/38715#issuecomment-2041993404 > Why not just always use `executemany` instead? It’s unclear to me why a new argument is needed. To ellaborate a bit, that's indeed what this pull request is doing, always use

Re: [PR] Fix 'use case' spelling [airflow]

2024-04-08 Thread via GitHub
potiuk merged PR #38817: URL: https://github.com/apache/airflow/pull/38817 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Make _get_ti compatible with RPC [airflow]

2024-04-08 Thread via GitHub
uranusjr commented on code in PR #38570: URL: https://github.com/apache/airflow/pull/38570#discussion_r1555274634 ## airflow/serialization/serialized_objects.py: ## @@ -1462,9 +1462,15 @@ def deserialize_dag(cls, encoded_dag: dict[str, Any]) -> SerializedDAG:

[I] Spark distributed computing using SagemakerProcessingOperator [airflow]

2024-04-08 Thread via GitHub
dhana-sekhar opened a new issue, #38822: URL: https://github.com/apache/airflow/issues/38822 ### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.4.3 ### What happened? we are

Re: [PR] Always use the fast_executemany method when inserting rows in DbApiHook as it's way much faster [airflow]

2024-04-08 Thread via GitHub
dabla commented on PR #38715: URL: https://github.com/apache/airflow/pull/38715#issuecomment-2041997794 > Would love other reviews, but it looks good. > > NIT: why changing types in tests to string? Will that work for all common.sql users? also NIT2 - we should let users know y

Re: [I] Adding UIAlert in local settings fails with Sentry enabled [airflow]

2024-04-08 Thread via GitHub
tanvn commented on issue #31442: URL: https://github.com/apache/airflow/issues/31442#issuecomment-2042056903 I faced a similar issue today with Airflow 2.8.4 Error log from the scheduler ``` ERROR! Maximum number of retries (20) reached. Last check result: $ airflow db

Re: [PR] fix: try002 for provider microsoft azure [airflow]

2024-04-08 Thread via GitHub
Taragolis merged PR #38805: URL: https://github.com/apache/airflow/pull/38805 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated (830b71b3be -> b687f59008)

2024-04-08 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 830b71b3be fix: try002 for provider amazon (#38789) add b687f59008 fix: try002 for provider microsoft azure

Re: [PR] add Wei Lee to committer list [airflow-site]

2024-04-08 Thread via GitHub
Lee-W commented on code in PR #996: URL: https://github.com/apache/airflow-site/pull/996#discussion_r1555366393 ## landing-pages/site/data/committers.json: ## @@ -161,6 +161,12 @@ "image": "https://github.com/vincbeck.png;, "nick": "vincbeck" }, + { +"name":

Re: [PR] Make _get_ti compatible with RPC [airflow]

2024-04-08 Thread via GitHub
uranusjr commented on code in PR #38570: URL: https://github.com/apache/airflow/pull/38570#discussion_r1555272673 ## airflow/cli/commands/task_command.py: ## @@ -156,8 +156,10 @@ def _get_dag_run( raise ValueError(f"unknown create_if_necessary value:

Re: [PR] [docs] update `DagBag` class docstring to include all params [airflow]

2024-04-08 Thread via GitHub
romsharon98 commented on PR #38814: URL: https://github.com/apache/airflow/pull/38814#issuecomment-2041988661 > > ``` > > > When ``True`` uses heuristics (files containing ``DAG`` and ``airflow`` strings) to filter > > > ``` > > > > @romsharon98, Sorry,

Re: [I] Spark distributed computing using SagemakerProcessingOperator [airflow]

2024-04-08 Thread via GitHub
boring-cyborg[bot] commented on issue #38822: URL: https://github.com/apache/airflow/issues/38822#issuecomment-2041991155 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

Re: [PR] Always use the fast_executemany method when inserting rows in DbApiHook as it's way much faster [airflow]

2024-04-08 Thread via GitHub
dabla commented on PR #38715: URL: https://github.com/apache/airflow/pull/38715#issuecomment-2042025338 > The part I don’t understand is why we’re adding the `_fast_executemany` flag on the hook. Is there a case we want to use `executemany` but does not want to set `fast_executemany`? The

(airflow) branch main updated: Fix 'use case' spelling (#38817)

2024-04-08 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 4cd15f8926 Fix 'use case' spelling (#38817)

Re: [PR] Remove SparkInK8s internal deprecations [airflow]

2024-04-08 Thread via GitHub
Taragolis commented on PR #38777: URL: https://github.com/apache/airflow/pull/38777#issuecomment-2042056628 I guess until Airflow 3 we still need to have SubDag operator into the `airflow/example_dags`, it use in multiple places include into the documentation part. What we could do it is

Re: [PR] fix: try002 for provider amazon [airflow]

2024-04-08 Thread via GitHub
Taragolis merged PR #38789: URL: https://github.com/apache/airflow/pull/38789 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated: fix: try002 for provider amazon (#38789)

2024-04-08 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 830b71b3be fix: try002 for provider amazon

Re: [PR] fix: try002 for provider google [airflow]

2024-04-08 Thread via GitHub
pankajkoti commented on code in PR #38803: URL: https://github.com/apache/airflow/pull/38803#discussion_r1555358313 ## airflow/providers/google/cloud/hooks/dataflow.py: ## @@ -252,7 +252,7 @@ def _get_current_jobs(self) -> list[dict]: self._job_id =

(airflow-site) branch add-weilee-to-committer-list updated (587ae05178 -> 43cd4e89d8)

2024-04-08 Thread weilee
This is an automated email from the ASF dual-hosted git repository. weilee pushed a change to branch add-weilee-to-committer-list in repository https://gitbox.apache.org/repos/asf/airflow-site.git discard 587ae05178 add Wei Lee to committer list add 43cd4e89d8 add Wei Lee to committer

Re: [PR] Implement start/end/debug for multiple executors in scheduler job [airflow]

2024-04-08 Thread via GitHub
uranusjr commented on code in PR #38514: URL: https://github.com/apache/airflow/pull/38514#discussion_r1555276527 ## airflow/jobs/job.py: ## @@ -104,12 +106,13 @@ class Job(Base, LoggingMixin): Only makes sense for SchedulerJob and BackfillJob instances. """ -

Re: [PR] Always use the fast_executemany method when inserting rows in DbApiHook as it's way much faster [airflow]

2024-04-08 Thread via GitHub
uranusjr commented on PR #38715: URL: https://github.com/apache/airflow/pull/38715#issuecomment-2042016703 The part I don’t understand is why we’re adding the `_fast_executemany` flag on the hook. Is there a case we want to use `executemany` but does not want to set `fast_executemany`? The

Re: [I] Automate links to configuration of providers [airflow]

2024-04-08 Thread via GitHub
potiuk commented on issue #34306: URL: https://github.com/apache/airflow/issues/34306#issuecomment-2042058864 > When you say "automate links", do you mean the links that have been done as a part of this PR as an example - https://github.com/apache/airflow/pull/34305 ? Precisely.

Re: [PR] Replace dill package to use cloudpickle [airflow]

2024-04-08 Thread via GitHub
VladaZakharova commented on code in PR #38531: URL: https://github.com/apache/airflow/pull/38531#discussion_r1555416136 ## airflow/models/taskinstance.py: ## @@ -1287,7 +1287,7 @@ class TaskInstance(Base, LoggingMixin): queued_dttm = Column(UtcDateTime)

svn commit: r68359 - /release/airflow/2.9.0/

2024-04-08 Thread ephraimanierobi
Author: ephraimanierobi Date: Mon Apr 8 08:50:03 2024 New Revision: 68359 Log: Release Airflow 2.9.0 from 2.9.0rc3 Added: release/airflow/2.9.0/ release/airflow/2.9.0/apache-airflow-2.9.0-source.tar.gz - copied unchanged from r68357,

Re: [PR] Paginate Airflow task logs [airflow]

2024-04-08 Thread via GitHub
uranusjr commented on code in PR #38807: URL: https://github.com/apache/airflow/pull/38807#discussion_r1555488988 ## airflow/providers/amazon/aws/log/s3_task_handler.py: ## @@ -178,7 +180,13 @@ def s3_read(self, remote_log_location: str, return_error: bool = False) -> str:

(airflow) annotated tag constraints-latest updated (f501e5b103 -> e996825d56)

2024-04-08 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to annotated tag constraints-latest in repository https://gitbox.apache.org/repos/asf/airflow.git *** WARNING: tag constraints-latest was modified! *** from f501e5b103 (tag) to

Re: [PR] Always use the executemany method when inserting rows in DbApiHook as it's way much faster [airflow]

2024-04-08 Thread via GitHub
uranusjr commented on code in PR #38715: URL: https://github.com/apache/airflow/pull/38715#discussion_r1555493833 ## airflow/providers/common/sql/hooks/sql.py: ## @@ -531,8 +539,6 @@ def insert_rows( target_fields=None, commit_every=1000,

(airflow) annotated tag 2.9.0 updated (50f22ffe91 -> 5473f1ad31)

2024-04-08 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a change to annotated tag 2.9.0 in repository https://gitbox.apache.org/repos/asf/airflow.git *** WARNING: tag 2.9.0 was modified! *** from 50f22ffe91 (commit) to 5473f1ad31 (tag) tagging

Re: [PR] Make _get_ti compatible with RPC [airflow]

2024-04-08 Thread via GitHub
uranusjr commented on code in PR #38570: URL: https://github.com/apache/airflow/pull/38570#discussion_r119108 ## airflow/serialization/pydantic/dag_run.py: ## @@ -78,7 +78,7 @@ def get_task_instances( def get_task_instance( self, task_id: str, -

(airflow) branch main updated (c81449372e -> f50cc6180c)

2024-04-08 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from c81449372e Ignore subdag operator warnings during the (re)initialise DB in tests (#38823) add f50cc6180c

(airflow) branch main updated (b687f59008 -> c81449372e)

2024-04-08 Thread taragolis
This is an automated email from the ASF dual-hosted git repository. taragolis pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from b687f59008 fix: try002 for provider microsoft azure (#38805) add c81449372e Ignore subdag operator warnings

  1   2   3   >