vincbeck commented on code in PR #76086:
URL: https://github.com/apache/airflow/pull/76086#discussion_r4428599228
##########
providers/amazon/docs/operators/duckdb.rst:
##########
@@ -132,9 +132,10 @@ DuckDB how to authenticate, and the
``credential_strategy`` parameter selects ho
``credential_chain`` (default)
Issues ``CREATE SECRET (TYPE s3, PROVIDER credential_chain)``, which
resolves credentials through
- the AWS SDK inside DuckDB's ``aws`` extension. Nothing secret is written
into SQL text and the SDK
- refreshes expiring credentials itself, so a long query does not fail
partway through when a set of
- temporary credentials expires.
+ the AWS SDK inside DuckDB's ``aws`` extension. Nothing secret is written
into SQL text, and the
+ secret is created with ``REFRESH 'auto'``: when S3 rejects expired
credentials, DuckDB resolves the
+ chain again and retries, so a long query does not fail partway through
when a set of temporary
+ credentials expires.
Review Comment:
Should we mention the limits from the PR description here (and in the hook
docstring)? On DuckDB 1.2.0/1.2.1 nothing is refreshed, and from 1.2.2 to 1.5.5
only reads that open a file refresh. As written, a user on 1.4 running a long
`COPY ... TO 's3://...'` would expect it not to fail.
##########
providers/amazon/src/airflow/providers/amazon/aws/hooks/duckdb.py:
##########
@@ -237,6 +240,10 @@ def _provider_clauses(self, strategy: CredentialStrategy)
-> list[str]:
clauses = ["PROVIDER credential_chain"]
if self.credential_chain:
clauses.append(f"CHAIN '{self._quote(self.credential_chain)}'")
+ # Without REFRESH, DuckDB keeps the credentials it resolved at
creation for the life of
+ # the secret. DuckDB 1.2.0 and 1.2.1 do not know the option and
fail the statement.
+ if Version(duckdb.__version__) >= Version("1.2.2"):
Review Comment:
Should we add `duckdb>=1.2.2` to the `duckdb` extra instead? Then no need
for the version check, and users on 1.2.0/1.2.1 don't silently lose the refresh.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
