morningman commented on code in PR #69031:
URL: https://github.com/apache/doris/pull/69031#discussion_r4270964604
##########
fe/pom.xml:
##########
@@ -360,7 +362,7 @@ under the License.
<!-- ATTN: avro version must be consistent with Iceberg version -->
<!-- Please modify iceberg.version and avro.version together,
you can find avro version info in iceberg mvn repository -->
- <iceberg.version>1.11.0</iceberg.version>
+ <iceberg.version>1.12.0</iceberg.version>
Review Comment:
No code change for this one. The limitation is now listed under Behavior
changed and in the Release note of the PR description.
The SDK facts hold, and the affected set is a bit larger than described:
1.12 also dropped the default signer path `v1/aws/s3/sign`, so a remote-signing
setup that never set an endpoint key fails the same way, and translating
`s3.signer.*` would not help it. Putting that default back is not an option
either. `signer.endpoint` takes precedence over the table-scoped
`rest.remote-signing.endpoint`, so it would send servers that already implement
the new endpoint back to the old path.
Why no shim:
- Doris does not support remote signing for storage access. BE reads data
files only with the catalog's static credentials or REST-vended ones, and the
CREATE CATALOG storage probe forces `s3.remote-signing-enabled=false`. "Worked
with 1.11" holds only for FE metadata reads, in setups where the server forces
signing and BE has static keys.
- The removal is Iceberg's own deprecation plan: `s3.signer.*` was
deprecated in 1.11.0 for removal in 1.12.0. Spark or Flink on Iceberg 1.12
behave the same against the same server, so the fix belongs on the server: send
`signer.uri` / `signer.endpoint`, or serve the table-scoped sign endpoint.
Current Lakekeeper sends `signer.*` along with the legacy names, so the
Lakekeeper static AK/SK setup from the Doris docs, which signs through
Lakekeeper because STS is off, keeps working.
- Server-supplied config is merged inside `RESTSessionCatalog`, so
translating it needs a custom `ioBuilder` on every REST catalog. With one set,
`tableFileIO()` never reuses the catalog FileIO: every table load builds a new
FileIO and S3 client, and the way Doris decides which table FileIOs to close
has to be reworked. The keys it would translate to, `signer.uri` /
`signer.endpoint`, are deprecated in 1.12.0 and scheduled for removal in 1.13.0
with no replacement, so the shim would have to go again at the next upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
