tuantran0910 commented on code in PR #62988:
URL: https://github.com/apache/doris/pull/62988#discussion_r4624493209


##########
fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java:
##########
@@ -455,6 +459,18 @@ public static Map<String, String> 
buildCatalogProperties(IcebergCatalogPropertie
         }
         chosenS3.ifPresent(storage -> 
storage.toBackendProperties().ifPresent(backend -> {
             Optional<GcsAuth> auth = GcsAuthResolver.resolve(backend.toMap());
+            // Keep explicit FileIO choices. The plugin bundles iceberg-gcp, 
so upstream ResolvingFileIO
+            // routes gs to GCSFileIO whenever that class is loadable - 
independently of which GCS
+            // credential mode is configured - and drops the fs.gs.* / 
fs.s3a.* configuration Doris
+            // builds for GCS. Native ADC / COMPUTE_ENGINE bindings always get 
the pin (they emit the
+            // S3FileIO dialect below). HMAC and anonymous bindings get it 
only on REST, the one flavor
+            // whose SDK default is ResolvingFileIO; hms / hadoop / jdbc 
default to HadoopFileIO and glue
+            // to S3FileIO, and HMS emits no S3FileIO options, so the pin 
would send its s3/s3a paths to
+            // an unconfigured S3FileIO.
+            auth.filter(gcsAuth -> gcsAuth.getNativeCredential().isPresent()
+                            || 
IcebergCatalogProperties.TYPE_REST.equals(flavor))
+                    .ifPresent(ignored -> 
opts.putIfAbsent(CatalogProperties.FILE_IO_IMPL,
+                            NativeGcsResolvingFileIO.class.getName()));

Review Comment:
   Thanks, confirmed and fixed in 245515cac8.
   
   Before this PR the plugin had no `iceberg-gcp`, so `ResolvingFileIO` could 
not load `GCSFileIO` and fell back to `HadoopFileIO`, which picks up the raw 
`fs.gs.*` keys passed through by `buildHadoopConfiguration()`. A REST catalog 
with an opaque warehouse and only `fs.gs.*` settings binds no typed GCS storage 
(`GcsAuthResolver.guessIsGcs()` does not look at `fs.*`), so `chosenS3` is 
empty and the default was never applied.
   
   `buildCatalogProperties` now also pins `NativeGcsResolvingFileIO` for REST 
catalogs that carry any raw `fs.gs.*` key, with `putIfAbsent`, so an explicit 
`io-impl` / `iceberg.rest.io-impl` still wins. `gs://` goes back to 
`HadoopFileIO` with the configured keyfile, as at the merge base.
   
   I gated it on `fs.gs.*` keys rather than applying it to every REST catalog: 
client properties override the server's `/v1/config` defaults, so an 
unconditional pin would replace a server-recommended `io-impl` on catalogs that 
never configured GCS. Happy to make it unconditional if you prefer that 
trade-off.
   
   Tests added in `IcebergCatalogFactoryTest`:
   - REST, empty `chosenS3`, only `fs.gs.impl` + keyfile keys -> `io-impl` is 
`NativeGcsResolvingFileIO`
   - explicit `io-impl` / `iceberg.rest.io-impl` is kept
   - REST without `fs.gs.*` keys -> no `io-impl` set
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to