JosefWomelsdorf opened a new issue, #4062:
URL: https://github.com/apache/polaris/issues/4062
## Summary
When using `SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION=true` (required for
S3-compatible stores like VAST that don't support STS AssumeRole),
`table-default.*` catalog properties such as `s3.endpoint` and
`s3.path-style-access` do not reach S3FileIO during table write operations.
Polaris falls back to the default AWS S3 endpoint.
## Affected versions
1.3.0-incubating (likely earlier versions too)
## Root cause
Three code paths are missing the merge of \`table-default.*\` properties:
### 1. \`PolarisIcebergCatalogTableBuilder\` constructor
\`PolarisIcebergCatalogViewBuilder\` correctly calls
\`withProperties(propertiesWithPrefix(..., "table-default."))\` in its
constructor. The table builder is missing this call, so
\`metadata.properties()\` in \`doCommit()\` never contains catalog defaults for
direct creates.
**Fix:** Add to \`PolarisIcebergCatalogTableBuilder\` constructor in
\`IcebergCatalog.java\`:
\`\`\`java
withProperties(
PropertyUtil.propertiesWithPrefix(IcebergCatalog.this.properties(),
"table-default."));
\`\`\`
### 2. \`IcebergCatalogHandler.stageTableCreateHelper()\`
Clients like DuckDB send \`stage-create: true\`. This path calls
\`TableMetadata.newTableMetadata()\` directly, bypassing \`buildTable()\`
entirely. The \`properties\` map passed to \`newTableMetadata()\` only contains
user-provided properties — no catalog defaults.
**Fix:** Merge catalog defaults before calling \`newTableMetadata()\` in
\`IcebergCatalogHandler.java\`:
\`\`\`java
Map<String, String> mergedProperties = Maps.newHashMap();
if (baseCatalog instanceof IcebergCatalog polarisCatalog) {
mergedProperties.putAll(PropertyUtil.propertiesWithPrefix(
polarisCatalog.properties(), "table-default."));
}
mergedProperties.putAll(properties);
// then pass mergedProperties to newTableMetadata()
\`\`\`
### 3. \`BasePolarisTableOperations.doCommit()\`
Even with fixes 1+2, clients using staged create (e.g. DuckDB) send sparse
commit payloads with only schema/location updates — no \`SetProperties\`
update. So \`metadata.properties()\` in \`doCommit()\` is empty of catalog
defaults, and the FileIO used for the metadata write ignores \`s3.endpoint\`.
**Fix:** Use \`tableDefaultProperties\` as baseline in \`doCommit()\` in
\`IcebergCatalog.java\`:
\`\`\`java
Map<String, String> commitFileIOProps = new
HashMap<>(tableDefaultProperties);
commitFileIOProps.putAll(metadata.properties());
tableFileIO = loadFileIOForTableLike(..., commitFileIOProps, ...);
\`\`\`
## Impact
All three fixes are needed together for DuckDB (staged create). Fix 1 alone
is sufficient for tools that use direct creates (\`stage-create: false\`).
## Workaround
Pre-compile and inject the three patched classes into the Polaris service
JAR via a Kubernetes init container.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
