[
https://issues.apache.org/jira/browse/NIFI-16335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jay Milagroso updated NIFI-16335:
---------------------------------
Description:
h3. Summary
PutIcebergRecord throws UnsupportedOperationException on every attempt
to load a target table, making the processor entirely unusable against
a REST Iceberg catalog performing freshness-aware table loading.
h3. Steps to Reproduce
1. Configure RESTIcebergCatalog controller service pointing at a working
Apache Polaris REST catalog endpoint (catalog, namespace, and table
independently confirmed reachable via direct REST API calls)
2. Configure PutIcebergRecord with valid Iceberg Catalog, Iceberg Writer
(ParquetIcebergWriter), and Record Reader (JsonTreeReader), targeting
an existing namespace/table
3. Feed the processor a valid flowfile matching the target schema
4. Start the processor
h3. Expected Result
Record is written to the Iceberg table; a new snapshot is committed.
h3. Actual Result
Processing halts immediately with:
{code:java}
java.lang.UnsupportedOperationException: Returning response headers is not
supported
at org.apache.iceberg.rest.RESTClient.get(RESTClient.java:131)
at
org.apache.iceberg.rest.RESTSessionCatalog.loadInternal(RESTSessionCatalog.java:438)
at
org.apache.iceberg.rest.RESTSessionCatalog.loadTable(RESTSessionCatalog.java:468)
at
org.apache.iceberg.catalog.BaseSessionCatalog$AsCatalog.loadTable(BaseSessionCatalog.java:105)
at
org.apache.nifi.processors.iceberg.PutIcebergRecord.getTable(PutIcebergRecord.java:223)
at
org.apache.nifi.processors.iceberg.PutIcebergRecord.processFlowFiles(PutIcebergRecord.java:165)
at
org.apache.nifi.processors.iceberg.PutIcebergRecord.onTrigger(PutIcebergRecord.java:157)
{code}
h3. Root Cause Analysis
Iceberg core's RESTSessionCatalog.loadInternal() calls RESTClient.get(...)
using the header-returning overload introduced for freshness-aware /
ETag-based table loading. NiFi's custom REST client implementation,
org.apache.nifi.services.iceberg.catalog.CredentialsRefreshRESTClient,
does not implement this overload and throws UnsupportedOperationException
instead of returning response headers.
This is reproducible independent of Access Delegation Strategy - confirmed
with both "Vended Credentials" and "Disabled" settings, ruling out anything
credential-vending-specific. The failure occurs on the base loadTable call
itself, before any data or credential handling.
h3. Confirmed NOT the Cause (ruled out during investigation)
* Catalog/namespace/table configuration - table independently verified
reachable and writable via direct curl against Polaris's REST API and
via pyiceberg's RestCatalog
* IAM/S3 permissions - confirmed working via the same direct tests
* Access Delegation Strategy - identical failure under both tested settings
h3. Workaround
Writing to the same table via pyiceberg's RestCatalog.load_table() and
.append() succeeds without error, confirming the catalog/table/permissions
are correctly configured and the bug is isolated to
CredentialsRefreshRESTClient's incomplete RESTClient interface
implementation.
h3. Suggested Fix
Implement the header-returning get() overload in CredentialsRefreshRESTClient
(delegating to the underlying HTTP client's response headers), consistent
w
was:
h3. Summary
PutIcebergRecord throws UnsupportedOperationException on every attempt
to load a target table, making the processor entirely unusable against
a REST Iceberg catalog performing freshness-aware table loading.
h3. Steps to Reproduce
1. Configure RESTIcebergCatalog controller service pointing at a working
Apache Polaris REST catalog endpoint (catalog, namespace, and table
independently confirmed reachable via direct REST API calls)
2. Configure PutIcebergRecord with valid Iceberg Catalog, Iceberg Writer
(ParquetIcebergWriter), and Record Reader (JsonTreeReader), targeting
an existing namespace/table
3. Feed the processor a valid flowfile matching the target schema
4. Start the processor
h3. Expected Result
Record is written to the Iceberg table; a new snapshot is committed.
h3. Actual Result
Processing halts immediately with:
{code}
java.lang.UnsupportedOperationException: Returning response headers is not
supported
at org.apache.iceberg.rest.RESTClient.get(RESTClient.java:131)
at
org.apache.iceberg.rest.RESTSessionCatalog.loadInternal(RESTSessionCatalog.java:438)
at
org.apache.iceberg.rest.RESTSessionCatalog.loadTable(RESTSessionCatalog.java:468)
at
org.apache.iceberg.catalog.BaseSessionCatalog$AsCatalog.loadTable(BaseSessionCatalog.java:105)
at
org.apache.nifi.processors.iceberg.PutIcebergRecord.getTable(PutIcebergRecord.java:223)
at
org.apache.nifi.processors.iceberg.PutIcebergRecord.processFlowFiles(PutIcebergRecord.java:165)
at
org.apache.nifi.processors.iceberg.PutIcebergRecord.onTrigger(PutIcebergRecord.java:157)
{code}
h3. Root Cause Analysis
Iceberg core's RESTSessionCatalog.loadInternal() calls RESTClient.get(...)
using the header-returning overload introduced for freshness-aware /
ETag-based table loading. NiFi's custom REST client implementation,
org.apache.nifi.services.iceberg.catalog.CredentialsRefreshRESTClient,
does not implement this overload and throws UnsupportedOperationException
instead of returning response headers.
This is reproducible independent of Access Delegation Strategy - confirmed
with both "Vended Credentials" and "Disabled" settings, ruling out anything
credential-vending-specific. The failure occurs on the base loadTable call
itself, before any data or credential handling.
h3. Confirmed NOT the Cause (ruled out during investigation)
* Catalog/namespace/table configuration - table independently verified
reachable and writable via direct curl against Polaris's REST API and
via pyiceberg's RestCatalog
* IAM/S3 permissions - confirmed working via the same direct tests
* Access Delegation Strategy - identical failure under both tested settings
h3. Workaround
Writing to the same table via pyiceberg's RestCatalog.load_table() and
.append() succeeds without error, confirming the catalog/table/permissions
are correctly configured and the bug is isolated to
CredentialsRefreshRESTClient's incomplete RESTClient interface
implementation.
h3. Suggested Fix
Implement the header-returning get() overload in CredentialsRefreshRESTClient
(delegating to the underlying HTTP client's response headers), consistent
w
> PutIcebergRecord fails with UnsupportedOperationException on every table load
> against a REST catalog (freshness-aware loading not implemented in
> CredentialsRefreshRESTClient)
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16335
> URL: https://issues.apache.org/jira/browse/NIFI-16335
> Project: Apache NiFi
> Issue Type: Bug
> Components: NiFi API
> Affects Versions: 2.11.0
> Environment: NiFi 2.11.0, Java 21 (Oracle JDK 21.0.5), standalone
> (non-clustered)
> REST Catalog: Apache Polaris 1.7.0 (Docker, PostgreSQL 16 persistence backend)
> Storage: AWS S3 via IAM role-based access
> Reporter: Jay Milagroso
> Assignee: David Handermann
> Priority: Major
>
> h3. Summary
> PutIcebergRecord throws UnsupportedOperationException on every attempt
> to load a target table, making the processor entirely unusable against
> a REST Iceberg catalog performing freshness-aware table loading.
>
> h3. Steps to Reproduce
> 1. Configure RESTIcebergCatalog controller service pointing at a working
> Apache Polaris REST catalog endpoint (catalog, namespace, and table
> independently confirmed reachable via direct REST API calls)
> 2. Configure PutIcebergRecord with valid Iceberg Catalog, Iceberg Writer
> (ParquetIcebergWriter), and Record Reader (JsonTreeReader), targeting
> an existing namespace/table
> 3. Feed the processor a valid flowfile matching the target schema
> 4. Start the processor
>
> h3. Expected Result
> Record is written to the Iceberg table; a new snapshot is committed.
>
> h3. Actual Result
> Processing halts immediately with:
> {code:java}
> java.lang.UnsupportedOperationException: Returning response headers is not
> supported
> at org.apache.iceberg.rest.RESTClient.get(RESTClient.java:131)
> at
> org.apache.iceberg.rest.RESTSessionCatalog.loadInternal(RESTSessionCatalog.java:438)
> at
> org.apache.iceberg.rest.RESTSessionCatalog.loadTable(RESTSessionCatalog.java:468)
> at
> org.apache.iceberg.catalog.BaseSessionCatalog$AsCatalog.loadTable(BaseSessionCatalog.java:105)
> at
> org.apache.nifi.processors.iceberg.PutIcebergRecord.getTable(PutIcebergRecord.java:223)
> at
> org.apache.nifi.processors.iceberg.PutIcebergRecord.processFlowFiles(PutIcebergRecord.java:165)
> at
> org.apache.nifi.processors.iceberg.PutIcebergRecord.onTrigger(PutIcebergRecord.java:157)
> {code}
>
> h3. Root Cause Analysis
> Iceberg core's RESTSessionCatalog.loadInternal() calls RESTClient.get(...)
> using the header-returning overload introduced for freshness-aware /
> ETag-based table loading. NiFi's custom REST client implementation,
> org.apache.nifi.services.iceberg.catalog.CredentialsRefreshRESTClient,
> does not implement this overload and throws UnsupportedOperationException
> instead of returning response headers.
>
> This is reproducible independent of Access Delegation Strategy - confirmed
> with both "Vended Credentials" and "Disabled" settings, ruling out anything
> credential-vending-specific. The failure occurs on the base loadTable call
> itself, before any data or credential handling.
>
> h3. Confirmed NOT the Cause (ruled out during investigation)
> * Catalog/namespace/table configuration - table independently verified
> reachable and writable via direct curl against Polaris's REST API and
> via pyiceberg's RestCatalog
> * IAM/S3 permissions - confirmed working via the same direct tests
> * Access Delegation Strategy - identical failure under both tested settings
> h3. Workaround
> Writing to the same table via pyiceberg's RestCatalog.load_table() and
> .append() succeeds without error, confirming the catalog/table/permissions
> are correctly configured and the bug is isolated to
> CredentialsRefreshRESTClient's incomplete RESTClient interface
> implementation.
>
> h3. Suggested Fix
> Implement the header-returning get() overload in CredentialsRefreshRESTClient
> (delegating to the underlying HTTP client's response headers), consistent
> w
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
