Jay Milagroso created NIFI-16335:
------------------------------------
Summary: PutIcebergRecord fails with UnsupportedOperationException
on every table load against a REST catalog (freshness-aware loading not
implemented in CredentialsRefreshRESTClient)
Key: NIFI-16335
URL: https://issues.apache.org/jira/browse/NIFI-16335
Project: Apache NiFi
Issue Type: Bug
Components: NiFi API
Affects Versions: 2.11.0
Environment: NiFi 2.11.0, Java 21 (Oracle JDK 21.0.5), standalone
(non-clustered)
REST Catalog: Apache Polaris 1.7.0 (Docker, PostgreSQL 16 persistence backend)
Storage: AWS S3 via IAM role-based access
Reporter: Jay Milagroso
h3. Summary
PutIcebergRecord throws UnsupportedOperationException on every attempt
to load a target table, making the processor entirely unusable against
a REST Iceberg catalog performing freshness-aware table loading.
h3. Steps to Reproduce
1. Configure RESTIcebergCatalog controller service pointing at a working
Apache Polaris REST catalog endpoint (catalog, namespace, and table
independently confirmed reachable via direct REST API calls)
2. Configure PutIcebergRecord with valid Iceberg Catalog, Iceberg Writer
(ParquetIcebergWriter), and Record Reader (JsonTreeReader), targeting
an existing namespace/table
3. Feed the processor a valid flowfile matching the target schema
4. Start the processor
h3. Expected Result
Record is written to the Iceberg table; a new snapshot is committed.
h3. Actual Result
Processing halts immediately with:
{code}
java.lang.UnsupportedOperationException: Returning response headers is not
supported
at org.apache.iceberg.rest.RESTClient.get(RESTClient.java:131)
at
org.apache.iceberg.rest.RESTSessionCatalog.loadInternal(RESTSessionCatalog.java:438)
at
org.apache.iceberg.rest.RESTSessionCatalog.loadTable(RESTSessionCatalog.java:468)
at
org.apache.iceberg.catalog.BaseSessionCatalog$AsCatalog.loadTable(BaseSessionCatalog.java:105)
at
org.apache.nifi.processors.iceberg.PutIcebergRecord.getTable(PutIcebergRecord.java:223)
at
org.apache.nifi.processors.iceberg.PutIcebergRecord.processFlowFiles(PutIcebergRecord.java:165)
at
org.apache.nifi.processors.iceberg.PutIcebergRecord.onTrigger(PutIcebergRecord.java:157)
{code}
h3. Root Cause Analysis
Iceberg core's RESTSessionCatalog.loadInternal() calls RESTClient.get(...)
using the header-returning overload introduced for freshness-aware /
ETag-based table loading. NiFi's custom REST client implementation,
org.apache.nifi.services.iceberg.catalog.CredentialsRefreshRESTClient,
does not implement this overload and throws UnsupportedOperationException
instead of returning response headers.
This is reproducible independent of Access Delegation Strategy - confirmed
with both "Vended Credentials" and "Disabled" settings, ruling out anything
credential-vending-specific. The failure occurs on the base loadTable call
itself, before any data or credential handling.
h3. Confirmed NOT the Cause (ruled out during investigation)
* Catalog/namespace/table configuration - table independently verified
reachable and writable via direct curl against Polaris's REST API and
via pyiceberg's RestCatalog
* IAM/S3 permissions - confirmed working via the same direct tests
* Access Delegation Strategy - identical failure under both tested settings
h3. Workaround
Writing to the same table via pyiceberg's RestCatalog.load_table() and
.append() succeeds without error, confirming the catalog/table/permissions
are correctly configured and the bug is isolated to
CredentialsRefreshRESTClient's incomplete RESTClient interface
implementation.
h3. Suggested Fix
Implement the header-returning get() overload in CredentialsRefreshRESTClient
(delegating to the underlying HTTP client's response headers), consistent
w
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
