Thanks Ryan, that simplifies the migration a lot. If servers always omit
residuals, the new expression syntax only flows from client to server, in
the
plan request filter, so no client capability signal is needed.
Unless there are objections, I'll:
1. Open a PR so the reference implementation omits residual-filter from
scan
tasks. Today CatalogHandlers sends a residual on every task.
2. Continue with the conformance fixtures for expression JSON, since the
same
JSON is still used in plan request filters, scan reports, and view/UDF
metadata.
Thanks,
Revanth
On Fri, 2 Oct 2026 at 14:34, Ryan Blue <[email protected]> wrote:
> I would recommend always dropping residuals. They are expensive to
> calculate and almost never applied.
>
> On Fri, Oct 2, 2026 at 11:18 AM Revanth Ch <[email protected]> wrote:
>
>> Hi everyone,
>>
>> I'd like the community's call on how REST servers should send residuals
>> while
>> clients migrate to the new expression syntax, and I'm proposing
>> conformance
>> fixtures in apache/iceberg-verification to track that migration.
>>
>> Background
>>
>> The expressions spec (apache/iceberg#17138) changed the preferred
>> predicate form:
>>
>> before (deprecated): {"type": "eq", "term": "id", "value": 5}
>> after (preferred): {"type": "eq", "left": {"type": "reference",
>> "name": "id"}, "right": 5}
>>
>> It also adds "literals" objects and "apply". Java, Go and C++ still read
>> and write
>> only the deprecated predicate form (Go checked by running it; Java and
>> C++ by
>> reading the source).
>>
>> 1. Residuals during the migration
>>
>> The spec says services should omit residuals unless the client supports
>> the new
>> syntax, "for example, via client version". But the Java reference server
>> always
>> sends residuals, and there's no reliable way to detect support:
>> X-Client-Version
>> isn't in the REST spec, and Java sends its library version while
>> iceberg-go sends
>> the REST spec version. Options:
>>
>> a. Always omit residuals. Simple and correct, but clients re-evaluate
>> the full
>> filter per task instead of the server's simplified residual.
>> b. Clients declare support explicitly (an optional PlanTableScanRequest
>> field
>> or a spec-defined header); servers send new-form residuals only to
>> them.
>> c. Infer from client version. Unreliable, for the reason above.
>>
>> I'd suggest (b), with readers in every implementation updated before any
>> writer
>> switches.
>>
>> 2. Conformance fixtures
>>
>> Add read-conformance surfaces to apache/iceberg-verification for
>> expression JSON
>> (deprecated and preferred forms) and REST scan tasks. Implementations opt
>> in, and
>> forms they can't read yet show up as UNSUPPORTED, so the migration is
>> visible
>> across implementations. Does this fit the next phase of
>> apache/iceberg-verification#10? Happy to open the PRs.
>>
>> Related spec-doc fixes, filed separately:
>> - Drop {"type":"true"} from OpenAPI:
>> https://github.com/apache/iceberg/issues/18351
>> - Hex case for binary/fixed:
>> https://github.com/apache/iceberg/issues/18352
>>
>> Thanks,
>> Revanth
>>
>
