helmanofer opened a new issue, #5128:
URL: https://github.com/apache/polaris/issues/5128
### Describe the bug
Polaris Generic Tables can register non-Iceberg table formats, but their API
model has no first-class schema field. The create/load payload only exposes
`name`, `format`, `base-location`, `doc`, and string `properties`.
As a result, a catalog integration cannot populate ordered columns, types,
or required/nullability metadata for a Generic Table. Clients and catalog UIs
can render a structured schema for Iceberg tables from `metadata.schemas`, but
equivalent metadata cannot be represented for Generic Tables.
### To Reproduce
1. Create a Generic Table for a non-Iceberg format.
2. Include its schema as JSON in a string property because no schema field
exists.
3. Load the Generic Table through the Polaris API.
4. Observe that schema information is available only as an opaque property;
there is no structured schema in the response and no schema update operation.
Example workaround property:
```json
{
"vastdb.schema-json":
"{\"columns\":[{\"name\":\"customer_id\",\"type\":\"long\",\"nullable\":true}]}"
}
```
### Actual Behavior
Column metadata must be serialized into arbitrary string properties. It is
not validated, typed, queryable, interoperable, or exposed through the same
schema model used for Iceberg tables. A schema change also requires replacing
the Generic Table metadata rather than updating a structured schema.
### Expected Behavior
Generic Tables should support an optional structured schema in their
create/load/update API, including:
- ordered field names;
- field types, including nested types;
- required/nullability information; and
- schema evolution or replacement.
This would let non-Iceberg formats participate as proper catalog tables
rather than name/location registrations with opaque properties.
### Additional context
This affects any integration cataloging external formats whose physical
system remains authoritative for schema. Encoding schema JSON in properties
preserves information but does not provide catalog-level schema semantics.
### System information
Polaris Catalog Version: 1.5.0
Object storage & setup: S3-compatible storage with native non-Iceberg
Generic Tables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
