markhoerth opened a new issue, #14430:
URL: https://github.com/apache/gravitino/issues/14430
### Version
main branch, commit bb634defdc5663a8124be5fca1c8d9e52ac096ac (2.0.0-SNAPSHOT)
### Describe what's wrong
A user creates a format-version 3 table through the Iceberg REST service
with required columns that carry `initial-default` and `write-default` values.
The IRC stores them and returns them in the table metadata. When the user loads
the same table through the Gravitino API (`GET
/api/metalakes/{ml}/catalogs/{c}/schemas/{s}/tables/{t}`), the columns come
back without any `defaultValue`. The Gravitino API and the IRC therefore report
different column definitions for the same table, and Gravitino clients (UI,
connectors, the Java and Python clients) cannot see the value a writer fills in
for the column.
Cause, in Gravitino's code:
- `IcebergCatalogOperations.loadTable`
(`catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogOperations.java`
lines 413-421) builds the table with `IcebergTable.fromIcebergTable`
(`catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergTable.java`
lines 160-178), which maps each field with `ConvertUtil::fromNestedField`
(line 167).
- `ConvertUtil.fromNestedField`
(`catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ConvertUtil.java`
lines 73-80) sets only name, nullability, comment and type. It never reads
`Types.NestedField.writeDefault()` (or `initialDefault()`), so the column's
default value stays unset.
- `IcebergCatalogCapability.columnDefaultValue()`
(`catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogCapability.java`
lines 39-43) still declares column defaults unsupported, with a comment that
Iceberg support is in progress, although the bundled Iceberg 1.11.0 supports
them for format-version 3.
`IcebergView` uses the same converter (`IcebergView.java` line 149), so
views have the same gap.
### Error message and/or stacktrace
IRC create, HTTP 200. Schema fields in the response:
```
[{"id":1,"name":"id","required":true,"type":"long"},{"id":2,"name":"region","required":true,"type":"string","initial-default":"US","write-default":"EU"},{"id":3,"name":"n","required":true,"type":"int","initial-default":7,"write-default":8}]
```
Gravitino API GET, HTTP 200. Columns:
```
[{"name":"id","type":"long","nullable":false,"autoIncrement":false},{"name":"region","type":"string","nullable":false,"autoIncrement":false},{"name":"n","type":"integer","nullable":false,"autoIncrement":false}]
```
No `defaultValue` is reported for `region` or `n`.
### How to reproduce
Precondition: the Iceberg REST service runs with
`gravitino.iceberg-rest.catalog-config-provider = dynamic-config-provider` and
`gravitino.iceberg-rest.gravitino-metalake = vrf_ml`.
1. Create a metalake:
```
curl -X POST -H "Authorization: Basic YWRtaW46eA==" -H "Content-Type:
application/json" http://localhost:8090/api/metalakes -d
'{"name":"vrf_ml","comment":"","properties":{}}'
```
2. Create an Iceberg catalog:
```
curl -X POST -H "Authorization: Basic YWRtaW46eA==" -H "Content-Type:
application/json" http://localhost:8090/api/metalakes/vrf_ml/catalogs -d
'{"name":"vrf_def_cat","type":"RELATIONAL","provider":"lakehouse-iceberg","comment":"","properties":{"catalog-backend":"jdbc","uri":"jdbc:h2:file:/tmp/vrf_def_wh/h2;AUTO_SERVER=TRUE;DATABASE_TO_LOWER=TRUE","jdbc-driver":"org.h2.Driver","jdbc-user":"sa","jdbc-password":"","jdbc-initialize":"true","warehouse":"/tmp/vrf_def_wh/wh"}}'
```
3. Create a namespace:
```
curl -X POST -H "Authorization: Basic YWRtaW46eA==" -H "Content-Type:
application/json" http://localhost:9001/iceberg/v1/vrf_def_cat/namespaces -d
'{"namespace":["vrf_def"],"properties":{}}'
```
4. Create a format-version 3 table with defaults through the IRC:
```
curl -X POST -H "Authorization: Basic YWRtaW46eA==" -H "Content-Type:
application/json"
http://localhost:9001/iceberg/v1/vrf_def_cat/namespaces/vrf_def/tables -d
'{"name":"d3","properties":{"format-version":"3"},"schema":{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":true,"type":"long"},{"id":2,"name":"region","required":true,"type":"string","initial-default":"US","write-default":"EU"},{"id":3,"name":"n","required":true,"type":"int","initial-default":7,"write-default":8}]}}'
```
5. Load it through the Gravitino API. The `region` and `n` columns have no
`defaultValue`:
```
curl -H "Authorization: Basic YWRtaW46eA=="
http://localhost:8090/api/metalakes/vrf_ml/catalogs/vrf_def_cat/schemas/vrf_def/tables/d3
```
### Additional context
Related: #1556 (closed, the Iceberg catalog rejects column default values at
create time) and #11954 (closed, format-version 3 table creation through the
Gravitino API). No existing issue covers reading Iceberg defaults back through
the Gravitino API. The cause is Gravitino's own converter, not Iceberg.
Found by the 2.0 OSS test suite check
`test_v3_required_columns_with_defaults_match_gravitino`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
