mfellner commented on issue #2418:
URL: https://github.com/apache/iceberg-rust/issues/2418#issuecomment-6112672992
Related to #2411 (v3 tracking).
Re-checked against main at a76678363.
**Current schema: covered on main.** Creating a v2 table, or loading v2
metadata whose current schema contains `timestamp_ns`, `timestamptz_ns`,
`variant` or `unknown`, now fails with:
```
DataInvalid => Invalid schema for v2:
- Invalid type for x: timestamp_ns is not supported until v3 but format
version is v2.
```
The type floors and this check were added in #2188 (merged 2026-07-16).
According to the git tags, #2188 is in v0.11.0-rc.1 and later, and not in
v0.10.1. `unknown` was added later in #2773, which is not in any tag up to
v0.11.0-rc.2. #2417, which referenced this issue, was closed without merging
with the comment "Covered by #2188".
**Remaining gap: non-current schemas.** The check runs only on the current
schema. `TableMetadata::try_normalize` calls
`validate_schema_format_compatibility`, which checks `current_schema()` only.
As a result, on a v2 table:
- `TableMetadataBuilder::add_schema(..)` followed by `build()` returns `Ok`
when a non-current schema contains `timestamp_ns`, `timestamptz_ns`, `variant`
or `unknown`.
- Deserializing v2 metadata JSON that contains such a historical schema
succeeds.
Java rejects the first case. `TableMetadata.Builder.addSchemaInternal` calls
`Schema.checkCompatibility(schema, formatVersion)`, which throws
`IllegalStateException`. This comes from reading apache/iceberg main, not from
running Java. Java's metadata parser does not run this check on the schemas it
reads, so the difference is limited to the builder path.
Expected: `add_schema` rejects a schema that uses types the table's format
version does not support, as Java does. Whether loading should also reject
historical schemas is for maintainers to decide, since Java's parser accepts
them.
Maintainers could close this issue and track the non-current-schema gap
separately, or retitle this one to cover it.
<details>
<summary>Repro (scratch copy of a76678363, integration test using only the
public API)</summary>
```rust
use iceberg::spec::{NestedField, PrimitiveType, Schema, TableMetadata,
TableMetadataBuilder, Type};
const V2_METADATA: &str =
include_str!("../testdata/table_metadata/TableMetadataV2Valid.json");
#[test]
fn builder_add_schema_with_timestamp_ns() {
let base: TableMetadata = serde_json::from_str(V2_METADATA).unwrap();
let schema = Schema::builder()
.with_schema_id(2)
.with_fields(vec![
NestedField::optional(2, "ts",
Type::Primitive(PrimitiveType::TimestampNs)).into(),
])
.build()
.unwrap();
let res = TableMetadataBuilder::new_from_metadata(base, None)
.add_schema(schema)
.and_then(|b| b.build());
println!("BUILDER ok={} err={:?}", res.is_ok(),
res.as_ref().err().map(|e| e.to_string()));
}
#[test]
fn deserialize_v2_metadata_with_historical_timestamp_ns_schema() {
let mut v: serde_json::Value =
serde_json::from_str(V2_METADATA).unwrap();
v["schemas"].as_array_mut().unwrap().push(serde_json::json!({
"type": "struct",
"schema-id": 2,
"fields": [{"id": 2, "name": "ts", "required": false, "type":
"timestamp_ns"}]
}));
let res = serde_json::from_value::<TableMetadata>(v);
println!("DESER ok={} err={:?}", res.is_ok(), res.as_ref().err().map(|e|
e.to_string()));
}
```
Output on main (`cargo test --offline -p iceberg --test repro_2418_min --
--nocapture`):
```
BUILDER ok=true err=None
DESER ok=true err=None
```
The same two tests with `timestamptz_ns`, `variant` and `unknown` in place
of `timestamp_ns` also return `ok=true`.
</details>
Found during an agent-assisted audit of v3 support (see #2411). The
reproduction above was run against main at a76678363.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
