a-agmon commented on issue #338:
URL: https://github.com/apache/iceberg-rust/issues/338#issuecomment-2092863958
Another way to resolve this, in a less workaround-ish way, is simply to
capture the fact that we have a V1 schema, a V2 schema, and a V2 compatibility
schema, which is identical to V2 with just the names according to V1.
Then in the `parse_with_version` function, simply try with V2 and fallback
to V2Compat if it failed.
something like this perhaps:
```rust
pub fn parse_with_version(
bs: &[u8],
version: FormatVersion,
partition_type_provider: impl Fn(i32) -> Result<Option<StructType>>,
) -> Result<ManifestList> {
match version {
FormatVersion::V1 => {
let reader =
Reader::with_schema(&MANIFEST_LIST_AVRO_SCHEMA_V1, bs)?;
let values =
Value::Array(reader.collect::<std::result::Result<Vec<Value>, _>>()?);
from_value::<_serde::ManifestListV1>(&values)?.try_into(partition_type_provider)
}
FormatVersion::V2 => {
let reader =
Reader::with_schema(&MANIFEST_LIST_AVRO_SCHEMA_V2, bs)?;
let read_result =
reader.collect::<std::result::Result<Vec<Value>, _>>();
match read_result {
Ok(records) => {
let values = Value::Array(records);
from_value::<_serde::ManifestListV2>(&values)?
.try_into(&partition_type_provider)
}
Err(e) => {
println!("Error reading values fallling back to
V2_COMAPT: {:?}", e);
let reader =
Reader::with_schema(&MANIFEST_LIST_AVRO_SCHEMA_V2_COMPAT, bs)?;
let records =
reader.collect::<std::result::Result<Vec<Value>, _>>()?;
let values = Value::Array(records);
from_value::<_serde::ManifestListV2Compat>(&values)?
.try_into(&partition_type_provider)
}
}
}
}
}
```
Check out this
[branch](https://github.com/a-agmon/iceberg-rust/blob/fix-support-v2-compat/crates/iceberg/src/spec/manifest_list.rs)
which implements this.
WDYT @Fokko @zeodtr ?
As @zeodtr says, breaking compact with Spark < 3.5 is going to be a major
issue unless we either support a compat schema or move to work with `field-id`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
