laskoviymishka commented on code in PR #2585:
URL: https://github.com/apache/iceberg-go/pull/2585#discussion_r4577981250
##########
schema.go:
##########
@@ -422,7 +435,7 @@ func (s *Schema) UnmarshalJSON(b []byte) error {
return fmt.Errorf("%w: invalid schema fields: %w",
ErrInvalidSchema, err)
}
- if err := checkDuplicateFieldIDs(nil, fields); err != nil {
+ if err := checkAllDuplicateFieldIDs(fields); err != nil {
Review Comment:
Agreeing with @badalprasadsingh's thread: could `AddSchema` and the view
builder reject duplicate IDs like `NewMetadata` does since #2475? `AddSchema`
only runs `checkSchemaCompatibility`, which catches duplicate names, not IDs:
https://github.com/apache/iceberg-go/blob/bf9d00165c992cccff1c803a0eced3ba56fff248/table/metadata.go#L810-L839.
A schema with two fields at ID 0 builds via `AddSchema`, then
`ParseMetadataBytes` fails with `multiple fields for id 0: a and b`.
`view.NewMetadata` fails the same way; `main` loads both.
##########
website/src/releases.md:
##########
@@ -49,6 +49,7 @@ Per-release notes (highlights, breaking changes,
contributors) are published on
- **SigV4 keeps the token or credential as `Original-Authorization`.** With
`rest.sigv4-enabled=true` (or `rest.auth.type=sigv4`) and a `token` or
`credential`, requests used to be signed with the token silently dropped,
because the SigV4 signature replaced the `Authorization` header. They now carry
it as `Original-Authorization`, included in the signed headers, as Iceberg Java
does. A server behind SigV4 that ignores the header is unaffected.
- **Float and double literal comparators use Iceberg ordering.**
`TypedLiteral.Comparator()` now orders `-0.0` before `+0.0` and NaN after all
other values; NaN compares equal to NaN. Callers that use these public
comparators may see a different order or equality result.
- **Table creation rejects duplicate field IDs.** `NewMetadata`, and
`CreateTable` on catalogs that build metadata through it (not REST), now return
`ErrInvalidSchema` when two fields in the schema share an ID, including unset
`0` or `-1` IDs, even for unpartitioned, unsorted tables. Before, the partition
spec or sort order could bind to the wrong column. Callers should assign unique
IDs (or let the schema be built with fresh IDs). This matches Java, whose
`Schema` constructor rejects duplicate IDs.
+- **Schema reads now reject duplicate field IDs.** Table, view and manifest
metadata whose schema has two fields sharing an ID now fails to load with
`ErrInvalidSchema`, for example `multiple fields for id 0: created_at and
event`. Duplicate positive IDs were already rejected on read, but `0` and
negative IDs were skipped because they are placeholders while a schema is being
built, so such metadata loaded and lookups by ID returned an arbitrary one of
the fields. Java Iceberg already rejects these schemas when `SchemaParser`
builds a `Schema`. Repair affected metadata through an atomic catalog metadata
replacement instead of editing the currently referenced metadata file in place.
Schemas built in process are unaffected.
Review Comment:
Same ask as @badalprasadsingh's thread on this line: could the note drop
"Schemas built in process are unaffected" and say who's affected? A schema with
two 0 IDs built through `AddSchema` or the view builder writes fine and now
fails to load. Every entry in `schemas` goes through `UnmarshalJSON`
(https://github.com/apache/iceberg-go/blob/bf9d00165c992cccff1c803a0eced3ba56fff248/table/metadata.go#L2399),
so an old bad schema in history blocks the load too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]