badalprasadsingh commented on code in PR #2585:
URL: https://github.com/apache/iceberg-go/pull/2585#discussion_r4575294230
##########
schema.go:
##########
@@ -422,7 +435,7 @@ func (s *Schema) UnmarshalJSON(b []byte) error {
return fmt.Errorf("%w: invalid schema fields: %w",
ErrInvalidSchema, err)
}
- if err := checkDuplicateFieldIDs(nil, fields); err != nil {
+ if err := checkAllDuplicateFieldIDs(fields); err != nil {
Review Comment:
it can now writes metadata it cannot read:
"table.MetadataBuilder.AddSchema" and "view.MetadataBuilder" keep caller ids
without a duplicate check -> so sql CreateView, or CommitTable +
NewAddSchemaUpdate on non-REST catalogs: accept unset ids then fail to load
better: make both builders reject duplicates, like NewMetadata does
a blocking approval comment
##########
schema_test.go:
##########
@@ -917,6 +917,42 @@ func TestUnmarshalSchemaRejectsDuplicateFieldIDs(t
*testing.T) {
}`,
contains: "multiple fields for id 1: id and key",
},
+ {
+ name: "duplicate zero ids",
+ schema: `{
+ "type": "struct",
+ "fields": [
+ {"id": 0, "name": "created_at", "type":
"timestamptz", "required": false},
Review Comment:
the new cases only cover struct fields
can you please add a duplicate -1, pleus a duplicate 0 on a list element and
a map value? each branch of "checkFieldIDs" passes allIDs on its own, and the
doc promises negatives too
##########
website/src/releases.md:
##########
@@ -49,6 +49,7 @@ Per-release notes (highlights, breaking changes,
contributors) are published on
- **SigV4 keeps the token or credential as `Original-Authorization`.** With
`rest.sigv4-enabled=true` (or `rest.auth.type=sigv4`) and a `token` or
`credential`, requests used to be signed with the token silently dropped,
because the SigV4 signature replaced the `Authorization` header. They now carry
it as `Original-Authorization`, included in the signed headers, as Iceberg Java
does. A server behind SigV4 that ignores the header is unaffected.
- **Float and double literal comparators use Iceberg ordering.**
`TypedLiteral.Comparator()` now orders `-0.0` before `+0.0` and NaN after all
other values; NaN compares equal to NaN. Callers that use these public
comparators may see a different order or equality result.
- **Table creation rejects duplicate field IDs.** `NewMetadata`, and
`CreateTable` on catalogs that build metadata through it (not REST), now return
`ErrInvalidSchema` when two fields in the schema share an ID, including unset
`0` or `-1` IDs, even for unpartitioned, unsorted tables. Before, the partition
spec or sort order could bind to the wrong column. Callers should assign unique
IDs (or let the schema be built with fresh IDs). This matches Java, whose
`Schema` constructor rejects duplicate IDs.
+- **Schema reads now reject duplicate field IDs.** Table, view and manifest
metadata whose schema has two fields sharing an ID now fails to load with
`ErrInvalidSchema`, for example `multiple fields for id 0: created_at and
event`. Duplicate positive IDs were already rejected on read, but `0` and
negative IDs were skipped because they are placeholders while a schema is being
built, so such metadata loaded and lookups by ID returned an arbitrary one of
the fields. Java Iceberg already rejects these schemas when `SchemaParser`
builds a `Schema`. Repair affected metadata through an atomic catalog metadata
replacement instead of editing the currently referenced metadata file in place.
Schemas built in process are unaffected.
Review Comment:
pllease say who is affected and how to repair
go wrote these: sql CreateView and non-REST "CommitTable" +
"NewAddSchemaUpdate" with unset ids, an old bad schema in history also blocks
loading
views can be dropped and recreated -> tables need the catalog pointed at a
metadata copy without that schema
a blocking approval comment as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]