EthanBlackburn opened a new issue, #398:
URL: https://github.com/apache/iceberg-go/issues/398
### Apache Iceberg version
None
### Please describe the bug 🐞
My schema contains a _nullable_ struct with _required_ fields. When I call
`AppendTable` with the data, I get the panic
```
error encountered during schema visitor invalid: field says not-nullable,
child #5 has nulls
stack trace: goroutine 103 [running]:
runtime/debug.Stack()
/Users/ethanblackburn/go/pkg/mod/golang.org/[email protected]/src/runtime/debug/stack.go:26
+0x64
github.com/apache/iceberg-go.VisitSchemaWithPartner[...].func1()
/Users/ethanblackburn/go/pkg/mod/github.com/apache/[email protected]/schema.go:1323
+0x80
panic({0x107544fc0?, 0x140012a6d60?})
/Users/ethanblackburn/go/pkg/mod/golang.org/[email protected]/src/runtime/panic.go:792
+0x124
github.com/apache/iceberg-go/table.retOrPanic[...](...)
/Users/ethanblackburn/go/pkg/mod/github.com/apache/[email protected]/table/arrow_utils.go:683
github.com/apache/iceberg-go/table.(*arrowProjectionVisitor).Struct(0x140012a63a0,
{{0x14000f1b888?, 0x109922ac0?, 0x107b2cbe0?}}, {0x107b60fe0, 0x14000e78f00},
{0x14000f57c00, 0x8, 0x14001678f00?})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/[email protected]/table/arrow_utils.go:820
+0x4e0
github.com/apache/iceberg-go.visitStructWithPartner[...]({{0x14000f1b888,
0x0?, 0x0?}}, {0x107b60fe0?, 0x14000e78f00}, {0x107b44d50, 0x140012a63a0},
{0x107b40220, 0x14001678fc0})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/[email protected]/schema.go:1366
+0x360
github.com/apache/iceberg-go.visitTypeWithPartner[...]({0x107b2c9a0?,
0x14000522168?}, {0x107b60fe0?, 0x14000e78f00?}, {0x107b44d50?,
0x140012a63a0?}, {0x107b40220?, 0x14001678fc0?})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/[email protected]/schema.go:1435
+0x98
github.com/apache/iceberg-go.visitStructWithPartner[...]({{0x14000e42e08,
0x0?, 0x0?}}, {0x107b60fe0?, 0x14000e78ec0}, {0x107b44d50, 0x140012a63a0},
{0x107b40220, 0x14001678fc0})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/[email protected]/schema.go:1359
+0x234
github.com/apache/iceberg-go.visitTypeWithPartner[...]({0x107b2c9a0?,
0x14000522138?}, {0x107b60fe0?, 0x14000e78ec0?}, {0x107b44d50?,
0x140012a63a0?}, {0x107b40220?, 0x14001678fc0?})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/[email protected]/schema.go:1435
+0x98
github.com/apache/iceberg-go.visitStructWithPartner[...]({{0x14001090008,
0x14000e78bc0?, 0x0?}}, {0x107b60fe0?, 0x14000e78bc0}, {0x107b44d50,
0x140012a63a0}, {0x107b40220, 0x14001678fc0})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/[email protected]/schema.go:1359
+0x234
github.com/apache/iceberg-go.VisitSchemaWithPartner[...](0x14000b91f10,
{0x107b60fe0, 0x14000e78bc0}, {0x107b44d50, 0x140012a63a0}, {0x107b40220,
0x14001678fc0})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/[email protected]/schema.go:1335
+0xf4
github.com/apache/iceberg-go/table.ToRequestedSchema({0x107b35350,
0x109922ac0}, 0x14000b91f10, 0x14001678fc0, {0x107b5b670?, 0x14000e71e00?},
0x0, 0x1, 0x0)
/Users/ethanblackburn/go/pkg/mod/github.com/apache/[email protected]/table/arrow_utils.go:893
+0x110
github.com/apache/iceberg-go/table.(*writer).writeFile(0x140009fce10,
{0x107b35350, 0x109922ac0}, {{0x7a, 0x10, 0x34, 0xae, 0x69, 0x34, 0x44, ...},
...})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/[email protected]/table/writer.go:65
+0x120
github.com/apache/iceberg-go/table.writeFiles.func3({{0x7a, 0x10, 0x34,
0xae, 0x69, 0x34, 0x44, 0x95, 0x94, 0x94, ...}, ...})
/Users/ethanblackburn/go/pkg/mod/github.com/apache/[email protected]/table/writer.go:126
+0x48
github.com/apache/iceberg-go/table/internal.MapExec[...].func1()
/Users/ethanblackburn/go/pkg/mod/github.com/apache/[email protected]/table/internal/utils.go:501
+0xd0
golang.org/x/sync/errgroup.(*Group).Go.func1()
/Users/ethanblackburn/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:79
+0x54
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
/Users/ethanblackburn/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:76
+0x94
```
## Schema
```
schema:
fields: 34
...
- finding_info: type=struct<analytic: struct<category: utf8, desc: utf8,
name: utf8, related_analytics: list<item: struct<category: utf8, desc: utf8,
name: utf8, type: utf8, type_id: int32, uid: utf8, version: utf8>, nullable>,
type: utf8, type_id: int32, uid: utf8, version: utf8>, uid: utf8>
metadata: ["iceberg.field_id": "17"]
...
```
### Sample data
```
...
finding_info (nullable=false) len=100 nulls=0 type=struct<analytic:
struct<category: utf8, desc: utf8, name: utf8, related_analytics: list<item:
struct<category: utf8, desc: utf8, name: utf8, type: utf8, type_id: int32, uid:
utf8, version: utf8>, nullable>, type: utf8, type_id: int32, uid: utf8,
version: utf8>, uid: utf8>
analytic (nullable=true) len=100 nulls=100 type=struct<category:
utf8, desc: utf8, name: utf8, related_analytics: list<item: struct<category:
utf8, desc: utf8, name: utf8, type: utf8, type_id: int32, uid: utf8, version:
utf8>, nullable>, type: utf8, type_id: int32, uid: utf8, version: utf8>
category (nullable=true) len=100 nulls=100 type=utf8
desc (nullable=true) len=100 nulls=100 type=utf8
name (nullable=true) len=100 nulls=100 type=utf8
related_analytics (nullable=true) len=100 nulls=100 type=list<item:
struct<category: utf8, desc: utf8, name: utf8, type: utf8, type_id: int32, uid:
utf8, version: utf8>, nullable>
related_analytics.element (nullable=true) len=0 nulls=0
type=struct<category: utf8, desc: utf8, name: utf8, type: utf8, type_id: int32,
uid: utf8, version: utf8>
category (nullable=true) len=0 nulls=0 type=utf8
desc (nullable=true) len=0 nulls=0 type=utf8
name (nullable=true) len=0 nulls=0 type=utf8
type (nullable=true) len=0 nulls=0 type=utf8
type_id (nullable=false) len=0 nulls=0 type=int32
uid (nullable=true) len=0 nulls=0 type=utf8
version (nullable=true) len=0 nulls=0 type=utf8
type (nullable=true) len=100 nulls=100 type=utf8
type_id (nullable=false) len=100 nulls=100 type=int32 <--- This
line triggers the panic
uid (nullable=true) len=100 nulls=100 type=utf8
...
```
Iceberg allows nullable structs with required fields, so I assume there's a
bug in the validation. I think its coming from `NewStructArrayWithFields` but
not positive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
