matanper opened a new issue, #1292:
URL: https://github.com/apache/arrow-go/issues/1292
### Describe the bug, including details regarding any error messages,
version, and platform.
Writing a shredded VARIANT column to Parquet fails when a nested list's
parent object is null or missing between populated rows. The input JSON is
valid:
```json
{"obj":{"items":[1]}}
{"obj":null}
{"obj":{"items":[2]}}
```
Observed with Apache Arrow Go v18.7.0 on darwin/arm64. The standalone
program below reproduces the error without Iceberg, AWS, external fixtures, or
application code:
```text
lists with non-zero length null components are not supported
```
Replacing the middle row with `{}` reproduces the same failure in an Iceberg
append regression. Non-null parent objects with lists write successfully.
#### Standalone reproducer
```go
package main
import (
"bytes"
"fmt"
"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/extensions"
"github.com/apache/arrow-go/v18/arrow/memory"
"github.com/apache/arrow-go/v18/parquet"
"github.com/apache/arrow-go/v18/parquet/pqarrow"
)
func main() {
typ := extensions.NewShreddedVariantType(arrow.StructOf(
arrow.Field{Name: "obj", Nullable: true, Type: arrow.StructOf(
arrow.Field{Name: "items", Nullable: true, Type:
arrow.ListOf(arrow.PrimitiveTypes.Int64)},
)},
))
b := extensions.NewVariantBuilder(memory.DefaultAllocator, typ)
defer b.Release()
err :=
b.UnmarshalJSON([]byte(`[{"obj":{"items":[1]}},{"obj":null},{"obj":{"items":[2]}}]`))
if err != nil {
panic(err)
}
values := b.NewArray()
defer values.Release()
schema := arrow.NewSchema([]arrow.Field{{Name: "event", Type: typ,
Nullable: true}}, nil)
rec := array.NewRecordBatch(schema, []arrow.Array{values}, 3)
defer rec.Release()
tbl := array.NewTableFromRecords(schema, []arrow.RecordBatch{rec})
defer tbl.Release()
var output bytes.Buffer
err = pqarrow.WriteTable(tbl, &output, 1024,
parquet.NewWriterProperties(), pqarrow.DefaultWriterProps())
fmt.Println(err)
}
```
Expected: successful Parquet writing with the populated lists and
null/missing parent preserved. Actual: `pqarrow.WriteTable` returns the error
above.
#### Traced cause
`shreddedObjBuilder.AppendMissing` appends a null parent struct and
recursively appends missing children. `shreddedArrayBuilder.AppendMissing`
appends a valid list plus a placeholder child element. The nested list
therefore has a nonempty child range beneath a null ancestor. The Parquet
writer in `parquet/pqarrow/encode_arrow.go` rejects the resulting discontiguous
visited leaf ranges.
A locally tested correction makes the missing-list builder append a null
list without adding child elements. This removes the failing layout;
regressions should cover absent and null parent objects between populated list
rows and verify logical values after read-back. The same `AppendMissing`
implementation is present on main at `6542b630f84dda25715f6238027cc1eed9d114c6`.
This is distinct from the dictionary-offset overflow reported in #1291.
Searched open issues for VARIANT, `shreddedArrayBuilder`, non-zero null
components, and null/list combinations before filing; no matching open report
was found.
### Component(s)
Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
