zeroshade opened a new issue, #1255:
URL: https://github.com/apache/arrow-go/issues/1255
### Describe the bug, including details regarding any error messages,
version, and platform.
A `FIXED_LEN_BYTE_ARRAY` column encoded with `BYTE_STREAM_SPLIT` decodes
**values shifted by one position** when the column chunk contains **both nulls
and more than one data page**. The data is silently wrong — no error is
returned.
Found on `main` (`f85b0b25` / `477eb633`), linux/amd64, Go 1.24. This is not
a recent regression; it reproduces on older commits as well. Surfaced while
reviewing #1172, which is unrelated and does not cause it.
**Isolation, already narrowed:**
* Encoding-specific — `PLAIN` passes on byte-identical data and writer
configuration; only `BYTE_STREAM_SPLIT` fails.
* Requires **both** conditions:
* a single large data page (1 MiB) with nulls **passes**
* multiple data pages without nulls **passes**
* multiple data pages *with* nulls **fails**
* Reproduced at widths 2, 3, 4, 7, 8, 16, 17, and 32 — not width-specific.
* First mismatch appears partway through, not at the start (e.g. around
index 259 for width 4).
Example failure at width 4:
```
mismatch at idx 259
expected: []byte{0x0c, 0x0d, 0x0e, 0x0f}
actual : []byte{0x10, 0x11, 0x12, 0x13}
```
`0x10 0x11 0x12 0x13` is the value belonging to index 260, i.e. the stream
is off by exactly one value from that point on.
**Not root-caused.** `ByteStreamSplitFixedLenByteArrayDecoder.SetData`
recomputing `nvals = len(data)/typeLen` and setting `stride` looks correct on
inspection, so the fault is likely elsewhere — either per-page decoder reset in
the column reader or the encoder's per-page flush. I have not determined
whether the file is written incorrectly or read incorrectly.
**Note on existing coverage:** the spaced-decoding unit tests use a single
`SetData` followed by one `DecodeSpaced`, so they exercise only the single-page
case and pass. A fix should come with a multi-page + nulls integration test.
### Reproducer
Drop into `parquet/pqarrow/` as `bss_flba_repro_test.go` and run
`go test ./parquet/pqarrow/ -run TestBSSFLBANullsMultiPage -v`.
```go
package pqarrow_test
import (
"bytes"
"context"
"fmt"
"testing"
"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/memory"
"github.com/apache/arrow-go/v18/parquet"
"github.com/apache/arrow-go/v18/parquet/file"
"github.com/apache/arrow-go/v18/parquet/pqarrow"
"github.com/stretchr/testify/require"
)
func TestBSSFLBANullsMultiPage(t *testing.T) {
encs := map[string]parquet.Encoding{
"BSS": parquet.Encodings.ByteStreamSplit,
"PLAIN": parquet.Encodings.Plain,
}
for encName, enc := range encs {
for _, pageSize := range []int64{512, 1 << 20} { // small pages
vs one big page
for _, width := range []int{4, 32} {
for _, nulls := range []bool{false, true} {
t.Run(fmt.Sprintf("%s/page%d/w%d/nulls=%v", encName, pageSize, width, nulls),
func(t *testing.T) {
mem := memory.DefaultAllocator
const n = 5000
dt :=
&arrow.FixedSizeBinaryType{ByteWidth: width}
sc :=
arrow.NewSchema([]arrow.Field{{Name: "v", Type: dt, Nullable: nulls}}, nil)
bldr :=
array.NewFixedSizeBinaryBuilder(mem, dt)
defer bldr.Release()
want := make([][]byte, 0, n)
for i := 0; i < n; i++ {
if nulls && i%7 == 3 {
bldr.AppendNull()
want =
append(want, nil)
continue
}
v := make([]byte, width)
for j := range v {
v[j] =
byte(i*width + j)
}
bldr.Append(v)
want = append(want, v)
}
arr := bldr.NewArray()
defer arr.Release()
rec := array.NewRecord(sc,
[]arrow.Array{arr}, int64(n))
defer rec.Release()
var buf bytes.Buffer
props :=
parquet.NewWriterProperties(
parquet.WithEncoding(enc),
parquet.WithDictionaryDefault(false),
parquet.WithDataPageSize(pageSize),
parquet.WithBatchSize(128),
)
w, err :=
pqarrow.NewFileWriter(sc, &buf, props, pqarrow.DefaultWriterProps())
require.NoError(t, err)
require.NoError(t, w.Write(rec))
require.NoError(t, w.Close())
rdr, err :=
file.NewParquetReader(bytes.NewReader(buf.Bytes()))
require.NoError(t, err)
defer rdr.Close()
fr, err :=
pqarrow.NewFileReader(rdr, pqarrow.ArrowReadProperties{BatchSize: 137}, mem)
require.NoError(t, err)
tbl, err :=
fr.ReadTable(context.Background())
require.NoError(t, err)
defer tbl.Release()
require.EqualValues(t, n,
tbl.NumRows())
idx := 0
for _, c := range
tbl.Column(0).Data().Chunks() {
fsb :=
c.(*array.FixedSizeBinary)
for i := 0; i <
fsb.Len(); i++ {
if want[idx] ==
nil {
require.True(t, fsb.IsNull(i), "idx %d should be null", idx)
} else {
require.False(t, fsb.IsNull(i), "idx %d should be valid", idx)
require.Equal(t, want[idx], fsb.Value(i), "mismatch at idx %d", idx)
}
idx++
}
}
require.Equal(t, n, idx)
})
}
}
}
}
}
```
Observed result — only the BSS multi-page-with-nulls combinations fail:
```
--- PASS: TestBSSFLBANullsMultiPage/BSS/page512/w4/nulls=false
--- FAIL: TestBSSFLBANullsMultiPage/BSS/page512/w4/nulls=true
--- PASS: TestBSSFLBANullsMultiPage/BSS/page512/w32/nulls=false
--- FAIL: TestBSSFLBANullsMultiPage/BSS/page512/w32/nulls=true
--- PASS: TestBSSFLBANullsMultiPage/BSS/page1048576/w4/nulls=true
--- PASS: TestBSSFLBANullsMultiPage/BSS/page1048576/w32/nulls=true
--- PASS: TestBSSFLBANullsMultiPage/PLAIN/... (all)
```
### Component(s)
Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
