matanper opened a new issue, #1291:
URL: https://github.com/apache/arrow-go/issues/1291
### Describe the bug, including details regarding any error messages,
version, and platform.
`variant.NewMetadata` panics on valid VARIANT metadata containing 126
dictionary keys with two-byte offsets:
```text
panic: runtime error: slice bounds out of range [255:1]
github.com/apache/arrow-go/v18/parquet/variant.(*Metadata).loadDictionary
```
Observed with v18.7.0 and reproduced on main at
`6542b630f84dda25715f6238027cc1eed9d114c6` (Go 1.26.4, darwin/arm64). This
affects reading valid data, not only malformed metadata.
#### Standalone reproducer
```go
package main
import (
"encoding/binary"
"fmt"
"github.com/apache/arrow-go/v18/parquet/variant"
)
func main() {
const count = 126
metadata := []byte{0x41} // version 1, two-byte offsets
metadata = binary.LittleEndian.AppendUint16(metadata, count)
keys := make([]string, count)
var offset uint16
for i := range keys {
keys[i] = fmt.Sprintf("key_%04d", i)
metadata = binary.LittleEndian.AppendUint16(metadata, offset)
offset += uint16(len(keys[i]))
}
metadata = binary.LittleEndian.AppendUint16(metadata, offset)
for _, key := range keys {
metadata = append(metadata, key...)
}
decoded, err := variant.NewMetadata(metadata) // panics
if err != nil {
panic(err)
}
fmt.Println(decoded.DictionarySize()) // expected: 126
}
```
Expected: successful decoding and preservation of all dictionary keys. A
125-key dictionary with the same encoding works; 126 reaches the overflowing
slice boundary. Three- and four-byte offset tables are affected as well.
#### Cause and related work
`offsetPos := hdrSizeBytes + offsetSz` infers `uint8` because `offsetSz` is
`uint8`. Advancing the cursor and computing `offsetPos + offsetSz` can wrap at
byte 256 even though the metadata buffer and offset table are valid. The byte
position needs a wider indexing type than the wire-format offset-width value.
Searched open and closed issues for VARIANT, metadata offsets, dictionary
overflow, `loadDictionary`, and slice-bounds panics before filing. The older
#623 concerns a different array-decoding failure. The validation added in #1063
does not widen this cursor; the reproducer still panics on current main.
Proposed fix and boundary regression tests: #1290.
### Component(s)
Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
