adamyeats opened a new issue, #1364:
URL: https://github.com/apache/arrow-go/issues/1364
### Describe the bug, including details regarding any error messages,
version, and platform.
`ipc.NewMappedFileReader` panics with `assignment to entry in nil map` when
the file schema has a dictionary-encoded field. `ipc.NewFileReader` reads the
same bytes without error.
Repro, as a `go test`:
```go
package arrowdict
import (
"bytes"
"testing"
"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/ipc"
"github.com/apache/arrow-go/v18/arrow/memory"
)
func TestMappedFileReaderDictionary(t *testing.T) {
dt := &arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Int32,
ValueType: arrow.BinaryTypes.String}
b := array.NewDictionaryBuilder(memory.DefaultAllocator,
dt).(*array.BinaryDictionaryBuilder)
defer b.Release()
if err := b.AppendString("a"); err != nil {
t.Fatal(err)
}
arr := b.NewArray()
defer arr.Release()
schema := arrow.NewSchema([]arrow.Field{{Name: "s", Type: dt}}, nil)
rec := array.NewRecordBatch(schema, []arrow.Array{arr}, 1)
defer rec.Release()
var buf bytes.Buffer
fw, err := ipc.NewFileWriter(&buf, ipc.WithSchema(schema))
if err != nil {
t.Fatal(err)
}
if err := fw.Write(rec); err != nil {
t.Fatal(err)
}
if err := fw.Close(); err != nil {
t.Fatal(err)
}
// Works: one record, one row, dictionary<values=utf8, indices=int32,
ordered=false>.
fr, err := ipc.NewFileReader(bytes.NewReader(buf.Bytes()))
if err != nil {
t.Fatal(err)
}
fr.Close()
// Panics: assignment to entry in nil map.
mr, err := ipc.NewMappedFileReader(buf.Bytes())
if err != nil {
t.Fatal(err)
}
mr.Close()
}
```
Output, arrow-go frames only:
```
panic: assignment to entry in nil map [recovered, repanicked]
github.com/apache/arrow-go/v18/arrow/internal/dictutils.(*Mapper).AddField(0x2905ffce83e8,
0x0, {0x2905ffce7518?, 0x1000000b0?, 0x11064e948?})
github.com/apache/arrow-go/[email protected]/arrow/internal/dictutils/dict.go:81
+0x12c
github.com/apache/arrow-go/v18/arrow/ipc.typeFromFB(0x2905ffc98440,
{0x2905ffce79b8, 0x0, 0x1}, {0x1031fbc60, 0x0, 0x0}, 0x2905ffce7828,
0x2905ffce83e8)
github.com/apache/arrow-go/[email protected]/arrow/ipc/metadata.go:653 +0x294
github.com/apache/arrow-go/v18/arrow/ipc.fieldFromFB(0x2905ffc98440,
{0x2905ffce79b8, 0x0, 0x1}, 0x2905ffce83e8)
github.com/apache/arrow-go/[email protected]/arrow/ipc/metadata.go:241 +0x444
github.com/apache/arrow-go/v18/arrow/ipc.schemaFromFB(0x2905ffc98420,
0x2905ffce83e8)
github.com/apache/arrow-go/[email protected]/arrow/ipc/metadata.go:1122 +0xcc
github.com/apache/arrow-go/v18/arrow/ipc.(*FileReader).readSchema(0x2905ffce83c0,
0x1)
github.com/apache/arrow-go/[email protected]/arrow/ipc/file_reader.go:310
+0x44
github.com/apache/arrow-go/v18/arrow/ipc.(*FileReader).init(0x2905ffce83c0,
0x2905ffcd2700)
github.com/apache/arrow-go/[email protected]/arrow/ipc/file_reader.go:288
+0x150
github.com/apache/arrow-go/v18/arrow/ipc.NewMappedFileReader({0x2905ffd0a000,
0x2da, 0x500}, {0x0?, 0x1?, 0x0?})
github.com/apache/arrow-go/[email protected]/arrow/ipc/file_reader.go:244
+0x13c
```
Expected: `NewMappedFileReader` returns a reader or an error, as
`NewFileReader` does.
Actual: the constructor panics before it returns, so the caller has no error
to handle. With a plain `utf8` column instead, the mapped reader works. Index
and value type, row count, nesting, compression and reader options make no
difference. A schema-only file with no batches panics too.
Go 1.27.1, darwin/arm64. Both panic with the same stack:
- v18.8.0, the latest tagged release
- main at b02cc0111095ac8c1659300c2a276354ddd29294 (2026-10-06)
The cause is an unset memo. `NewFileReader` sets [`memo:
dictutils.NewMemo()`](https://github.com/apache/arrow-go/blob/v18.8.0/arrow/ipc/file_reader.go#L261)
in its literal, and
[`NewMemo`](https://github.com/apache/arrow-go/blob/v18.8.0/arrow/internal/dictutils/dict.go#L235-L245)
allocates the four `Memo` maps.
[`NewMappedFileReader`](https://github.com/apache/arrow-go/blob/v18.8.0/arrow/ipc/file_reader.go#L234-L240)
sets only `r` and `mem`, so its memo is the zero value. `init` calls
`readSchema`, which calls [`schemaFromFB(schema,
&f.memo)`](https://github.com/apache/arrow-go/blob/v18.8.0/arrow/ipc/file_reader.go#L310).
For a dictionary field,
[`typeFromFB`](https://github.com/apache/arrow-go/blob/v18.8.0/arrow/ipc/metadata.go#L653)
calls `memo.Mapper.AddField`, which [writes into the nil `pathToID`
map](https://github.com/apache/arrow-go/blob/v18.8.0/arrow/internal/dictutils/dict.go#L81).
These files are identical on
[main](https://github.com/apache/arrow-go/blob/b02cc0111095ac8c1659300c2a276354ddd29294/arrow/ipc/file_reader.go#L234-L240).
#216 added the mapped reader without the memo, so releases from v18.1.0 are
affected.
[`readDictionary`](https://github.com/apache/arrow-go/blob/v18.8.0/arrow/ipc/file_reader.go#L340)
also writes into the memo, so the constructor is the place to fix it.
Suggested fix: add `memo: dictutils.NewMemo(),` to the `FileReader` literal
in `NewMappedFileReader`, matching `NewFileReader`, and add a dictionary column
to `TestMappedReader`.
grafana-plugin-sdk-go hit this when it switched its frame decoder to the
mapped reader, in https://github.com/grafana/grafana-plugin-sdk-go/pull/1768.
### Component(s)
Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
