thorfour opened a new issue, #36671:
URL: https://github.com/apache/arrow/issues/36671
### Describe the enhancement requested
The memotable `GetOrInsert` function was implement with the `interface{}` to
make it generic between builders.
https://github.com/apache/arrow/blob/7690409568e8a4b51946f292b109075629ed1ee7/go/internal/hashing/xxh3_memo_table.go#L55
However this empty interface conversion can come with a cost. We've noticed
that when building large binary dictionary arrays that a large portion of CPU
and memory allocations are spent converting the byte slice into `interface{}`
CPU:
Allocs:
I copy/pastad a new memo table that only operates on `[]byte` to prove this
theory; below is the profile with those changes.
CPU:
Allocs:
I see two straightforward ways to potentially address this. Use a `[]byte`
specific memo table for `BinaryDictionaryBuilder` since it's going to be the
only builder that suffers from this type of allocation. Or to replace the
memotables with generics
```go
type MemoTable[T any] interface {
...
GetOrInsert(val T) (idx int, existed bool, err error)
...
}
```
### Component(s)
Go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
