hcrosse opened a new issue, #1483:
URL: https://github.com/apache/arrow-go/issues/1483
### Describe the bug, including details regarding any error messages,
version, and platform.
`array.Validate` and `array.ValidateFull` reject valid extension arrays over
sparse union, dense union or dictionary storage, and `ValidateFull` rejects
extension arrays over null storage. The storage arrays themselves pass.
`Validate` is the check documented for data from untrusted sources such as
Flight servers, so a caller that validates received data rejects these
extension columns.
I reproduced this on main at `66722e30` and on v18.8.0, on macOS arm64 with
Go 1.26.10.
**Reproduction**
Save this as `main.go` in a new subdirectory of an arrow-go checkout at
`66722e30df98ddd6f7bb46023fa13287b84308f6`, then run `go run .` from that
directory:
```go
package main
import (
"fmt"
"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/extensions"
"github.com/apache/arrow-go/v18/arrow/memory"
)
func main() {
mem := memory.DefaultAllocator
storage := array.NewNull(3)
ext :=
array.NewExtensionArrayWithStorage(extensions.NewOpaqueType(arrow.Null, "t",
"v"), storage)
fmt.Println("null storage: storage ValidateFull:",
array.ValidateFull(storage))
fmt.Println("null storage: ext ValidateFull:", array.ValidateFull(ext))
union := arrow.SparseUnionOf([]arrow.Field{{Name: "i", Type:
arrow.PrimitiveTypes.Int32, Nullable: true}}, []arrow.UnionTypeCode{0})
ua := array.MakeArrayOfNull(mem, extensions.NewOpaqueType(union, "t",
"v"), 3)
fmt.Println("union storage: ext Validate:", array.Validate(ua))
dict := &arrow.DictionaryType{IndexType: arrow.PrimitiveTypes.Int8,
ValueType: arrow.BinaryTypes.String}
da := array.MakeArrayOfNull(mem, extensions.NewOpaqueType(dict, "t",
"v"), 3)
fmt.Println("dict storage: ext Validate:", array.Validate(da))
}
```
On main:
```text
null storage: storage ValidateFull: <nil>
null storage: ext ValidateFull: arrow/array: null count value (3) does not
match actual number of nulls in array (0)
union storage: ext Validate: arrow/array: expected at most 1 buffers for
extension<arrow.opaque[storage_type=sparse_union<i: type=int32, nullable=0>,
type_name=t, vendor_name=v]>, got 2
dict storage: ext Validate: arrow/array: datatype
extension<arrow.opaque[storage_type=dictionary<values=utf8, indices=int8,
ordered=false>, type_name=t, vendor_name=v]> declares dictionary storage but is
not a dictionary type
```
Dense union storage fails the same way with `expected at most 2 buffers` and
`got 3`. Extension arrays over dictionary storage built with
`array.MakeFromData`, as the IPC reader builds them, fail with the same
dictionary error.
**Expected behavior**
All four checks should return `<nil>`. `ValidateFull` should still reject a
dictionary index outside its dictionary.
**Cause**
The null count rule, the union buffer offset and the dictionary check look
at the extension type instead of its storage type:
https://github.com/apache/arrow-go/blob/66722e30df98ddd6f7bb46023fa13287b84308f6/arrow/array/validate.go#L140-L152
https://github.com/apache/arrow-go/blob/66722e30df98ddd6f7bb46023fa13287b84308f6/arrow/array/validate.go#L193-L200
https://github.com/apache/arrow-go/blob/66722e30df98ddd6f7bb46023fa13287b84308f6/arrow/array/validate.go#L280-L284
I have a patch with tests and can open a PR.
### Component(s)
Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
