This is an automated email from the ASF dual-hosted git repository.
zeroshade pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-go.git
The following commit(s) were added to refs/heads/main by this push:
new d5dcde9b fix(arrow): compare LargeListType like ListType in TypeEqual
(#1440)
d5dcde9b is described below
commit d5dcde9bb8cd36aae7a3b73e0152c24977ada9e0
Author: Ondřej Pavela <[email protected]>
AuthorDate: Mon Nov 2 21:23:47 2026 +0100
fix(arrow): compare LargeListType like ListType in TypeEqual (#1440)
### Rationale for this change
`arrow.TypeEqual` has a case for `*ListType` but none for
`*LargeListType`, so large lists fall through to `reflect.DeepEqual`.
That compares the element field's metadata even when `CheckMetadata` is
not given, and it also tells `nil` metadata apart from empty metadata.
Two large list types that print identically can therefore compare
unequal, and `array.Concatenate` rejects their arrays with "arrays to be
concatenated must be identically typed".
We hit this while concatenating batches read with iceberg-go's
`arrow.use_large_types` scan option: a column read from a Parquet file
and the same column filled with nulls for a file that predates it end up
with element fields whose metadata differs only in being `nil` or empty.
### What changes are included in this PR?
A `*LargeListType` case in `TypeEqual`, identical to the `*ListType`
one: compare the element types, compare element metadata only with
`CheckMetadata`, and compare element nullability.
`*LargeListViewType` also has no case and falls through to
`reflect.DeepEqual`; this PR leaves it unchanged.
### Are these changes tested?
Yes. `TestTypeEqual` gains list and large list cases for element
metadata (`nil` vs. empty, differing values with and without
`CheckMetadata`), element nullability, and list vs. large list. The
large list metadata cases fail without the fix.
### Are there any user-facing changes?
`TypeEqual` now reports large list types equal when they differ only in
element field metadata and `CheckMetadata` is not set, as it already
does for list types.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
arrow/compare.go | 8 ++++++++
arrow/compare_test.go | 35 +++++++++++++++++++++++++++++++++++
2 files changed, 43 insertions(+)
diff --git a/arrow/compare.go b/arrow/compare.go
index 997d25ad..52397fcf 100644
--- a/arrow/compare.go
+++ b/arrow/compare.go
@@ -65,6 +65,14 @@ func TypeEqual(left, right DataType, opts
...TypeEqualOption) bool {
return false
}
return l.elem.Nullable == right.(*ListType).elem.Nullable
+ case *LargeListType:
+ if !TypeEqual(l.Elem(), right.(*LargeListType).Elem(), opts...)
{
+ return false
+ }
+ if cfg.metadata &&
!l.elem.Metadata.Equal(right.(*LargeListType).elem.Metadata) {
+ return false
+ }
+ return l.elem.Nullable == right.(*LargeListType).elem.Nullable
case *FixedSizeListType:
if !TypeEqual(l.Elem(), right.(*FixedSizeListType).Elem(),
opts...) {
return false
diff --git a/arrow/compare_test.go b/arrow/compare_test.go
index c78eb457..65b625d5 100644
--- a/arrow/compare_test.go
+++ b/arrow/compare_test.go
@@ -333,6 +333,41 @@ func TestTypeEqual(t *testing.T) {
},
false, true,
},
+ {
+ ListOfField(Field{Name: "item", Type:
BinaryTypes.String, Nullable: true}),
+ ListOfField(Field{Name: "item", Type:
BinaryTypes.String, Nullable: true, Metadata:
MetadataFrom(map[string]string{})}),
+ true, false,
+ },
+ {
+ ListOfField(Field{Name: "item", Type:
BinaryTypes.String, Nullable: true, Metadata:
MetadataFrom(map[string]string{"k1": "v1"})}),
+ ListOfField(Field{Name: "item", Type:
BinaryTypes.String, Nullable: true, Metadata:
MetadataFrom(map[string]string{"k1": "v2"})}),
+ false, true,
+ },
+ {
+ LargeListOfField(Field{Name: "item", Type:
BinaryTypes.String, Nullable: true}),
+ LargeListOfField(Field{Name: "item", Type:
BinaryTypes.String, Nullable: true, Metadata:
MetadataFrom(map[string]string{})}),
+ true, false,
+ },
+ {
+ LargeListOfField(Field{Name: "item", Type:
BinaryTypes.String, Nullable: true, Metadata:
MetadataFrom(map[string]string{"k1": "v1"})}),
+ LargeListOfField(Field{Name: "item", Type:
BinaryTypes.String, Nullable: true, Metadata:
MetadataFrom(map[string]string{"k1": "v2"})}),
+ true, false,
+ },
+ {
+ LargeListOfField(Field{Name: "item", Type:
BinaryTypes.String, Nullable: true, Metadata:
MetadataFrom(map[string]string{"k1": "v1"})}),
+ LargeListOfField(Field{Name: "item", Type:
BinaryTypes.String, Nullable: true, Metadata:
MetadataFrom(map[string]string{"k1": "v2"})}),
+ false, true,
+ },
+ {
+ LargeListOf(BinaryTypes.String),
+ LargeListOfNonNullable(BinaryTypes.String),
+ false, false,
+ },
+ {
+ LargeListOf(BinaryTypes.String),
+ ListOf(BinaryTypes.String),
+ false, false,
+ },
{
MapOf(BinaryTypes.String, PrimitiveTypes.Int32),
MapOf(BinaryTypes.String, PrimitiveTypes.Int32),
