This is an automated email from the ASF dual-hosted git repository.
JingsongLi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/paimon.git
The following commit(s) were added to refs/heads/master by this push:
new c81dcd5a59 [core] Support array set predicates in multivalue index
(#9294)
c81dcd5a59 is described below
commit c81dcd5a59627e838f209ed8694189319e0e9e5f
Author: Jingsong Lee <[email protected]>
AuthorDate: Tue Aug 18 23:27:18 2026 +0800
[core] Support array set predicates in multivalue index (#9294)
---
docs/docs/multimodal-table/global-index.mdx | 2 +-
.../multimodal-table/global-index/multivalue.mdx | 28 ++++--
docs/docs/primary-key-table/global-index.mdx | 20 +++--
.../apache/paimon/fileindex/FileIndexReader.java | 10 +++
.../globalindex/ConstantGlobalIndexReader.java | 12 +++
.../paimon/globalindex/GlobalIndexReader.java | 12 +++
.../globalindex/OffsetGlobalIndexReader.java | 12 +++
.../paimon/globalindex/SortedFileMetaSelector.java | 12 +++
.../paimon/globalindex/UnionGlobalIndexReader.java | 12 +++
.../bitmap/MultiValueBitmapIndexReader.java | 65 +++++++++++++-
.../apache/paimon/predicate/ArrayContainsAll.java | 99 ++++++++++++++++++++++
.../org/apache/paimon/predicate/ArraysOverlap.java | 99 ++++++++++++++++++++++
.../apache/paimon/predicate/FunctionVisitor.java | 8 ++
.../org/apache/paimon/predicate/LeafFunction.java | 2 +
.../predicate/OnlyPartitionKeyEqualVisitor.java | 10 +++
.../apache/paimon/predicate/PredicateBuilder.java | 31 +++++++
.../globalindex/GlobalIndexEvaluatorTest.java | 58 +++++++++++++
.../bitmap/MultiValueBitmapIndexReaderTest.java | 40 +++++++++
.../apache/paimon/predicate/LeafPredicateTest.java | 30 +++++++
.../paimon/predicate/PredicateBuilderTest.java | 41 +++++++++
.../paimon/predicate/PredicateJsonSerdeTest.java | 20 +++++
.../table/source/PrimaryKeySortedIndexScan.java | 30 +++++++
.../table/MultiValueGlobalIndexTableTest.java | 10 +++
.../source/PrimaryKeySortedIndexScanTest.java | 67 +++++++++++++++
.../orc/filter/OrcPredicateFunctionVisitor.java | 12 +++
.../parquet/filter2/predicate/ParquetFilters.java | 10 +++
26 files changed, 731 insertions(+), 21 deletions(-)
diff --git a/docs/docs/multimodal-table/global-index.mdx
b/docs/docs/multimodal-table/global-index.mdx
index a506a08759..ea0bd8588f 100644
--- a/docs/docs/multimodal-table/global-index.mdx
+++ b/docs/docs/multimodal-table/global-index.mdx
@@ -43,7 +43,7 @@ without full-table scans. Paimon supports multiple global
index types:
|---|---|---|
| BTree | Scalar filters on numeric, string, date, and timestamp columns |
Best when predicates are selective, such as equality, IN, range, and null
checks. |
| Bitmap | Enum-like dimensions and tag columns | Best for equality, IN,
string prefix match, complement predicates, and null checks over compressed
row-id bitmaps. |
-| Multivalue | Membership tests on arrays of supported scalar elements | Best
for `ARRAY_CONTAINS(array_column, element)`. |
+| Multivalue | Membership tests on arrays of supported scalar elements | Best
for `ARRAY_CONTAINS`, `ARRAYS_OVERLAP`, and `ARRAY_CONTAINS_ALL`. |
| Vector | Top-K similarity search on embeddings | Uses ANN algorithms. Tune
build-time and search-time options to balance recall, latency, and index size. |
| Full-Text | Keyword search over text columns | Uses full-text scoring and
tokenizer configuration stored with each index file. |
| Hybrid Search | Combining multiple vector routes, multiple full-text routes,
or vector and full-text retrieval together | Runs multiple scored routes and
merges them with a ranker before reading rows. |
diff --git a/docs/docs/multimodal-table/global-index/multivalue.mdx
b/docs/docs/multimodal-table/global-index/multivalue.mdx
index 484da0dcdf..200900b809 100644
--- a/docs/docs/multimodal-table/global-index/multivalue.mdx
+++ b/docs/docs/multimodal-table/global-index/multivalue.mdx
@@ -25,14 +25,18 @@ under the License.
# Multivalue Index
A Multivalue index maps every distinct non-null element of an `ARRAY` column
to a compressed
-64-bit row-id bitmap. It accelerates element-membership predicates without
treating the complete
-array as one scalar key.
+64-bit row-id bitmap. It accelerates `ARRAY_CONTAINS`, `ARRAYS_OVERLAP`, and
+`ARRAY_CONTAINS_ALL` predicates without treating the complete array as one
scalar key.
For membership lookup, the index has these semantics:
- Duplicate elements in one array contribute the row ID only once.
- Null arrays, empty arrays, and null elements contribute no posting. The
index does not
distinguish those cases and does not accelerate array `IS NULL` or `IS NOT
NULL` predicates.
+- `ARRAYS_OVERLAP` unions the postings for its non-null literal elements.
+ `ARRAY_CONTAINS_ALL` intersects the postings for its distinct literal
elements. A null literal
+ element cannot match; an empty contains-all literal safely falls back
because the index cannot
+ distinguish a null array from a non-null array with no indexed elements.
The indexed column must be an `ARRAY` whose element type is supported by the
global-index key
serializer. A Multivalue index is single-column.
@@ -77,21 +81,27 @@ coverage behavior.
## Query Through Paimon Core
-The Core predicate API exposes array membership directly:
+The Core predicate API exposes single-element, any-element, and all-element
membership directly:
```java
int tagsFieldIndex = table.rowType().getFieldIndex("tags");
-Predicate predicate = new PredicateBuilder(table.rowType())
- .arrayContains(tagsFieldIndex, BinaryString.fromString("blue"));
-
-ReadBuilder readBuilder = table.newReadBuilder().withFilter(predicate);
+PredicateBuilder predicates = new PredicateBuilder(table.rowType());
+Predicate contains = predicates.arrayContains(tagsFieldIndex,
BinaryString.fromString("blue"));
+Predicate overlaps = predicates.arraysOverlap(
+ tagsFieldIndex,
+ Arrays.asList(BinaryString.fromString("blue"),
BinaryString.fromString("green")));
+Predicate containsAll = predicates.arrayContainsAll(
+ tagsFieldIndex,
+ Arrays.asList(BinaryString.fromString("blue"),
BinaryString.fromString("green")));
+
+ReadBuilder readBuilder = table.newReadBuilder().withFilter(overlaps);
TableScan.Plan plan = readBuilder.newScan().plan();
```
Paimon automatically uses matching Multivalue global index files and maps
their row IDs to Data
Evolution splits. SQL engines use the index after their connector translates
an array-membership
-expression to Paimon's `ARRAY_CONTAINS` predicate; that connector translation
is separate from the
-Core index implementation.
+expression to Paimon's `ARRAY_CONTAINS`, `ARRAYS_OVERLAP`, or
`ARRAY_CONTAINS_ALL` predicate; that
+connector translation is separate from the Core index implementation.
## Drop the Index
diff --git a/docs/docs/primary-key-table/global-index.mdx
b/docs/docs/primary-key-table/global-index.mdx
index 2bd2f3886e..00361af4c7 100644
--- a/docs/docs/primary-key-table/global-index.mdx
+++ b/docs/docs/primary-key-table/global-index.mdx
@@ -65,12 +65,12 @@ See [Bitmap Index](../multimodal-table/global-index/bitmap)
for predicate and fo
<TabItem value="multivalue" label="Multivalue">
Use Multivalue for element-membership predicates on an `ARRAY` column. It
stores one compressed
-bitmap posting list per distinct non-null element and accelerates Paimon Core
`ARRAY_CONTAINS`
-predicates. Null arrays, empty arrays, and null elements produce no membership
posting; array null
-checks fall back to the ordinary data path.
+bitmap posting list per distinct non-null element and accelerates Paimon Core
`ARRAY_CONTAINS`,
+`ARRAYS_OVERLAP`, and `ARRAY_CONTAINS_ALL` predicates. Null arrays, empty
arrays, and null elements
+produce no membership posting; array null checks fall back to the ordinary
data path.
The Core predicate and index pushdown are available independently of SQL
connectors. A connector
-uses this index after it translates its array-membership expression to
Paimon's `ARRAY_CONTAINS`
+uses this index after it translates its array-membership expression to the
corresponding Paimon
predicate.
For an append-only or Data Evolution table whose Multivalue index is built
independently, see
@@ -323,19 +323,23 @@ become full-text searchable only after compaction
publishes an eligible data fil
BTree, Bitmap, and Multivalue indexes are applied automatically to
snapshot-scoped Core batch
scans after a supported predicate reaches Paimon. BTree and Bitmap can
accelerate equality, `IN`,
null checks, comparisons and ranges, complement predicates, and supported
string predicates.
-Multivalue accelerates `ARRAY_CONTAINS(array_column, element)`.
+Multivalue accelerates `ARRAY_CONTAINS`, `ARRAYS_OVERLAP`, and
`ARRAY_CONTAINS_ALL` predicates.
The Java/Core predicate can be constructed directly:
```java
Predicate predicate = new PredicateBuilder(rowType)
- .arrayContains(tagsFieldIndex, BinaryString.fromString("blue"));
+ .arraysOverlap(
+ tagsFieldIndex,
+ Arrays.asList(
+ BinaryString.fromString("blue"),
+ BinaryString.fromString("green")));
ReadBuilder readBuilder = table.newReadBuilder().withFilter(predicate);
TableScan.Plan plan = readBuilder.newScan().plan();
```
-SQL engines need a connector-specific expression conversion before their
`array_contains` or
-equivalent function reaches this Core predicate.
+SQL engines need a connector-specific expression conversion before their
array-membership
+function reaches the corresponding Core predicate.
Paimon combines indexed predicate results as follows:
diff --git
a/paimon-common/src/main/java/org/apache/paimon/fileindex/FileIndexReader.java
b/paimon-common/src/main/java/org/apache/paimon/fileindex/FileIndexReader.java
index 1dca420976..cf4e562c8b 100644
---
a/paimon-common/src/main/java/org/apache/paimon/fileindex/FileIndexReader.java
+++
b/paimon-common/src/main/java/org/apache/paimon/fileindex/FileIndexReader.java
@@ -68,6 +68,16 @@ public abstract class FileIndexReader implements
FunctionVisitor<FileIndexResult
return REMAIN;
}
+ @Override
+ public FileIndexResult visitArraysOverlap(FieldRef fieldRef, List<Object>
literals) {
+ return REMAIN;
+ }
+
+ @Override
+ public FileIndexResult visitArrayContainsAll(FieldRef fieldRef,
List<Object> literals) {
+ return REMAIN;
+ }
+
@Override
public FileIndexResult visitLike(FieldRef fieldRef, Object literal) {
return REMAIN;
diff --git
a/paimon-common/src/main/java/org/apache/paimon/globalindex/ConstantGlobalIndexReader.java
b/paimon-common/src/main/java/org/apache/paimon/globalindex/ConstantGlobalIndexReader.java
index dd0c87b9b7..380d4aa5e9 100644
---
a/paimon-common/src/main/java/org/apache/paimon/globalindex/ConstantGlobalIndexReader.java
+++
b/paimon-common/src/main/java/org/apache/paimon/globalindex/ConstantGlobalIndexReader.java
@@ -78,6 +78,18 @@ public class ConstantGlobalIndexReader implements
GlobalIndexReader {
return result;
}
+ @Override
+ public CompletableFuture<Optional<GlobalIndexResult>> visitArraysOverlap(
+ FieldRef fieldRef, List<Object> literals) {
+ return result;
+ }
+
+ @Override
+ public CompletableFuture<Optional<GlobalIndexResult>>
visitArrayContainsAll(
+ FieldRef fieldRef, List<Object> literals) {
+ return result;
+ }
+
@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitLike(
FieldRef fieldRef, Object literal) {
diff --git
a/paimon-common/src/main/java/org/apache/paimon/globalindex/GlobalIndexReader.java
b/paimon-common/src/main/java/org/apache/paimon/globalindex/GlobalIndexReader.java
index 065bd8814e..f1098cf68e 100644
---
a/paimon-common/src/main/java/org/apache/paimon/globalindex/GlobalIndexReader.java
+++
b/paimon-common/src/main/java/org/apache/paimon/globalindex/GlobalIndexReader.java
@@ -47,6 +47,18 @@ public interface GlobalIndexReader
return CompletableFuture.completedFuture(Optional.empty());
}
+ @Override
+ default CompletableFuture<Optional<GlobalIndexResult>> visitArraysOverlap(
+ FieldRef fieldRef, List<Object> literals) {
+ return CompletableFuture.completedFuture(Optional.empty());
+ }
+
+ @Override
+ default CompletableFuture<Optional<GlobalIndexResult>>
visitArrayContainsAll(
+ FieldRef fieldRef, List<Object> literals) {
+ return CompletableFuture.completedFuture(Optional.empty());
+ }
+
@Override
default CompletableFuture<Optional<GlobalIndexResult>> visitBetween(
FieldRef fieldRef, Object from, Object to) {
diff --git
a/paimon-common/src/main/java/org/apache/paimon/globalindex/OffsetGlobalIndexReader.java
b/paimon-common/src/main/java/org/apache/paimon/globalindex/OffsetGlobalIndexReader.java
index 585984dcc0..64be91a8f3 100644
---
a/paimon-common/src/main/java/org/apache/paimon/globalindex/OffsetGlobalIndexReader.java
+++
b/paimon-common/src/main/java/org/apache/paimon/globalindex/OffsetGlobalIndexReader.java
@@ -80,6 +80,18 @@ public class OffsetGlobalIndexReader implements
GlobalIndexReader {
return wrapped.visitArrayContains(fieldRef,
literal).thenApply(this::applyOffset);
}
+ @Override
+ public CompletableFuture<Optional<GlobalIndexResult>> visitArraysOverlap(
+ FieldRef fieldRef, List<Object> literals) {
+ return wrapped.visitArraysOverlap(fieldRef,
literals).thenApply(this::applyOffset);
+ }
+
+ @Override
+ public CompletableFuture<Optional<GlobalIndexResult>>
visitArrayContainsAll(
+ FieldRef fieldRef, List<Object> literals) {
+ return wrapped.visitArrayContainsAll(fieldRef,
literals).thenApply(this::applyOffset);
+ }
+
@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitLike(
FieldRef fieldRef, Object literal) {
diff --git
a/paimon-common/src/main/java/org/apache/paimon/globalindex/SortedFileMetaSelector.java
b/paimon-common/src/main/java/org/apache/paimon/globalindex/SortedFileMetaSelector.java
index b12ade7a2e..6b7e1f01a4 100644
---
a/paimon-common/src/main/java/org/apache/paimon/globalindex/SortedFileMetaSelector.java
+++
b/paimon-common/src/main/java/org/apache/paimon/globalindex/SortedFileMetaSelector.java
@@ -107,6 +107,18 @@ public class SortedFileMetaSelector implements
FunctionVisitor<Optional<List<Glo
return Optional.empty();
}
+ @Override
+ public Optional<List<GlobalIndexIOMeta>> visitArraysOverlap(
+ FieldRef fieldRef, List<Object> literals) {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<List<GlobalIndexIOMeta>> visitArrayContainsAll(
+ FieldRef fieldRef, List<Object> literals) {
+ return Optional.empty();
+ }
+
@Override
public Optional<List<GlobalIndexIOMeta>> visitLike(FieldRef fieldRef,
Object literal) {
return Optional.of(filter(meta -> literal != null &&
!meta.onlyNulls()));
diff --git
a/paimon-common/src/main/java/org/apache/paimon/globalindex/UnionGlobalIndexReader.java
b/paimon-common/src/main/java/org/apache/paimon/globalindex/UnionGlobalIndexReader.java
index c2316b9427..8bee57ebe8 100644
---
a/paimon-common/src/main/java/org/apache/paimon/globalindex/UnionGlobalIndexReader.java
+++
b/paimon-common/src/main/java/org/apache/paimon/globalindex/UnionGlobalIndexReader.java
@@ -83,6 +83,18 @@ public class UnionGlobalIndexReader implements
GlobalIndexReader {
return unionAsync(reader -> reader.visitArrayContains(fieldRef,
literal));
}
+ @Override
+ public CompletableFuture<Optional<GlobalIndexResult>> visitArraysOverlap(
+ FieldRef fieldRef, List<Object> literals) {
+ return unionAsync(reader -> reader.visitArraysOverlap(fieldRef,
literals));
+ }
+
+ @Override
+ public CompletableFuture<Optional<GlobalIndexResult>>
visitArrayContainsAll(
+ FieldRef fieldRef, List<Object> literals) {
+ return unionAsync(reader -> reader.visitArrayContainsAll(fieldRef,
literals));
+ }
+
@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitLike(
FieldRef fieldRef, Object literal) {
diff --git
a/paimon-common/src/main/java/org/apache/paimon/globalindex/bitmap/MultiValueBitmapIndexReader.java
b/paimon-common/src/main/java/org/apache/paimon/globalindex/bitmap/MultiValueBitmapIndexReader.java
index 3305e3afa7..f8856ba221 100644
---
a/paimon-common/src/main/java/org/apache/paimon/globalindex/bitmap/MultiValueBitmapIndexReader.java
+++
b/paimon-common/src/main/java/org/apache/paimon/globalindex/bitmap/MultiValueBitmapIndexReader.java
@@ -28,7 +28,10 @@ import org.apache.paimon.types.ArrayType;
import org.apache.paimon.types.DataType;
import java.io.IOException;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
import java.util.List;
+import java.util.Map;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
@@ -38,6 +41,7 @@ public class MultiValueBitmapIndexReader implements
GlobalIndexReader {
private final DataType elementType;
private final boolean compatibleElementType;
+ private final KeySerializer keySerializer;
private final LazyFilteredBitmapReader bitmapReader;
MultiValueBitmapIndexReader(
@@ -48,6 +52,7 @@ public class MultiValueBitmapIndexReader implements
GlobalIndexReader {
long totalRowCount,
ExecutorService executor) {
this.elementType = elementType;
+ this.keySerializer = keySerializer;
this.compatibleElementType =
files.stream()
.allMatch(
@@ -72,14 +77,62 @@ public class MultiValueBitmapIndexReader implements
GlobalIndexReader {
@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitArrayContains(
FieldRef fieldRef, Object literal) {
- if (!compatibleElementType
- || !(fieldRef.type() instanceof ArrayType)
- || !((ArrayType)
fieldRef.type()).getElementType().equals(elementType)) {
+ if (!supports(fieldRef)) {
return unsupported();
}
return bitmapReader.visitEqual(fieldRef, literal);
}
+ @Override
+ public CompletableFuture<Optional<GlobalIndexResult>> visitArraysOverlap(
+ FieldRef fieldRef, List<Object> literals) {
+ if (!supports(fieldRef)) {
+ return unsupported();
+ }
+ return bitmapReader.visitIn(fieldRef, literals);
+ }
+
+ @Override
+ public CompletableFuture<Optional<GlobalIndexResult>>
visitArrayContainsAll(
+ FieldRef fieldRef, List<Object> literals) {
+ if (!supports(fieldRef) || literals.isEmpty()) {
+ return unsupported();
+ }
+
+ Map<BitmapGlobalIndexFormat.SerializedKey, Object> distinctLiterals =
new LinkedHashMap<>();
+ for (Object literal : literals) {
+ if (literal == null) {
+ return CompletableFuture.completedFuture(
+ Optional.of(GlobalIndexResult.createEmpty()));
+ }
+ distinctLiterals.put(
+
BitmapGlobalIndexFormat.SerializedKey.fromObject(keySerializer, literal),
+ literal);
+ }
+
+ List<CompletableFuture<Optional<GlobalIndexResult>>> futures =
+ new ArrayList<>(distinctLiterals.size());
+ for (Object literal : distinctLiterals.values()) {
+ futures.add(bitmapReader.visitEqual(fieldRef, literal));
+ }
+ return CompletableFuture.allOf(futures.toArray(new
CompletableFuture[0]))
+ .thenApply(
+ ignored -> {
+ Optional<GlobalIndexResult> result =
Optional.empty();
+ for
(CompletableFuture<Optional<GlobalIndexResult>> future : futures) {
+ Optional<GlobalIndexResult> current =
future.join();
+ if (!current.isPresent()) {
+ return Optional.empty();
+ }
+ result =
+ result.isPresent()
+ ?
Optional.of(result.get().and(current.get()))
+ : current;
+ }
+ return result;
+ });
+ }
+
@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitStartsWith(
FieldRef fieldRef, Object literal) {
@@ -172,4 +225,10 @@ public class MultiValueBitmapIndexReader implements
GlobalIndexReader {
private static CompletableFuture<Optional<GlobalIndexResult>>
unsupported() {
return CompletableFuture.completedFuture(Optional.empty());
}
+
+ private boolean supports(FieldRef fieldRef) {
+ return compatibleElementType
+ && fieldRef.type() instanceof ArrayType
+ && ((ArrayType)
fieldRef.type()).getElementType().equals(elementType);
+ }
}
diff --git
a/paimon-common/src/main/java/org/apache/paimon/predicate/ArrayContainsAll.java
b/paimon-common/src/main/java/org/apache/paimon/predicate/ArrayContainsAll.java
new file mode 100644
index 0000000000..bcd598af8f
--- /dev/null
+++
b/paimon-common/src/main/java/org/apache/paimon/predicate/ArrayContainsAll.java
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.predicate;
+
+import org.apache.paimon.types.ArrayType;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.utils.Preconditions;
+
+import
org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonCreator;
+
+import java.util.List;
+import java.util.Optional;
+
+/** A {@link LeafNAryFunction} testing whether an array contains all literal
elements. */
+public class ArrayContainsAll extends LeafNAryFunction {
+
+ public static final String NAME = "ARRAY_CONTAINS_ALL";
+
+ public static final ArrayContainsAll INSTANCE = new ArrayContainsAll();
+
+ @JsonCreator
+ private ArrayContainsAll() {}
+
+ @Override
+ public DataType literalType(DataType fieldType) {
+ return elementType(fieldType);
+ }
+
+ @Override
+ public boolean test(DataType type, Object field, List<Object> literals) {
+ if (field == null) {
+ return false;
+ }
+ for (Object literal : literals) {
+ if (literal == null || !ArrayContains.INSTANCE.test(type, field,
literal)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public boolean test(
+ DataType type,
+ long rowCount,
+ Object min,
+ Object max,
+ Long nullCount,
+ List<Object> literals) {
+ if (nullCount != null && rowCount == nullCount) {
+ return false;
+ }
+ for (Object literal : literals) {
+ if (literal == null) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public Optional<LeafFunction> negate() {
+ return Optional.empty();
+ }
+
+ @Override
+ public <T> T visit(FunctionVisitor<T> visitor, FieldRef fieldRef,
List<Object> literals) {
+ return visitor.visitArrayContainsAll(fieldRef, literals);
+ }
+
+ @Override
+ public String toJson() {
+ return NAME;
+ }
+
+ static DataType elementType(DataType fieldType) {
+ Preconditions.checkArgument(
+ fieldType instanceof ArrayType,
+ "ARRAY_CONTAINS_ALL requires an ARRAY field, but field type is
%s.",
+ fieldType);
+ return ((ArrayType) fieldType).getElementType();
+ }
+}
diff --git
a/paimon-common/src/main/java/org/apache/paimon/predicate/ArraysOverlap.java
b/paimon-common/src/main/java/org/apache/paimon/predicate/ArraysOverlap.java
new file mode 100644
index 0000000000..5ff9070f2d
--- /dev/null
+++ b/paimon-common/src/main/java/org/apache/paimon/predicate/ArraysOverlap.java
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.predicate;
+
+import org.apache.paimon.types.ArrayType;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.utils.Preconditions;
+
+import
org.apache.paimon.shade.jackson2.com.fasterxml.jackson.annotation.JsonCreator;
+
+import java.util.List;
+import java.util.Optional;
+
+/** A {@link LeafNAryFunction} testing whether an array overlaps literal
elements. */
+public class ArraysOverlap extends LeafNAryFunction {
+
+ public static final String NAME = "ARRAYS_OVERLAP";
+
+ public static final ArraysOverlap INSTANCE = new ArraysOverlap();
+
+ @JsonCreator
+ private ArraysOverlap() {}
+
+ @Override
+ public DataType literalType(DataType fieldType) {
+ return elementType(fieldType);
+ }
+
+ @Override
+ public boolean test(DataType type, Object field, List<Object> literals) {
+ if (field == null) {
+ return false;
+ }
+ for (Object literal : literals) {
+ if (literal != null && ArrayContains.INSTANCE.test(type, field,
literal)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ @Override
+ public boolean test(
+ DataType type,
+ long rowCount,
+ Object min,
+ Object max,
+ Long nullCount,
+ List<Object> literals) {
+ if (nullCount != null && rowCount == nullCount) {
+ return false;
+ }
+ for (Object literal : literals) {
+ if (literal != null) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ @Override
+ public Optional<LeafFunction> negate() {
+ return Optional.empty();
+ }
+
+ @Override
+ public <T> T visit(FunctionVisitor<T> visitor, FieldRef fieldRef,
List<Object> literals) {
+ return visitor.visitArraysOverlap(fieldRef, literals);
+ }
+
+ @Override
+ public String toJson() {
+ return NAME;
+ }
+
+ static DataType elementType(DataType fieldType) {
+ Preconditions.checkArgument(
+ fieldType instanceof ArrayType,
+ "ARRAYS_OVERLAP requires an ARRAY field, but field type is
%s.",
+ fieldType);
+ return ((ArrayType) fieldType).getElementType();
+ }
+}
diff --git
a/paimon-common/src/main/java/org/apache/paimon/predicate/FunctionVisitor.java
b/paimon-common/src/main/java/org/apache/paimon/predicate/FunctionVisitor.java
index 6216e1ce2d..6ffa97cee2 100644
---
a/paimon-common/src/main/java/org/apache/paimon/predicate/FunctionVisitor.java
+++
b/paimon-common/src/main/java/org/apache/paimon/predicate/FunctionVisitor.java
@@ -70,6 +70,14 @@ public interface FunctionVisitor<T> extends
PredicateVisitor<T> {
throw new UnsupportedOperationException();
}
+ default T visitArraysOverlap(FieldRef fieldRef, List<Object> literals) {
+ throw new UnsupportedOperationException();
+ }
+
+ default T visitArrayContainsAll(FieldRef fieldRef, List<Object> literals) {
+ throw new UnsupportedOperationException();
+ }
+
T visitLike(FieldRef fieldRef, Object literal);
T visitLessThan(FieldRef fieldRef, Object literal);
diff --git
a/paimon-common/src/main/java/org/apache/paimon/predicate/LeafFunction.java
b/paimon-common/src/main/java/org/apache/paimon/predicate/LeafFunction.java
index b4b50695e0..77326abd32 100644
--- a/paimon-common/src/main/java/org/apache/paimon/predicate/LeafFunction.java
+++ b/paimon-common/src/main/java/org/apache/paimon/predicate/LeafFunction.java
@@ -64,6 +64,8 @@ public abstract class LeafFunction implements Serializable {
registry.put(EndsWith.NAME, EndsWith.INSTANCE);
registry.put(Contains.NAME, Contains.INSTANCE);
registry.put(ArrayContains.NAME, ArrayContains.INSTANCE);
+ registry.put(ArraysOverlap.NAME, ArraysOverlap.INSTANCE);
+ registry.put(ArrayContainsAll.NAME, ArrayContainsAll.INSTANCE);
registry.put(Like.NAME, Like.INSTANCE);
registry.put(In.NAME, In.INSTANCE);
registry.put(NotIn.NAME, NotIn.INSTANCE);
diff --git
a/paimon-common/src/main/java/org/apache/paimon/predicate/OnlyPartitionKeyEqualVisitor.java
b/paimon-common/src/main/java/org/apache/paimon/predicate/OnlyPartitionKeyEqualVisitor.java
index 35f482924d..fb6e6e1c85 100644
---
a/paimon-common/src/main/java/org/apache/paimon/predicate/OnlyPartitionKeyEqualVisitor.java
+++
b/paimon-common/src/main/java/org/apache/paimon/predicate/OnlyPartitionKeyEqualVisitor.java
@@ -72,6 +72,16 @@ public class OnlyPartitionKeyEqualVisitor implements
FunctionVisitor<Boolean> {
return false;
}
+ @Override
+ public Boolean visitArraysOverlap(FieldRef fieldRef, List<Object>
literals) {
+ return false;
+ }
+
+ @Override
+ public Boolean visitArrayContainsAll(FieldRef fieldRef, List<Object>
literals) {
+ return false;
+ }
+
@Override
public Boolean visitLike(FieldRef fieldRef, Object literal) {
return false;
diff --git
a/paimon-common/src/main/java/org/apache/paimon/predicate/PredicateBuilder.java
b/paimon-common/src/main/java/org/apache/paimon/predicate/PredicateBuilder.java
index ed80a4f767..04e813cdcf 100644
---
a/paimon-common/src/main/java/org/apache/paimon/predicate/PredicateBuilder.java
+++
b/paimon-common/src/main/java/org/apache/paimon/predicate/PredicateBuilder.java
@@ -181,6 +181,28 @@ public class PredicateBuilder {
return leaf(ArrayContains.INSTANCE, transform, elementLiteral);
}
+ public Predicate arraysOverlap(int idx, List<?> elementLiterals) {
+ DataField field = rowType.getFields().get(idx);
+ ArraysOverlap.elementType(field.type());
+ return leaf(ArraysOverlap.INSTANCE, idx, new
ArrayList<>(elementLiterals));
+ }
+
+ public Predicate arraysOverlap(Transform transform, List<?>
elementLiterals) {
+ ArraysOverlap.elementType(transform.outputType());
+ return leaf(ArraysOverlap.INSTANCE, transform, new
ArrayList<>(elementLiterals));
+ }
+
+ public Predicate arrayContainsAll(int idx, List<?> elementLiterals) {
+ DataField field = rowType.getFields().get(idx);
+ ArrayContainsAll.elementType(field.type());
+ return leaf(ArrayContainsAll.INSTANCE, idx, new
ArrayList<>(elementLiterals));
+ }
+
+ public Predicate arrayContainsAll(Transform transform, List<?>
elementLiterals) {
+ ArrayContainsAll.elementType(transform.outputType());
+ return leaf(ArrayContainsAll.INSTANCE, transform, new
ArrayList<>(elementLiterals));
+ }
+
public Predicate like(int idx, Object patternLiteral) {
Pair<LeafBinaryFunction, Object> optimized =
LikeOptimization.tryOptimize(patternLiteral)
@@ -204,6 +226,15 @@ public class PredicateBuilder {
return LeafPredicate.of(transform, function, singletonList(literal));
}
+ private Predicate leaf(LeafFunction function, int idx, List<Object>
literals) {
+ DataField field = rowType.getFields().get(idx);
+ return new LeafPredicate(function, field.type(), idx, field.name(),
literals);
+ }
+
+ private Predicate leaf(LeafFunction function, Transform transform,
List<Object> literals) {
+ return LeafPredicate.of(transform, function, literals);
+ }
+
private Predicate leaf(LeafUnaryFunction function, int idx) {
DataField field = rowType.getFields().get(idx);
return new LeafPredicate(
diff --git
a/paimon-common/src/test/java/org/apache/paimon/globalindex/GlobalIndexEvaluatorTest.java
b/paimon-common/src/test/java/org/apache/paimon/globalindex/GlobalIndexEvaluatorTest.java
index 8245da25f0..5bb156393d 100644
---
a/paimon-common/src/test/java/org/apache/paimon/globalindex/GlobalIndexEvaluatorTest.java
+++
b/paimon-common/src/test/java/org/apache/paimon/globalindex/GlobalIndexEvaluatorTest.java
@@ -800,6 +800,18 @@ class GlobalIndexEvaluatorTest {
return CompletableFuture.completedFuture(
Optional.of(GlobalIndexResult.createEmpty()));
}
+
+ @Override
+ public CompletableFuture<Optional<GlobalIndexResult>>
visitArraysOverlap(
+ FieldRef fieldRef, List<Object> literals) {
+ return
CompletableFuture.completedFuture(Optional.of(resultOf(1, 2, 3)));
+ }
+
+ @Override
+ public CompletableFuture<Optional<GlobalIndexResult>>
visitArrayContainsAll(
+ FieldRef fieldRef, List<Object> literals) {
+ return
CompletableFuture.completedFuture(Optional.of(resultOf(2)));
+ }
};
GlobalIndexEvaluator evaluator =
new GlobalIndexEvaluator(
@@ -814,11 +826,19 @@ class GlobalIndexEvaluatorTest {
evaluator.evaluate(
PredicateBuilder.and(
builder.arrayContains(0, 1),
builder.arrayContains(0, 2)));
+ Optional<GlobalIndexResult> overlap =
+ evaluator.evaluate(builder.arraysOverlap(0, Arrays.asList(1,
2)));
+ Optional<GlobalIndexResult> containsAll =
+ evaluator.evaluate(builder.arrayContainsAll(0,
Arrays.asList(1, 2)));
assertThat(any).isPresent();
assertBitmapContainsExactly(any.get().results(), 1L, 2L, 3L);
assertThat(all).isPresent();
assertBitmapContainsExactly(all.get().results(), 2L);
+ assertThat(overlap).isPresent();
+ assertBitmapContainsExactly(overlap.get().results(), 1L, 2L, 3L);
+ assertThat(containsAll).isPresent();
+ assertBitmapContainsExactly(containsAll.get().results(), 2L);
evaluator.close();
}
@@ -844,6 +864,18 @@ class GlobalIndexEvaluatorTest {
FieldRef fieldRef, Object literal) {
return
CompletableFuture.completedFuture(Optional.of(resultOf(1, 3)));
}
+
+ @Override
+ public CompletableFuture<Optional<GlobalIndexResult>>
visitArraysOverlap(
+ FieldRef fieldRef, List<Object> literals) {
+ return
CompletableFuture.completedFuture(Optional.of(resultOf(1, 3)));
+ }
+
+ @Override
+ public CompletableFuture<Optional<GlobalIndexResult>>
visitArrayContainsAll(
+ FieldRef fieldRef, List<Object> literals) {
+ return
CompletableFuture.completedFuture(Optional.of(resultOf(1, 3)));
+ }
};
GlobalIndexReader wrapped =
new UnionGlobalIndexReader(
@@ -855,6 +887,16 @@ class GlobalIndexEvaluatorTest {
assertThat(result).isPresent();
assertBitmapContainsExactly(result.get().results(), 11L, 13L);
+ Optional<GlobalIndexResult> overlapResult =
+ evaluator.evaluate(
+ new PredicateBuilder(rowType).arraysOverlap(0,
Arrays.asList(2)));
+ Optional<GlobalIndexResult> containsAllResult =
+ evaluator.evaluate(
+ new PredicateBuilder(rowType).arrayContainsAll(0,
Arrays.asList(2)));
+ assertThat(overlapResult).isPresent();
+ assertBitmapContainsExactly(overlapResult.get().results(), 11L, 13L);
+ assertThat(containsAllResult).isPresent();
+ assertBitmapContainsExactly(containsAllResult.get().results(), 11L,
13L);
Optional<GlobalIndexResult> constantResult =
new ConstantGlobalIndexReader(resultOf(4))
.visitArrayContains(
@@ -862,6 +904,22 @@ class GlobalIndexEvaluatorTest {
.join();
assertThat(constantResult).isPresent();
assertBitmapContainsExactly(constantResult.get().results(), 4L);
+ ConstantGlobalIndexReader constantReader = new
ConstantGlobalIndexReader(resultOf(5));
+ FieldRef fieldRef = new FieldRef(0, "tags",
DataTypes.ARRAY(DataTypes.INT()));
+ assertBitmapContainsExactly(
+ constantReader
+ .visitArraysOverlap(fieldRef, Arrays.asList(2))
+ .join()
+ .get()
+ .results(),
+ 5L);
+ assertBitmapContainsExactly(
+ constantReader
+ .visitArrayContainsAll(fieldRef, Arrays.asList(2))
+ .join()
+ .get()
+ .results(),
+ 5L);
evaluator.close();
}
diff --git
a/paimon-common/src/test/java/org/apache/paimon/globalindex/bitmap/MultiValueBitmapIndexReaderTest.java
b/paimon-common/src/test/java/org/apache/paimon/globalindex/bitmap/MultiValueBitmapIndexReaderTest.java
index a268c549f8..118d1d1f7c 100644
---
a/paimon-common/src/test/java/org/apache/paimon/globalindex/bitmap/MultiValueBitmapIndexReaderTest.java
+++
b/paimon-common/src/test/java/org/apache/paimon/globalindex/bitmap/MultiValueBitmapIndexReaderTest.java
@@ -128,6 +128,32 @@ class MultiValueBitmapIndexReaderTest {
assertRows(reader.visitArrayContains(fieldRef, str("B")).join(),
0L, 3L);
assertRows(reader.visitArrayContains(fieldRef,
str("missing")).join());
assertRows(reader.visitArrayContains(fieldRef, null).join());
+ assertRows(
+ reader.visitArraysOverlap(
+ fieldRef,
+ java.util.Arrays.asList(
+ str("missing"), null, str("B"),
str("B")))
+ .join(),
+ 0L,
+ 3L);
+ assertRows(
+ reader.visitArraysOverlap(
+ fieldRef,
java.util.Arrays.asList(str("missing"), null))
+ .join());
+ assertRows(
+ reader.visitArrayContainsAll(
+ fieldRef,
java.util.Arrays.asList(str("A"), str("B"), str("A")))
+ .join(),
+ 0L);
+ assertRows(
+ reader.visitArrayContainsAll(
+ fieldRef,
java.util.Arrays.asList(str("A"), str("C")))
+ .join());
+ assertRows(
+ reader.visitArrayContainsAll(fieldRef,
java.util.Arrays.asList(str("A"), null))
+ .join());
+ assertThat(reader.visitArrayContainsAll(fieldRef,
Collections.emptyList()).join())
+ .isEmpty();
assertThat(
reader.visitArrayContains(
new FieldRef(
@@ -135,6 +161,20 @@ class MultiValueBitmapIndexReaderTest {
1L)
.join())
.isEmpty();
+ assertThat(
+ reader.visitArraysOverlap(
+ new FieldRef(
+ 1, "tags",
DataTypes.ARRAY(DataTypes.BIGINT())),
+ Collections.singletonList(1L))
+ .join())
+ .isEmpty();
+ assertThat(
+ reader.visitArrayContainsAll(
+ new FieldRef(
+ 1, "tags",
DataTypes.ARRAY(DataTypes.BIGINT())),
+ Collections.singletonList(1L))
+ .join())
+ .isEmpty();
assertThat(reader.visitIsNull(fieldRef).join()).isEmpty();
assertThat(reader.visitIsNotNull(fieldRef).join()).isEmpty();
diff --git
a/paimon-common/src/test/java/org/apache/paimon/predicate/LeafPredicateTest.java
b/paimon-common/src/test/java/org/apache/paimon/predicate/LeafPredicateTest.java
index 7fac8ddb88..a179753b0f 100644
---
a/paimon-common/src/test/java/org/apache/paimon/predicate/LeafPredicateTest.java
+++
b/paimon-common/src/test/java/org/apache/paimon/predicate/LeafPredicateTest.java
@@ -93,6 +93,36 @@ class LeafPredicateTest {
.isTrue();
}
+ @Test
+ public void testArraySetPredicateSerializationUsesElementSerializer()
+ throws IOException, ClassNotFoundException {
+ PredicateBuilder builder =
+ new
PredicateBuilder(RowType.of(DataTypes.ARRAY(DataTypes.STRING())));
+ LeafPredicate overlap =
+ (LeafPredicate)
+ builder.arraysOverlap(
+ 0,
+ java.util.Arrays.asList(
+ BinaryString.fromString("trial"),
+ BinaryString.fromString("vip")));
+ LeafPredicate containsAll =
+ (LeafPredicate)
+ builder.arrayContainsAll(
+ 0,
+ java.util.Arrays.asList(
+ BinaryString.fromString("trial"),
+ BinaryString.fromString("vip")));
+
+ GenericRow row =
+ GenericRow.of(
+ new GenericArray(
+ new BinaryString[] {
+ BinaryString.fromString("trial"),
BinaryString.fromString("vip")
+ }));
+ assertThat(InstantiationUtil.clone(overlap).test(row)).isTrue();
+ assertThat(InstantiationUtil.clone(containsAll).test(row)).isTrue();
+ }
+
private LeafPredicate create() {
List<Object> inputs = new ArrayList<>();
inputs.add(new FieldRef(0, "f0", DataTypes.STRING()));
diff --git
a/paimon-common/src/test/java/org/apache/paimon/predicate/PredicateBuilderTest.java
b/paimon-common/src/test/java/org/apache/paimon/predicate/PredicateBuilderTest.java
index b3f1c616b8..92250b8746 100644
---
a/paimon-common/src/test/java/org/apache/paimon/predicate/PredicateBuilderTest.java
+++
b/paimon-common/src/test/java/org/apache/paimon/predicate/PredicateBuilderTest.java
@@ -250,6 +250,41 @@ public class PredicateBuilderTest {
assertThat(containsTwo.negate()).isEmpty();
}
+ @Test
+ public void testArraysOverlap() {
+ PredicateBuilder builder =
+ new
PredicateBuilder(RowType.of(DataTypes.ARRAY(DataTypes.INT())));
+ Predicate overlap = builder.arraysOverlap(0, Arrays.asList(9, null, 2,
2));
+ Predicate noOverlap = builder.arraysOverlap(0, Arrays.asList(9, null));
+ Predicate empty = builder.arraysOverlap(0, new ArrayList<>());
+
+ GenericRow row = GenericRow.of(new GenericArray(new Integer[] {1,
null, 2, 2}));
+ assertThat(overlap.test(row)).isTrue();
+ assertThat(noOverlap.test(row)).isFalse();
+ assertThat(empty.test(row)).isFalse();
+ assertThat(overlap.test(GenericRow.of((Object) null))).isFalse();
+ assertThat(overlap.negate()).isEmpty();
+ }
+
+ @Test
+ public void testArrayContainsAll() {
+ PredicateBuilder builder =
+ new
PredicateBuilder(RowType.of(DataTypes.ARRAY(DataTypes.INT())));
+ Predicate containsAll = builder.arrayContainsAll(0, Arrays.asList(2,
1, 2));
+ Predicate missing = builder.arrayContainsAll(0, Arrays.asList(1, 3));
+ Predicate containsNull = builder.arrayContainsAll(0, Arrays.asList(1,
null));
+ Predicate empty = builder.arrayContainsAll(0, new ArrayList<>());
+
+ GenericRow row = GenericRow.of(new GenericArray(new Integer[] {1,
null, 2, 2}));
+ assertThat(containsAll.test(row)).isTrue();
+ assertThat(missing.test(row)).isFalse();
+ assertThat(containsNull.test(row)).isFalse();
+ assertThat(empty.test(row)).isTrue();
+ assertThat(empty.test(GenericRow.of(new GenericArray(new
Integer[0])))).isTrue();
+ assertThat(empty.test(GenericRow.of((Object) null))).isFalse();
+ assertThat(containsAll.negate()).isEmpty();
+ }
+
@Test
public void testArrayContainsRequiresArrayField() {
PredicateBuilder builder = new
PredicateBuilder(RowType.of(DataTypes.INT()));
@@ -257,6 +292,12 @@ public class PredicateBuilderTest {
assertThatThrownBy(() -> builder.arrayContains(0, 1))
.isInstanceOf(IllegalArgumentException.class)
.hasMessageContaining("ARRAY_CONTAINS requires an ARRAY
field");
+ assertThatThrownBy(() -> builder.arraysOverlap(0, Arrays.asList(1, 2)))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessageContaining("ARRAYS_OVERLAP requires an ARRAY
field");
+ assertThatThrownBy(() -> builder.arrayContainsAll(0, Arrays.asList(1,
2)))
+ .isInstanceOf(IllegalArgumentException.class)
+ .hasMessageContaining("ARRAY_CONTAINS_ALL requires an ARRAY
field");
}
// ---- or()/and() binary tree structure tests ----
diff --git
a/paimon-common/src/test/java/org/apache/paimon/predicate/PredicateJsonSerdeTest.java
b/paimon-common/src/test/java/org/apache/paimon/predicate/PredicateJsonSerdeTest.java
index e3814736c5..362f34f917 100644
---
a/paimon-common/src/test/java/org/apache/paimon/predicate/PredicateJsonSerdeTest.java
+++
b/paimon-common/src/test/java/org/apache/paimon/predicate/PredicateJsonSerdeTest.java
@@ -176,6 +176,26 @@ class PredicateJsonSerdeTest {
.expectJson(
"{\"kind\":\"LEAF\",\"transform\":{\"name\":\"FIELD_REF\",\"fieldRef\":{\"index\":4,\"name\":\"f4\",\"type\":{\"type\":\"ARRAY\",\"element\":\"STRING\"}}},\"function\":\"ARRAY_CONTAINS\",\"literals\":[\"vip\"]}"),
+ // LeafPredicate - ArraysOverlap uses the element type for
literal serde
+ TestSpec.forPredicate(
+ builder.arraysOverlap(
+ 4,
+ Arrays.asList(
+ BinaryString.fromString("vip"),
+
BinaryString.fromString("trial"))))
+ .expectJson(
+
"{\"kind\":\"LEAF\",\"transform\":{\"name\":\"FIELD_REF\",\"fieldRef\":{\"index\":4,\"name\":\"f4\",\"type\":{\"type\":\"ARRAY\",\"element\":\"STRING\"}}},\"function\":\"ARRAYS_OVERLAP\",\"literals\":[\"vip\",\"trial\"]}"),
+
+ // LeafPredicate - ArrayContainsAll uses the element type for
literal serde
+ TestSpec.forPredicate(
+ builder.arrayContainsAll(
+ 4,
+ Arrays.asList(
+ BinaryString.fromString("vip"),
+
BinaryString.fromString("trial"))))
+ .expectJson(
+
"{\"kind\":\"LEAF\",\"transform\":{\"name\":\"FIELD_REF\",\"fieldRef\":{\"index\":4,\"name\":\"f4\",\"type\":{\"type\":\"ARRAY\",\"element\":\"STRING\"}}},\"function\":\"ARRAY_CONTAINS_ALL\",\"literals\":[\"vip\",\"trial\"]}"),
+
// LeafPredicate - Between
TestSpec.forPredicate(builder.between(0, 3, 7))
.expectJson(
diff --git
a/paimon-core/src/main/java/org/apache/paimon/table/source/PrimaryKeySortedIndexScan.java
b/paimon-core/src/main/java/org/apache/paimon/table/source/PrimaryKeySortedIndexScan.java
index d919cb85ac..cfe3fcea64 100644
---
a/paimon-core/src/main/java/org/apache/paimon/table/source/PrimaryKeySortedIndexScan.java
+++
b/paimon-core/src/main/java/org/apache/paimon/table/source/PrimaryKeySortedIndexScan.java
@@ -457,6 +457,22 @@ public final class PrimaryKeySortedIndexScan {
() -> reader().visitArrayContains(fieldRef, literal));
}
+ @Override
+ public CompletableFuture<Optional<GlobalIndexResult>>
visitArraysOverlap(
+ FieldRef fieldRef, List<Object> literals) {
+ return query(
+ QueryKey.ofLiterals(QueryOperation.ARRAYS_OVERLAP,
fieldRef, literals),
+ () -> reader().visitArraysOverlap(fieldRef, literals));
+ }
+
+ @Override
+ public CompletableFuture<Optional<GlobalIndexResult>>
visitArrayContainsAll(
+ FieldRef fieldRef, List<Object> literals) {
+ return query(
+ QueryKey.ofLiterals(QueryOperation.ARRAY_CONTAINS_ALL,
fieldRef, literals),
+ () -> reader().visitArrayContainsAll(fieldRef, literals));
+ }
+
@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitLike(
FieldRef fieldRef, Object literal) {
@@ -639,6 +655,8 @@ public final class PrimaryKeySortedIndexScan {
ENDS_WITH,
CONTAINS,
ARRAY_CONTAINS,
+ ARRAYS_OVERLAP,
+ ARRAY_CONTAINS_ALL,
LIKE,
LESS_THAN,
GREATER_OR_EQUAL,
@@ -739,6 +757,18 @@ public final class PrimaryKeySortedIndexScan {
return localize(wrapped.visitArrayContains(fieldRef, literal));
}
+ @Override
+ public CompletableFuture<Optional<GlobalIndexResult>>
visitArraysOverlap(
+ FieldRef fieldRef, List<Object> literals) {
+ return localize(wrapped.visitArraysOverlap(fieldRef, literals));
+ }
+
+ @Override
+ public CompletableFuture<Optional<GlobalIndexResult>>
visitArrayContainsAll(
+ FieldRef fieldRef, List<Object> literals) {
+ return localize(wrapped.visitArrayContainsAll(fieldRef, literals));
+ }
+
@Override
public CompletableFuture<Optional<GlobalIndexResult>> visitLike(
FieldRef fieldRef, Object literal) {
diff --git
a/paimon-core/src/test/java/org/apache/paimon/table/MultiValueGlobalIndexTableTest.java
b/paimon-core/src/test/java/org/apache/paimon/table/MultiValueGlobalIndexTableTest.java
index ed52c8c727..90d440a98c 100644
---
a/paimon-core/src/test/java/org/apache/paimon/table/MultiValueGlobalIndexTableTest.java
+++
b/paimon-core/src/test/java/org/apache/paimon/table/MultiValueGlobalIndexTableTest.java
@@ -41,6 +41,7 @@ import org.apache.paimon.types.DataTypes;
import org.junit.jupiter.api.Test;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.Collections;
import java.util.List;
@@ -82,6 +83,15 @@ public class MultiValueGlobalIndexTableTest extends
TableTestBase {
PredicateBuilder builder = new PredicateBuilder(table.rowType());
Predicate containsRed = builder.arrayContains(1, RED);
assertThat(readIds(table, containsRed)).containsExactlyInAnyOrder(1,
5);
+ assertThat(readIds(table, builder.arraysOverlap(1, Arrays.asList(BLUE,
9))))
+ .containsExactlyInAnyOrder(1, 2);
+ assertThat(readIds(table, builder.arrayContainsAll(1,
Arrays.asList(RED, BLUE))))
+ .containsExactly(1);
+ assertThat(readIds(table, builder.arrayContainsAll(1,
Arrays.asList(RED, RED))))
+ .containsExactlyInAnyOrder(1, 5);
+ assertThat(readIds(table, builder.arrayContainsAll(1,
Arrays.asList(RED, null)))).isEmpty();
+ assertThat(readIdsWithFallback(table, builder.arrayContainsAll(1,
Collections.emptyList())))
+ .containsExactlyInAnyOrder(1, 2, 4, 5);
assertThat(readIdsWithFallback(table,
builder.isNull(1))).containsExactly(3);
write(table, GenericRow.of(6, array(RED)));
diff --git
a/paimon-core/src/test/java/org/apache/paimon/table/source/PrimaryKeySortedIndexScanTest.java
b/paimon-core/src/test/java/org/apache/paimon/table/source/PrimaryKeySortedIndexScanTest.java
index d4b6d73ed9..c9c09ea7d4 100644
---
a/paimon-core/src/test/java/org/apache/paimon/table/source/PrimaryKeySortedIndexScanTest.java
+++
b/paimon-core/src/test/java/org/apache/paimon/table/source/PrimaryKeySortedIndexScanTest.java
@@ -420,6 +420,73 @@ class PrimaryKeySortedIndexScanTest {
verify(reader).close();
}
+ @Test
+ void testArraySetPredicatesAreCachedAndLocalized() throws IOException {
+ DataFileMeta first = dataFile("data-1", 2);
+ DataFileMeta second = dataFile("data-2", 3);
+ DataSplit split = dataSplit(11, 0, first, second);
+ PrimaryKeyIndexDefinition definition =
+ definition(
+ 7,
+ MultiValueGlobalIndexerFactory.IDENTIFIER,
+ PrimaryKeyIndexDefinition.Family.MULTI_VALUE);
+ IndexFileMeta mergedPayload =
+ payload(
+ "multivalue-merged",
+ Arrays.asList(
+ new PrimaryKeyIndexSourceFile("data-1", 2),
+ new PrimaryKeyIndexSourceFile("data-2", 3)),
+ "multivalue",
+ 7,
+ 5);
+ PrimaryKeySortedIndexScan.Plan plan =
+ PrimaryKeySortedIndexScan.plan(
+ 11,
+ Collections.singletonList(split),
+ Collections.singletonList(definition),
+ Collections.singletonList(payloadEntry(0,
mergedPayload)));
+ RowType rowType = RowType.of(new DataField(7, "tags",
DataTypes.ARRAY(DataTypes.INT())));
+ List<Object> literals = Arrays.asList(42, 43);
+ PredicateBuilder builder = new PredicateBuilder(rowType);
+ Predicate predicate =
+ PredicateBuilder.and(
+ builder.arraysOverlap(0, literals),
builder.arrayContainsAll(0, literals));
+ AtomicInteger queries = new AtomicInteger();
+ GlobalIndexReader reader = mock(GlobalIndexReader.class);
+ when(reader.visitArraysOverlap(any(), eq(literals)))
+ .thenAnswer(
+ ignored -> {
+ queries.incrementAndGet();
+ return completedResult(1, 2, 4);
+ });
+ when(reader.visitArrayContainsAll(any(), eq(literals)))
+ .thenAnswer(
+ ignored -> {
+ queries.incrementAndGet();
+ return completedResult(1, 4);
+ });
+
+ PrimaryKeySortedIndexScan.EvaluatedPlan evaluated =
+ PrimaryKeySortedIndexScan.evaluate(
+ plan,
+ rowType,
+ predicate,
+ Collections.singletonList(definition),
+ (ignoredFile, ignoredDefinition, payloads,
totalRowCount) -> {
+
assertThat(payloads).containsExactly(mergedPayload);
+ assertThat(totalRowCount).isEqualTo(5);
+ return reader;
+ });
+
+ assertThat(queries).hasValue(2);
+ assertThat(evaluated.files()).hasSize(2);
+ assertThat(evaluated.files().get(0).result()).isPresent();
+
assertThat(evaluated.files().get(0).result().get().results()).containsExactly(1L);
+ assertThat(evaluated.files().get(1).result()).isPresent();
+
assertThat(evaluated.files().get(1).result().get().results()).containsExactly(2L);
+ verify(reader).close();
+ }
+
@Test
void testPerFileBooleanFallbackSemantics() {
DataSplit split = dataSplit(11, 0, dataFile("data-1", 4));
diff --git
a/paimon-format/src/main/java/org/apache/paimon/format/orc/filter/OrcPredicateFunctionVisitor.java
b/paimon-format/src/main/java/org/apache/paimon/format/orc/filter/OrcPredicateFunctionVisitor.java
index b67364d33e..f72478c8a1 100644
---
a/paimon-format/src/main/java/org/apache/paimon/format/orc/filter/OrcPredicateFunctionVisitor.java
+++
b/paimon-format/src/main/java/org/apache/paimon/format/orc/filter/OrcPredicateFunctionVisitor.java
@@ -87,6 +87,18 @@ public class OrcPredicateFunctionVisitor
return Optional.empty();
}
+ @Override
+ public Optional<OrcFilters.Predicate> visitArraysOverlap(
+ FieldRef fieldRef, List<Object> literals) {
+ return Optional.empty();
+ }
+
+ @Override
+ public Optional<OrcFilters.Predicate> visitArrayContainsAll(
+ FieldRef fieldRef, List<Object> literals) {
+ return Optional.empty();
+ }
+
@Override
public Optional<OrcFilters.Predicate> visitLike(FieldRef fieldRef, Object
literal) {
return Optional.empty();
diff --git
a/paimon-format/src/main/java/org/apache/parquet/filter2/predicate/ParquetFilters.java
b/paimon-format/src/main/java/org/apache/parquet/filter2/predicate/ParquetFilters.java
index 9afb6f8981..e28b8ee437 100644
---
a/paimon-format/src/main/java/org/apache/parquet/filter2/predicate/ParquetFilters.java
+++
b/paimon-format/src/main/java/org/apache/parquet/filter2/predicate/ParquetFilters.java
@@ -204,6 +204,16 @@ public class ParquetFilters {
throw new UnsupportedOperationException();
}
+ @Override
+ public FilterPredicate visitArraysOverlap(FieldRef fieldRef,
List<Object> literals) {
+ throw new UnsupportedOperationException();
+ }
+
+ @Override
+ public FilterPredicate visitArrayContainsAll(FieldRef fieldRef,
List<Object> literals) {
+ throw new UnsupportedOperationException();
+ }
+
@Override
public FilterPredicate visitLike(FieldRef fieldRef, Object literal) {
throw new UnsupportedOperationException();
