JingsongLi commented on code in PR #11277:
URL: https://github.com/apache/paimon/pull/11277#discussion_r4643245725


##########
paimon-core/src/main/java/org/apache/paimon/partition/PartitionPredicate.java:
##########
@@ -282,6 +296,46 @@ public boolean test(
                     return false;
                 }
             }
+            if (partitionValues == null || rowCount <= 0) {
+                return true;
+            }
+            Object[] lower = converter.convert(minValues);
+            Object[] upper = converter.convert(maxValues);
+            long[] nulls = new long[fieldNum];
+            for (int i = 0; i < fieldNum; i++) {
+                nulls[i] =
+                        nullCounts.size() == 0 || nullCounts.isNullAt(i)
+                                ? -1
+                                : nullCounts.getLong(i);
+            }
+            // At least one requested partition must match on every field for 
the manifest to
+            // matter.
+            for (Object[] partition : partitionValues) {
+                if (mayContainPartition(partition, lower, upper, nulls, 
rowCount)) {
+                    return true;
+                }
+            }
+            return false;
+        }
+
+        private boolean mayContainPartition(
+                Object[] partition, Object[] lower, Object[] upper, long[] 
nulls, long rowCount) {
+            for (int i = 0; i < fieldNum; i++) {
+                Object value = partition[i];
+                if (value == null) {
+                    if (nulls[i] == 0) {
+                        return false;
+                    }
+                } else if (nulls[i] == rowCount) {
+                    return false;
+                } else if (lower[i] != null && upper[i] != null) {
+                    DataType type = converter.rowType().getTypeAt(i);
+                    if (compareLiteral(type, value, lower[i]) < 0
+                            || compareLiteral(type, value, upper[i]) > 0) {

Review Comment:
   **[P1] Keep manifests containing floating partitions from older PyPaimon 
writers.** I wrote two real Parquet files to one manifest using PyPaimon's 
Python writer and committer, first `(id=1, p=5.0)` and then `(id=2, p=NaN)`, 
with a DOUBLE partition key. Java `SnapshotReader.withPartitionFilter` for the 
exact partition values `{0.0, NaN}` planned one split and read ID 2 on master. 
It planned zero splits on this head. The Python manifest writer uses Python 
min/max, which record 5.0 as both bounds in this ordering. The combined bounds 
keep the manifest, but the new per-value comparisons conclude that neither 
requested partition can occur. Please keep the per-value check conservative for 
floating bounds that may use a different ordering, and add a 
Python-write/Java-read regression for existing manifests.



##########
paimon-core/src/main/java/org/apache/paimon/partition/PartitionPredicate.java:
##########
@@ -282,6 +296,46 @@ public boolean test(
                     return false;
                 }
             }
+            if (partitionValues == null || rowCount <= 0) {
+                return true;
+            }
+            Object[] lower = converter.convert(minValues);
+            Object[] upper = converter.convert(maxValues);
+            long[] nulls = new long[fieldNum];
+            for (int i = 0; i < fieldNum; i++) {
+                nulls[i] =
+                        nullCounts.size() == 0 || nullCounts.isNullAt(i)
+                                ? -1
+                                : nullCounts.getLong(i);
+            }
+            // At least one requested partition must match on every field for 
the manifest to
+            // matter.
+            for (Object[] partition : partitionValues) {

Review Comment:
   **[P2] Bound per-manifest work for large multi-partition selections.** 
Dynamic overwrite builds this predicate from every touched partition. A wide 
overwrite can still have thousands of manifests whose stats hit one selected 
partition. In a JDK 8 probe with five STRING fields, 10,000 distinct 
partitions, and 13,151 candidate manifests that matched the last partition in 
iteration order, the new pass took 8.8 s and skipped zero manifests. The master 
predicate took about 2 ms on the same inputs. The extra work runs in scan 
planning and in the overwrite commit path. Please bound the extra checks and 
conservatively keep the manifest when the budget is exhausted, or use an 
indexed candidate lookup. Cover both sparse pruning and large low-selectivity 
selections.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to