JingsongLi commented on code in PR #11277:
URL: https://github.com/apache/paimon/pull/11277#discussion_r4643245725
##########
paimon-core/src/main/java/org/apache/paimon/partition/PartitionPredicate.java:
##########
@@ -282,6 +296,46 @@ public boolean test(
return false;
}
}
+ if (partitionValues == null || rowCount <= 0) {
+ return true;
+ }
+ Object[] lower = converter.convert(minValues);
+ Object[] upper = converter.convert(maxValues);
+ long[] nulls = new long[fieldNum];
+ for (int i = 0; i < fieldNum; i++) {
+ nulls[i] =
+ nullCounts.size() == 0 || nullCounts.isNullAt(i)
+ ? -1
+ : nullCounts.getLong(i);
+ }
+ // At least one requested partition must match on every field for
the manifest to
+ // matter.
+ for (Object[] partition : partitionValues) {
+ if (mayContainPartition(partition, lower, upper, nulls,
rowCount)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private boolean mayContainPartition(
+ Object[] partition, Object[] lower, Object[] upper, long[]
nulls, long rowCount) {
+ for (int i = 0; i < fieldNum; i++) {
+ Object value = partition[i];
+ if (value == null) {
+ if (nulls[i] == 0) {
+ return false;
+ }
+ } else if (nulls[i] == rowCount) {
+ return false;
+ } else if (lower[i] != null && upper[i] != null) {
+ DataType type = converter.rowType().getTypeAt(i);
+ if (compareLiteral(type, value, lower[i]) < 0
+ || compareLiteral(type, value, upper[i]) > 0) {
Review Comment:
**[P1] Keep manifests containing floating partitions from older PyPaimon
writers.** I wrote two real Parquet files to one manifest using PyPaimon's
Python writer and committer, first `(id=1, p=5.0)` and then `(id=2, p=NaN)`,
with a DOUBLE partition key. Java `SnapshotReader.withPartitionFilter` for the
exact partition values `{0.0, NaN}` planned one split and read ID 2 on master.
It planned zero splits on this head. The Python manifest writer uses Python
min/max, which record 5.0 as both bounds in this ordering. The combined bounds
keep the manifest, but the new per-value comparisons conclude that neither
requested partition can occur. Please keep the per-value check conservative for
floating bounds that may use a different ordering, and add a
Python-write/Java-read regression for existing manifests.
##########
paimon-core/src/main/java/org/apache/paimon/partition/PartitionPredicate.java:
##########
@@ -282,6 +296,46 @@ public boolean test(
return false;
}
}
+ if (partitionValues == null || rowCount <= 0) {
+ return true;
+ }
+ Object[] lower = converter.convert(minValues);
+ Object[] upper = converter.convert(maxValues);
+ long[] nulls = new long[fieldNum];
+ for (int i = 0; i < fieldNum; i++) {
+ nulls[i] =
+ nullCounts.size() == 0 || nullCounts.isNullAt(i)
+ ? -1
+ : nullCounts.getLong(i);
+ }
+ // At least one requested partition must match on every field for
the manifest to
+ // matter.
+ for (Object[] partition : partitionValues) {
Review Comment:
**[P2] Bound per-manifest work for large multi-partition selections.**
Dynamic overwrite builds this predicate from every touched partition. A wide
overwrite can still have thousands of manifests whose stats hit one selected
partition. In a JDK 8 probe with five STRING fields, 10,000 distinct
partitions, and 13,151 candidate manifests that matched the last partition in
iteration order, the new pass took 8.8 s and skipped zero manifests. The master
predicate took about 2 ms on the same inputs. The extra work runs in scan
planning and in the overwrite commit path. Please bound the extra checks and
conservatively keep the manifest when the budget is exhausted, or use an
indexed candidate lookup. Cover both sparse pruning and large low-selectivity
selections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]