JingsongLi commented on code in PR #10663:
URL: https://github.com/apache/paimon/pull/10663#discussion_r4314646764
##########
paimon-hive/paimon-hive-connector-common/src/main/java/org/apache/paimon/hive/utils/HiveSplitGenerator.java:
##########
@@ -130,6 +128,29 @@ public static InputSplit[] generateSplits(
return splits.toArray(new InputSplit[0]);
}
+ private static List<String> locationsToPlan(
+ FileStoreTable table, JobConf jobConf, String locations) {
+ List<String> belonging = new ArrayList<>();
+ String foreign = null;
+ for (String location : locations.split(",")) {
+ // Hive create dummy file for empty table or partition. If this
location doesn't
+ // belong to this table, nothing to do.
+ if (location.startsWith(table.location().toString())) {
+ belonging.add(location);
+ } else if (foreign == null && !location.isEmpty()) {
+ foreign = location;
+ }
+ }
+ // With LOCATION_IN_PROPERTIES, Hive gives the table a location under
its own warehouse,
+ // which does not start with the paimon location. If the table has no
Hive partitions,
+ // that location stands for the whole table. Hive partitions of such a
table all share
+ // it, so they cannot be told apart and are still skipped.
+ if (belonging.isEmpty() && foreign != null &&
!jobConf.getBoolean(HIVE_PARTITIONED, true)) {
+ return Collections.singletonList(foreign);
Review Comment:
[P2] Keep a foreign whole-table location out of partition extraction
The returned foreign location is still passed to `createPartitionPredicate`
in `generateSplits`, which extracts matching `key=value` components from every
ancestor. Hive accepts a warehouse root such as `/hive/pt=plain/warehouse`; for
a Hive-unpartitioned table whose Paimon data is partitioned by `pt`, the
generated metastore location therefore adds an unrelated `pt=plain` filter.
With two committed rows in two Paimon partitions, I read both rows using
`/hive/warehouse/db/table`, only the `plain` row using
`/hive/pt=plain/warehouse/db/table`, and zero rows with `pt=missing`, all
without a query predicate and with `HIVE_PARTITIONED=false`. This violates the
new whole-table fallback and silently returns incomplete results for a valid
warehouse configuration. Preserve the foreign path for Hive split routing, but
bypass location-derived partition predicates for this fallback while retaining
the actual pushed-down query filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
