JingsongLi commented on code in PR #9402:
URL: https://github.com/apache/paimon/pull/9402#discussion_r3930444034
##########
paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/copy/ListIndexFilesOperator.java:
##########
@@ -66,15 +70,27 @@ public List<CopyFileInfo> execute(
FileStoreTable targetTable = (FileStoreTable)
targetCatalog.getTable(targetIdentifier);
List<CopyFileInfo> indexFiles = new ArrayList<>();
IndexFileHandler sourceIndexHandler =
sourceTable.store().newIndexFileHandler();
+ FileStorePathFactory sourceFileStorePathFactory =
sourceTable.store().pathFactory();
FileStorePathFactory targetFileStorePathFactory =
targetTable.store().pathFactory();
List<IndexManifestEntry> indexManifestEntries =
sourceIndexHandler.readManifestWithIOException(snapshot.indexManifest());
+ Set<IndexManifestEntry> compatibleGlobalIndexes =
+ new HashSet<>(
+ GlobalIndexSchemaCompatibility.filterCompatible(
+ sourceTable, indexManifestEntries));
for (IndexManifestEntry indexManifestEntry : indexManifestEntries) {
+ boolean globalIndex =
indexManifestEntry.indexFile().globalIndexMeta() != null;
+ if (globalIndex &&
!compatibleGlobalIndexes.contains(indexManifestEntry)) {
Review Comment:
[P2] Preserve primary-key global-index payloads during copy
This filter applies to every entry with `globalIndexMeta() != null`, but the
production PK payload builders still create their `IndexFileMeta` without a
schema id (`PkFullTextIndexFile`, `PkSortedIndexFile`, and
`PkVectorAnnSegmentFile`). `ManifestEntryChanges` only propagates that null
value, so `filterCompatible` rejects all of these payloads and this branch
silently omits them from `copy_files`. Before this PR they were copied. The
target data can generally fall back to raw/exact reads, but the copied table
loses all PK sorted/full-text/vector acceleration, and FAST-mode behavior can
change.
Please either stamp the build schema id in those PK payload builders and
validate them on read, or scope this compatibility filter to index families
that actually carry schema identity. A copy test should assert each PK payload
remains in the target manifest and produces equivalent queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
