Akash3121 commented on code in PR #11151:
URL: https://github.com/apache/paimon/pull/11151#discussion_r4579631997
##########
paimon-core/src/test/java/org/apache/paimon/migrate/CopiedDataFilesTest.java:
##########
@@ -323,6 +325,13 @@ public void
testDedicatedFilesFollowingTheirRowsAreAccepted() throws Exception {
.extracting(DataFileMeta::fileName)
.containsExactly(
"n0.parquet", "v0.vector.parquet", "n1.parquet",
"v1.vector.parquet");
+ List<ManifestEntry> entries = new ArrayList<>();
+ for (DataFileMeta file : vectors) {
+ entries.add(ManifestEntry.create(FileKind.ADD,
BinaryRow.EMPTY_ROW, 0, -1, file));
+ }
+ assertThat(RowTrackingCommitUtils.assignRowTracking(1L, 0L,
entries).assignedEntries)
+ .extracting(entry -> entry.file().firstRowId())
+ .containsExactly(0L, 0L, 100L, 150L);
Review Comment:
Nice catch preserving `vectorStoreStart` when a copied vector file extends
past the next normal file's start!
One follow-up question: right below in
`testDedicatedFilesTheCommitWouldGiveOtherRowsAreRefused` (lines 342–348 and
362–366), `CopiedDataFiles.checkDedicatedFilesFollowTheirRows`
(`CopiedDataFiles.java:183-190, 229-236`) still replays the old cursor behavior
(without `vectorStoreStart = Math.max(vectorStoreStart, start)`) and refuses
copying tables where a vector column was added after the first normal-only
files (`// a vector column added after the first rows: the vector-store file
would get rows 0 and 1`). Should
`CopiedDataFiles.checkDedicatedFilesFollowTheirRows` also be updated to mirror
this change (similar to how blob columns added after the first rows are
accepted in `blobs` below), or do you prefer to keep that for a separate
follow-up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]