Caideyipi commented on code in PR #19335:
URL: https://github.com/apache/iotdb/pull/19335#discussion_r4638051177
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/LoadTsFileSnapshot.java:
##########
@@ -167,6 +167,10 @@ private static void copyTask(final File taskDir, final
File targetDir) throws IO
}
private static void discardUnrecordedStagedFiles(final File snapshotTaskDir)
throws IOException {
+ // A whole-file task has no chunk records by design: its marker and its
bytes are the state.
+ if (new File(snapshotTaskDir,
TsFileWriterManager.WHOLE_FILE_META_NAME).isFile()) {
Review Comment:
[P2] Catch up whole-file bytes to the snapshot boundary. `copyTask` copies
staged data before progress logs, then relies on `catchUpStagedFiles` to repair
concurrent appends from recorded ranges. Whole-file writes do not record
per-slice ranges, so this early return preserves whatever prefix the copy
captured. If a snapshot overlaps a PIECE already covered by its consensus
boundary, the restored staging file can miss that slice tail; later slices then
fail the offset check in `appendWholeFile`, or PREPARE rejects the file length.
Please record/catch up a whole-file applied offset or otherwise serialize these
writes with snapshot copying.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/load/TsFileWriterManager.java:
##########
@@ -830,6 +1189,52 @@ TsFileWriterManager recoverFromDisk() throws IOException {
return this;
}
+ private TsFileWriterManager recoverWholeFileFromDisk(final File metaFile)
throws IOException {
+ final WholeFileMeta meta = readWholeFileMeta(metaFile);
+ if (meta == null) {
+ return this;
+ }
+ wholeFile = new File(taskDir, meta.tsFileName);
+ if (!wholeFile.isFile()) {
+ return this;
+ }
+ wholeFileChannel =
+ FileChannel.open(wholeFile.toPath(), StandardOpenOption.READ,
StandardOpenOption.WRITE);
+ wholeFileSourceLength = meta.sourceLength;
+ wholeFileSealedLength = -1L;
+ wholeFileTimePartitionStart = meta.timePartitionStart;
+ wholeFileTimePartition = new
TTimePartitionSlot(wholeFileTimePartitionStart);
+ wholeFileGeneratedByPipe = meta.generatedByPipe;
+ wholeFileProgress = new LoadTsFileProgress(wholeFile);
+
+ final File modsV2 = ModificationFile.getExclusiveMods(wholeFile);
+ final File modsV1 = ModificationFileV1.getNormalMods(wholeFile);
+ if (modsV2.isFile()) {
+ wholeFileMods = modsV2;
+ wholeFileModsKind = WholeFileData.Kind.MODS_V2;
+ } else if (modsV1.isFile()) {
+ wholeFileMods = modsV1;
+ wholeFileModsKind = WholeFileData.Kind.MODS_V1;
+ }
+ if (wholeFileMods != null) {
+ wholeFileModsChannel =
+ FileChannel.open(
+ wholeFileMods.toPath(), StandardOpenOption.READ,
StandardOpenOption.WRITE);
+ wholeFileModsSourceLength = wholeFileMods.length();
Review Comment:
[P2] Persist the declared mods length for recovery. `ensureWholeFileMods`
compares this field with each incoming slice sourceLength, but recovery assigns
the current staged prefix length here. After a restart during a multi-slice
.mods transfer, the next replayed slice carries the original full source length
and is rejected, so the transaction cannot resume. Persist the mods kind and
declared source length separately from the staged file length, and restore
those values here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]