jt2594838 commented on code in PR #18456:
URL: https://github.com/apache/iotdb/pull/18456#discussion_r3765620292
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java:
##########
@@ -1414,6 +1414,12 @@ private void loadWALProps(TrimProperties properties)
throws IOException {
conf.setWALCacheShrinkClearEnabled(WALInsertNodeCacheShrinkClearEnabled);
}
+ conf.setWalFileListCacheEnabled(
Review Comment:
Loads the setting only during startup configuration processing. It
intentionally remains absent from hot-reload handling because each WALNode
captures the setting while it is constructed.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/buffer/WALBuffer.java:
##########
@@ -168,16 +194,18 @@ private int getCompressedByteBufferSize(int size) {
@Override
protected File rollLogWriter(long searchIndex, WALFileStatus fileStatus)
throws IOException {
- File file = super.rollLogWriter(searchIndex, fileStatus);
+ File sealedWalFile = super.rollLogWriter(searchIndex, fileStatus);
currentWALFileWriter.setCompressedByteBuffer(compressedByteBuffer);
+ // Update the WAL node's ordered file index before waking readers waiting
for this roll.
+ walFileRolledListener.accept(sealedWalFile,
currentWALFileWriter.getLogFile());
Review Comment:
Publishes both the sealed and new active file to WALNode before notifying
waiting readers. This ordering prevents a reader awakened by a roll from
observing a stale ordered index.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/node/WALNode.java:
##########
@@ -120,6 +121,15 @@ public class WALNode implements IWALNode {
// WAL files with versionId >= this value are retained for subscription
consumers
private volatile long subscriptionRetainedMinVersionId = Long.MAX_VALUE;
+ private final boolean walFileListCacheEnabled;
+ private final Object walFileListCacheLock = new Object();
+ // Maintains the cache incrementally by version; all mutations are protected
by the cache lock.
+ private final TreeMap<Long, File> walFilesByVersion = new TreeMap<>();
Review Comment:
Uses a TreeMap as the incrementally maintained, version-ordered source of
truth. The separate volatile array is an immutable published snapshot, avoiding
a directory scan and sort on steady-state accesses.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/node/WALNode.java:
##########
@@ -135,9 +145,80 @@ public WALNode(
logger.info(StorageEngineMessages.CREATE_FOLDER_FOR_WAL_NODE,
logDirectory, identifier);
}
this.checkpointManager = new CheckpointManager(identifier, logDirectory);
+ this.walFileListCacheEnabled = config.isWalFileListCacheEnabled();
this.buffer =
new WALBuffer(
- identifier, logDirectory, checkpointManager, startFileVersion,
startSearchIndex);
+ identifier,
+ logDirectory,
+ checkpointManager,
+ startFileVersion,
+ startSearchIndex,
+ this::updateSortedWalFilesCacheAfterRoll);
+ initializeSortedWalFilesCacheIfEnabled();
+ }
+
+ private File[] getSortedWalFiles() {
Review Comment:
Builds the array only after a reader or another file-list consumer accesses
a dirty cache. The lock protects mutation and snapshot publication; readers can
reuse an already published snapshot without locking.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/node/WALNode.java:
##########
@@ -413,6 +493,7 @@ private void deleteOutdatedFilesAndUpdateMetric() {
}
buffer.subtractDiskUsage(deleteFileSize);
buffer.subtractFileNum(successfullyDeleted.size());
+ removeDeletedWalFilesFromCache(successfullyDeleted);
Review Comment:
Removes successfully deleted WAL versions from the index in the same
deletion path, then defers replacement of the snapshot until the next access.
This keeps deletion incremental and prevents stale files being returned.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/node/WALNode.java:
##########
@@ -916,8 +997,7 @@ private void reset() {
}
private void updateFilesToSearch() {
- File[] filesToSearch = WALFileUtils.listAllWALFiles(logDirectory);
- WALFileUtils.ascSortByVersionId(filesToSearch);
+ File[] filesToSearch = getSortedWalFiles();
Review Comment:
Consensus request readers now obtain the ordered snapshot through the shared
accessor. The same accessor is also used by WAL deletion-bound queries, while
disabled mode retains the prior scan-and-sort behavior.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/conf/PropertiesTest.java:
##########
@@ -41,6 +41,34 @@
import static com.tngtech.archunit.lang.syntax.ArchRuleDefinition.noClasses;
public class PropertiesTest {
+ /**
+ * Verifies that WAL file-list caching defaults on, supports startup
override, and is
+ * restart-only.
+ */
+ @Test
+ public void testWalFileListCacheConfiguration() throws Exception {
Review Comment:
Covers all configuration contracts introduced here: Java and template
defaults are enabled, startup can explicitly disable the feature, and hot
reload cannot alter this restart-only setting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
