Caideyipi commented on code in PR #18693:
URL: https://github.com/apache/iotdb/pull/18693#discussion_r4069904551


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/io/WALMetaData.java:
##########
@@ -218,9 +222,17 @@ public static WALMetaData deserialize(ByteBuffer buffer) {
   public static WALMetaData deserialize(ByteBuffer buffer, WALFileVersion 
version) {
     long firstSearchIndex = buffer.getLong();
     int entriesNum = buffer.getInt();
+    // Reject corrupted counts before allocating arrays from an untrusted 
footer.
+    if (entriesNum < 0 || entriesNum > buffer.remaining() / Integer.BYTES) {
+      throw new 
IllegalArgumentException(StorageEngineMessages.UNEXPECTED_EXCEPTION);
+    }
     List<Integer> buffersSize = new ArrayList<>(entriesNum);
     for (int i = 0; i < entriesNum; ++i) {
-      buffersSize.add(buffer.getInt());
+      int size = buffer.getInt();
+      if (size <= 0) {

Review Comment:
   Positive entry sizes still need to be validated against the actual logical 
WAL data.
   
   A corrupted footer can change an entry size to another positive value, so 
this check accepts the footer and `WALRepairWriter.hasReadableMetadata()` 
reports success without repairing it. I reproduced this with a one-entry V3 WAL 
by decreasing the stored size by one byte: `isRecoveredFromEntries()` remained 
`false`, `WALEntry.deserialize(reader.next())` threw `EOFException`, and 
`repair()` still returned `true`. During startup, `recoverTsFiles()` catches 
that failure and skips the WAL, so an otherwise readable entry is not replayed. 
A large positive value can also reach `ByteBuffer.allocate(size)`.
   
   Please cross-check the sizes, using overflow-safe arithmetic, against the 
logical segment lengths and the end marker before treating the footer as valid.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/io/WALInputStream.java:
##########
@@ -223,6 +253,18 @@ private void loadNextSegmentV1() throws IOException {
   private void loadNextSegmentV2() throws IOException {
     long position = channel.position();
     SegmentInfo segmentInfo = getNextSegmentInfo();
+    long remainingBytes = fileSize - channel.position();
+    if (recoveringEntries && segmentInfo.compressionType == 
CompressionType.UNCOMPRESSED) {
+      // Complete entries in a partially written uncompressed segment remain 
readable. A compressed
+      // segment needs its full payload before any of its entries can be 
recovered.
+      segmentInfo.dataInDiskSize = (int) Math.min(segmentInfo.dataInDiskSize, 
remainingBytes);
+      segmentInfo.uncompressedSize = segmentInfo.dataInDiskSize;
+    }
+    if (segmentInfo.dataInDiskSize <= 0
+        || segmentInfo.uncompressedSize <= 0

Review Comment:
   `uncompressedSize` also needs a maximum bound before it is used by 
`ByteBuffer.allocateDirect` below. In recovery mode the bytes being inspected 
are potentially corrupt, and the current predicate only checks that this value 
is positive.
   
   I reproduced an `OutOfMemoryError` with a 16-byte WAL containing an LZ4 
segment header with `dataInDiskSize = 1` and `uncompressedSize = 64 MiB` under 
an 8 MiB direct-memory limit. Because `OutOfMemoryError` is not caught by the 
surrounding `catch (Exception)` or by `WALReader`, the file is not quarantined 
and startup recovery can abort or hang rather than continue.
   
   Please reject values above the maximum logical segment size the writer can 
produce, with overflow-safe checks, before allocating the direct buffer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to