Caideyipi commented on code in PR #18693:
URL: https://github.com/apache/iotdb/pull/18693#discussion_r4069904551
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/io/WALMetaData.java:
##########
@@ -218,9 +222,17 @@ public static WALMetaData deserialize(ByteBuffer buffer) {
public static WALMetaData deserialize(ByteBuffer buffer, WALFileVersion
version) {
long firstSearchIndex = buffer.getLong();
int entriesNum = buffer.getInt();
+ // Reject corrupted counts before allocating arrays from an untrusted
footer.
+ if (entriesNum < 0 || entriesNum > buffer.remaining() / Integer.BYTES) {
+ throw new
IllegalArgumentException(StorageEngineMessages.UNEXPECTED_EXCEPTION);
+ }
List<Integer> buffersSize = new ArrayList<>(entriesNum);
for (int i = 0; i < entriesNum; ++i) {
- buffersSize.add(buffer.getInt());
+ int size = buffer.getInt();
+ if (size <= 0) {
Review Comment:
Positive entry sizes still need to be validated against the actual logical
WAL data.
A corrupted footer can change an entry size to another positive value, so
this check accepts the footer and `WALRepairWriter.hasReadableMetadata()`
reports success without repairing it. I reproduced this with a one-entry V3 WAL
by decreasing the stored size by one byte: `isRecoveredFromEntries()` remained
`false`, `WALEntry.deserialize(reader.next())` threw `EOFException`, and
`repair()` still returned `true`. During startup, `recoverTsFiles()` catches
that failure and skips the WAL, so an otherwise readable entry is not replayed.
A large positive value can also reach `ByteBuffer.allocate(size)`.
Please cross-check the sizes, using overflow-safe arithmetic, against the
logical segment lengths and the end marker before treating the footer as valid.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/io/WALInputStream.java:
##########
@@ -223,6 +253,18 @@ private void loadNextSegmentV1() throws IOException {
private void loadNextSegmentV2() throws IOException {
long position = channel.position();
SegmentInfo segmentInfo = getNextSegmentInfo();
+ long remainingBytes = fileSize - channel.position();
+ if (recoveringEntries && segmentInfo.compressionType ==
CompressionType.UNCOMPRESSED) {
+ // Complete entries in a partially written uncompressed segment remain
readable. A compressed
+ // segment needs its full payload before any of its entries can be
recovered.
+ segmentInfo.dataInDiskSize = (int) Math.min(segmentInfo.dataInDiskSize,
remainingBytes);
+ segmentInfo.uncompressedSize = segmentInfo.dataInDiskSize;
+ }
+ if (segmentInfo.dataInDiskSize <= 0
+ || segmentInfo.uncompressedSize <= 0
Review Comment:
`uncompressedSize` also needs a maximum bound before it is used by
`ByteBuffer.allocateDirect` below. In recovery mode the bytes being inspected
are potentially corrupt, and the current predicate only checks that this value
is positive.
I reproduced an `OutOfMemoryError` with a 16-byte WAL containing an LZ4
segment header with `dataInDiskSize = 1` and `uncompressedSize = 64 MiB` under
an 8 MiB direct-memory limit. Because `OutOfMemoryError` is not caught by the
surrounding `catch (Exception)` or by `WALReader`, the file is not quarantined
and startup recovery can abort or hang rather than continue.
Please reject values above the maximum logical segment size the writer can
produce, with overflow-safe checks, before allocating the direct buffer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]