hongzhi-gao commented on code in PR #18693:
URL: https://github.com/apache/iotdb/pull/18693#discussion_r4078735823
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/io/WALMetaData.java:
##########
@@ -218,9 +222,17 @@ public static WALMetaData deserialize(ByteBuffer buffer) {
public static WALMetaData deserialize(ByteBuffer buffer, WALFileVersion
version) {
long firstSearchIndex = buffer.getLong();
int entriesNum = buffer.getInt();
+ // Reject corrupted counts before allocating arrays from an untrusted
footer.
+ if (entriesNum < 0 || entriesNum > buffer.remaining() / Integer.BYTES) {
+ throw new
IllegalArgumentException(StorageEngineMessages.UNEXPECTED_EXCEPTION);
+ }
List<Integer> buffersSize = new ArrayList<>(entriesNum);
for (int i = 0; i < entriesNum; ++i) {
- buffersSize.add(buffer.getInt());
+ int size = buffer.getInt();
+ if (size <= 0) {
Review Comment:
[P2] Checking only `size > 0` still accepts a corrupted entry length such as
`Integer.MAX_VALUE` in an otherwise parseable footer.
`readFromWALFileWithoutRecovery` then treats the footer as valid, and
`WALByteBufReader.next()` calls `ByteBuffer.allocate(size)` before reading any
entry bytes, potentially exhausting the process heap. Please validate entry
sizes against the actual readable data (or impose a safe allocation bound) so
this case enters recovery instead.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/io/WALInputStream.java:
##########
@@ -223,6 +255,28 @@ private void loadNextSegmentV1() throws IOException {
private void loadNextSegmentV2() throws IOException {
long position = channel.position();
SegmentInfo segmentInfo = getNextSegmentInfo();
+ long remainingBytes = fileSize - channel.position();
+ if (recoveringEntries && segmentInfo.compressionType ==
CompressionType.UNCOMPRESSED) {
+ // Complete entries in a partially written uncompressed segment remain
readable. A compressed
+ // segment needs its full payload before any of its entries can be
recovered.
+ segmentInfo.dataInDiskSize = (int) Math.min(segmentInfo.dataInDiskSize,
remainingBytes);
+ segmentInfo.uncompressedSize = segmentInfo.dataInDiskSize;
+ }
+ if (segmentInfo.dataInDiskSize <= 0
+ || segmentInfo.uncompressedSize <= 0
+ || segmentInfo.dataInDiskSize > remainingBytes) {
+ throw new EOFException(StorageEngineMessages.UNEXPECTED_END_OF_FILE);
+ }
+ // Recovery inspects untrusted headers. Bound allocations by the writer's
configured segment
+ // capacity so a tiny corrupt payload cannot request a huge decompression
buffer.
+ if (recoveringEntries
+ && (segmentInfo.uncompressedSize > WALBuffer.ONE_THIRD_WAL_BUFFER_SIZE
Review Comment:
[P1] This limit is derived from the *current* `wal_buffer_size`, but a WAL
may have been written with a larger value before a configuration change. On
restart, recovery will reject a valid segment from that file. If it is the
first segment and the footer is damaged, the scan finds no entries and
quarantines the WAL, losing its readable data. Could the recovery bound be
independent of the current writer buffer setting (while still limiting
allocations)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]