jt2594838 commented on code in PR #18693:
URL: https://github.com/apache/iotdb/pull/18693#discussion_r4088998236
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/io/WALInputStream.java:
##########
@@ -223,6 +255,28 @@ private void loadNextSegmentV1() throws IOException {
private void loadNextSegmentV2() throws IOException {
long position = channel.position();
SegmentInfo segmentInfo = getNextSegmentInfo();
+ long remainingBytes = fileSize - channel.position();
+ if (recoveringEntries && segmentInfo.compressionType ==
CompressionType.UNCOMPRESSED) {
+ // Complete entries in a partially written uncompressed segment remain
readable. A compressed
+ // segment needs its full payload before any of its entries can be
recovered.
+ segmentInfo.dataInDiskSize = (int) Math.min(segmentInfo.dataInDiskSize,
remainingBytes);
+ segmentInfo.uncompressedSize = segmentInfo.dataInDiskSize;
+ }
+ if (segmentInfo.dataInDiskSize <= 0
+ || segmentInfo.uncompressedSize <= 0
+ || segmentInfo.dataInDiskSize > remainingBytes) {
+ throw new EOFException(StorageEngineMessages.UNEXPECTED_END_OF_FILE);
+ }
+ // Recovery inspects untrusted headers. Bound allocations by the writer's
configured segment
+ // capacity so a tiny corrupt payload cannot request a huge decompression
buffer.
+ if (recoveringEntries
+ && (segmentInfo.uncompressedSize > WALBuffer.ONE_THIRD_WAL_BUFFER_SIZE
Review Comment:
It is too difficult to decide on a safe recovery bound without introducing
additional OOM risk.
Since this thing is overall a best-effort attempt, I would prefer to leave
the WAL quarantined and let the DBA decide whether increasing the memory
setting and restarting are applicable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]