zymap opened a new pull request #12123:
URL: https://github.com/apache/pulsar/pull/12123
---
*Motivation*
We found the BlobStoreBackedReadHandler enter an infinite loop
when
reading a offload ledger.
We saw in the heap dump, there has two ledger 1 and 2, and we have
a consumer is reading ledger 1, but the heap shows the buffer it
reading is ledger 2. Then the read handler read a wrong entry id
and the entry id is out of range between the firstEntryId and the
lastEntryId, it will try to keep seeking to the right position, so
that it enters an infinite loop.
The buffer in the `BlobStoreBackedInputStreamImpl` is a wrong buffer,
so the read handler can not read a correct entry from it. After
investigating,
the buffer used in the `BlobStoreBackedInputStreamImpl` is allocated from
`PulsarByteBufAllocator.DEFAULT.buffer(bufferSize, bufferSize)`, by default,
we used a `PooledByteBufAllocator.DEFAULT` to allocate memory for that
buffer,
a weird thing we found is the closed buffer still can read data if there has
new buffer allocated and write things.
This is the test code:
```
ByteBuf buf = PulsarByteBufAllocator.DEFAULT.buffer(1024, 1024);
buf.writeByte(1);
System.out.println(buf.readByte()); // print 1
buf.release();
//System.out.println(buf.readByte()); // will throw exception
ByteBuf newBuf = PulsarByteBufAllocator.DEFAULT.buffer(1024, 1024);
newBuf.writeByte(2);
System.out.println(buf.readByte()); // print 2
newBuf.release();
//System.out.println(buf.readByte());
```
So we suspect there has a race condition between the read and close
operation. That cause there has a thread start reading after
the read handler closed.
*Modifications*
- Add a state check before reading entries.
- exit loop when the entryID is bigger than the lastEntryID
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
