codelipenghui commented on issue #20968:
URL: https://github.com/apache/pulsar/issues/20968#issuecomment-1678356174
I don't think we have related fixes in 3.1.0. Is It only happening on the
partitioned topic?
I also tried to reproduce the issue on my laptop, but not able to reproduce
it.
My reproduction steps:
1. Change the `managedLedgerMinLedgerRolloverTimeMinutes=1` in broker.conf
to make the ledger rollover more frequently
2. Create a partitioned topic (bin/pulsar-admin topics
create-partitioned-topic test -p 3)
3. Set data retention (bin/pulsar-admin topics set-retention -s -1 -t -1
test)
4. Get data retention to make sure the data retention is applied properly
5. Start a perf producer to publish messages to the topic (bin/pulsar-perf
produce -bm 1 -r 10000 test)
6. Check the internal stats to see if there are ledgers that have been
removed (bin/pulsar-admin topics stats-internal test-partition-0)
```json
{
"entriesAddedCounter" : 3249591,
"numberOfEntries" : 3249591,
"totalSize" : 3461950342,
"currentLedgerEntries" : 45797,
"currentLedgerSize" : 48819602,
"lastLedgerCreatedTimestamp" : "2023-08-15T10:51:13.877+08:00",
"waitingCursorsCount" : 0,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "67:45796",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 19,
"entries" : 200046,
"size" : 213032478,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 22,
"entries" : 200126,
"size" : 213134190,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 25,
"entries" : 200173,
"size" : 213184245,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 28,
"entries" : 200887,
"size" : 213944655,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 31,
"entries" : 199916,
"size" : 212910540,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 34,
"entries" : 200138,
"size" : 213146970,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 37,
"entries" : 200400,
"size" : 213426000,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 40,
"entries" : 200165,
"size" : 213175725,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 43,
"entries" : 200232,
"size" : 213247080,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 46,
"entries" : 200352,
"size" : 213374880,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 49,
"entries" : 199942,
"size" : 213043455,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 52,
"entries" : 200235,
"size" : 213450510,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 55,
"entries" : 200280,
"size" : 213498480,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 58,
"entries" : 200405,
"size" : 213631730,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 61,
"entries" : 200196,
"size" : 213408936,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 64,
"entries" : 200301,
"size" : 213520866,
"offloaded" : false,
"underReplicated" : false
}, {
"ledgerId" : 67,
"entries" : 0,
"size" : 0,
"offloaded" : false,
"underReplicated" : false
} ],
"cursors" : { },
"schemaLedgers" : [ ],
"compactedLedger" : {
"ledgerId" : -1,
"entries" : -1,
"size" : -1,
"offloaded" : false,
"underReplicated" : false
}
}
```
No ledgers were removed.
> Still the question is how it's going to work with compacted, partitioned
topics, which should also retain the messages.
Have you enabled compaction for the topic? The compacted data will have
infinite data retention
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
