thetumbled commented on PR #23601:
URL: https://github.com/apache/pulsar/pull/23601#issuecomment-2478166382
List some experiment data:
```
static long trimLowerBit(long timestamp, int bits) {
return timestamp & (-1L << bits);
}
public static void main(String[] args) throws IOException {
ConcurrentLongLongPairHashMap map1 =
ConcurrentLongLongPairHashMap.newBuilder()
.autoShrink(true)
.concurrencyLevel(16)
.build();
// timestamp -> ledgerId -> entryId, no need to batch index, if
different messages have
// different timestamp, there will be multiple entries in the map
// AVL Tree -> LongOpenHashMap -> Roaring64Bitmap
// there are many timestamp, a few ledgerId, many entryId
Long2ObjectSortedMap<Long2ObjectMap<Roaring64Bitmap>> map2 = new
Long2ObjectAVLTreeMap<>();
long numMessages = 1000000, numLedgers=100;
long numEntries = numMessages/numLedgers;
long ledgerId, entryId, partitionIndex,
timestamp=System.currentTimeMillis();
for(long i=0; i<numLedgers; i++) {
ledgerId = 10000+i;
for(long j=0; j<numEntries; j++) {
entryId = 10000+j;
partitionIndex = 0;
// 1ms per message
timestamp++;
map1.put(ledgerId, entryId, partitionIndex, timestamp);
timestamp = trimLowerBit(timestamp, 8);
if (map2.containsKey(timestamp)) {
Long2ObjectMap<Roaring64Bitmap> ledgerMap =
map2.get(timestamp);
if (ledgerMap.containsKey(ledgerId)) {
ledgerMap.get(ledgerId).add(entryId);
} else {
Roaring64Bitmap entrySet = new Roaring64Bitmap();
entrySet.add(entryId);
ledgerMap.put(ledgerId, entrySet);
}
} else {
Long2ObjectMap<Roaring64Bitmap> ledgerMap = new
Long2ObjectOpenHashMap<>();
Roaring64Bitmap entrySet = new Roaring64Bitmap();
entrySet.add(entryId);
ledgerMap.put(ledgerId, entrySet);
map2.put(timestamp, ledgerMap);
}
}
}
try {
Thread.sleep(10000000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
