[GitHub] hrsakai commented on issue #3535: Deduplication causes a lot of Full GCs

[GitBox]

hrsakai commented on issue #3535: Deduplication causes a lot of Full GCs
URL: https://github.com/apache/pulsar/issues/3535#issuecomment-463862000
 
 
   @ivankelly 
   At this time, I produced messages to a partitioned topic (16 partitions).
   And There are 4 producers, 0 consumer and 1 subscription.
   
   > Can you try enabling debug logging for 
org.apache.pulsar.broker.service.persistent.MessageDeduplication? 
   
   The broker outputed a lot of logs as follow:
   ```
   08:33:16.719 [bookkeeper-ml-workers-OrderedExecutor-3-0] DEBUG 
o.a.p.b.s.p.MessageDeduplication     - 
[persistent://XXXXX/global/deduplication/p-topic2-partition-3] Taking snapshot 
of sequence ids map
   08:33:16.719 [bookkeeper-ml-workers-OrderedExecutor-3-0] DEBUG 
o.a.p.b.s.p.MessageDeduplication     - 
[persistent://XXXXX/global/deduplication/p-topic2-partition-3] Stored new 
deduplication snapshot at 790835:45361
   08:33:16.719 [bookkeeper-ml-workers-OrderedExecutor-3-0] DEBUG 
o.a.p.b.s.p.MessageDeduplication     - 
[persistent://XXXXX/global/deduplication/p-topic2-partition-3] Taking snapshot 
of sequence ids map
   08:33:16.719 [bookkeeper-ml-workers-OrderedExecutor-3-0] DEBUG 
o.a.p.b.s.p.MessageDeduplication     - 
[persistent://XXXXX/global/deduplication/p-topic2-partition-3] Stored new 
deduplication snapshot at 790835:45362
   08:33:16.719 [bookkeeper-ml-workers-OrderedExecutor-3-0] DEBUG 
o.a.p.b.s.p.MessageDeduplication     - 
[persistent://XXXXX/global/deduplication/p-topic2-partition-3] Taking snapshot 
of sequence ids map
   08:33:16.719 [bookkeeper-ml-workers-OrderedExecutor-3-0] DEBUG 
o.a.p.b.s.p.MessageDeduplication     - 
[persistent://XXXXX/global/deduplication/p-topic2-partition-3] Stored new 
deduplication snapshot at 790835:45363
   08:33:16.719 [bookkeeper-ml-workers-OrderedExecutor-1-0] DEBUG 
o.a.p.b.s.p.MessageDeduplication     - 
[persistent://XXXXX/global/deduplication/p-topic2-partition-5] Taking snapshot 
of sequence ids map
   08:33:16.719 [bookkeeper-ml-workers-OrderedExecutor-1-0] DEBUG 
o.a.p.b.s.p.MessageDeduplication     - 
[persistent://XXXXX/global/deduplication/p-topic2-partition-5] Stored new 
deduplication snapshot at 790833:57439
   08:33:16.719 [bookkeeper-ml-workers-OrderedExecutor-1-0] DEBUG 
o.a.p.b.s.p.MessageDeduplication     - 
[persistent://XXXXX/global/deduplication/p-topic2-partition-9] Taking snapshot 
of sequence ids map
   08:33:16.719 [bookkeeper-ml-workers-OrderedExecutor-1-0] DEBUG 
o.a.p.b.s.p.MessageDeduplication     - 
[persistent://XXXXX/global/deduplication/p-topic2-partition-9] Stored new 
deduplication snapshot at 790831:57486
   .
   .
   ```
   
   `brokerDeduplicationEntriesInterval` is 1 on this broker. This may be 
affecting.
   However,  this issue occurs even if `brokerDeduplicationEntriesInterval` is 
default value(1000).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services