dlg99 commented on issue #2820: URL: https://github.com/apache/bookkeeper/issues/2820#issuecomment-941199314
@hangc0276 I'd start with monitoring of the situation. Is the disk a bottleneck? I understand that NVME SSDs are very fast, but you have: - journal writes with fsync - memtable flush / entrylog writes with fsync - possibly reads (concurrent reads+writes can affect write throughput) - compaction - index writes (flat files or RocksDB) - small but with fsync iostat, sar, etc should help with this. Watch for io - writes/reads, io queues, latencies, paging etc. Most likely something like memtable flush causes slower journal writes and results in a longer journal queue, queue put blocks. there are metrics around that, do you see queue size maxing up? As a simple test you can try configuring 2-3GB ramdrive (if node memory permits), move the journal there + disable journalSyncData. Is the journal/queue still a bottleneck? Check if the disk configuration is optimized (deadline or noop scheduler for disk, ec, google for ideas to optimize IO for nvme ssd) After that you can try tuning some parameters, like increase journalBufferedWritesThreshold, increase journalPreAllocSizeMB, journalWriteBufferSizeKB, journalQueueSize, etc. Experiment with read/write buffers for entry log, maybe try flushEntrylogBytes of e.g.64-128MB (more frequent but faster flushes instead of) tune compaction intervals to make it run less frequently and with smaller data volumes to rewrite, if the disk space permits. I don't remember that well, but I think disabling journalRemoveFromPageCache might help if the node has a lot of memory. If you can confirm that the queue (and not the disk) is definitely a bottleneck, we have JCTools as a dependency already and can try swapping the queue implementation. See https://github.com/apache/bookkeeper/pull/1682 Hope this helps. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
