> > >> Could you give some information on what those shortcomings are? Also, do > > >> let me know if you need any more information from our end. > > > Off the top of my head: > > > - reads and writes are handled in the same thread (as you have observed) > > > - each entry read requires a single RPC. > > > - entries are read in parallel > > > By parallel, you mean the BufferedChannel wrapper on top of FileChannel, > right? No, i mean the RPCs from the client are all sent in parallel. This isn't a problem in this case, and is really related to the second point.
> > > > > > > Not all of these could result in the high latency you see, but if each > > > entry is being read separately, a sync on the ledger disk in between > > > will make a mess of the disk head scheduling. > > > Increasing the time interval between flushing log files might possibly > help in this case then? Possibly, but really the biggest problem, I think, is that each entry is being read individually. So if we want to read all of ledger L1 and write traffic is occurring, we can get the scenario. - read L1, entry 1 (disk seek for index, disk seek for entry) - Sync (many disk seeks) - read L2, entry 2 (disk seek for entry) - Sync (many disk seeks) - read L2, entry 3 (disk seek for entry) - Sync (many disk seeks) - read L2, entry 4 (disk seek for entry) ... Also, if the sync is causing the index to be flushed out of the index cache, there will be an extra seek for the read. I think the solution would be to have reads hold off syncs while all entries in one ledger are read. This should be safe, as the journal is ensuring the data is persisted. How big is the disk readahead on your disks? Also, what is the average entry size? -Ivan
