Thanks, Howard! The last point is awesome so that I do not have to change logic for different use cases. I appreciate the simplicity of LMDB a lot, and given its performance I will switch to Linux only if we become really IO bound so that it becomes a bottleneck. Now our other parts are yet slower.
Happy holidays & New Year! On Thu, Dec 31, 2015 at 12:43 AM, Howard Chu <[email protected]> wrote: > Victor Baybekov wrote: > >> Thanks for the explanation! The ultimate speed is needed for server usage >> where, e.g. now, I have several TB free space and could allocate the >> entire >> space at start. For clients, where I use LMDB for replicated cache of data >> subsets, even the 3x perf drop is kind of tolerable given the absolute >> numbers. Could we have a flag to use the patch optionally? So that on a >> server >> where I could allocate 1+Tb and use hacks with WriteMap as we discussed >> previously, I could enjoy the speedup of WriteMap and the fact that the >> pointer addresses from MDB_RESERVE remain fixed over the lifetime of a >> process? >> > > I am loathe to add more option flags. All of this violates the principle > of simplicity which is core to LMDB. > > Another question: with this patch, do addresses of pointers remain the same >> after the map file size increases, or we must follow the LMDB rules as if >> there is no WriteMap flag is used? Does Windows reserve the entire virtual >> memory space, and the patch just affects the file size on a disk, or file >> growth with the patch is equivalent to remapping and pointer addresses >> become >> invalid every time? I am referring to this discussion >> <http://www.openldap.org/lists/openldap-technical/201510/msg00022.html> >> we had >> before. >> > > The entire virtual memory space is reserved, so there is no change in > behavior here. The patch would have been pointless if it required such a > change. > > P.S. I know your hate for Windows and am starting to share it given rich >> tools >> available on *nix, but for small non-hardcore-programmers teams doing >> exploratory number crunching the benefits of RDP & GUIs etc. outweigh the >> benefits *nix gives for high load production systems, so we use Windows >> as an >> internal server. LMDB is just a perfect off-heap data structure for many >> use >> cases. >> >> >> On Wed, Dec 30, 2015 at 11:32 PM, Howard Chu <[email protected] >> <mailto:[email protected]>> wrote: >> >> Victor Baybekov wrote: >> >> Hi, >> >> Thanks a lot for ITS#8324! For embedded, not server, use case that >> change adds >> much convenience. >> >> I have tested the master from .NET via P/Invoke and do not see >> any major >> slowdown with default options. To insert 10M <int32,int32> pairs >> inside a >> single transaction v.0.9.14 takes minimum 3400 msec, latest >> master takes >> minimum 3750 msec. This is not scientific, just best result from >> 10 runs. >> Sometimes both timings increase to 5000+ msecs. On average >> slowdown is >> visible >> but tolerable - from 2.9 Mops to 2.6 Mops (absolute numbers are >> still >> awesome!). With Append and NoSync I could get 3.45 Mops on the >> same >> test with >> master build. >> >> However, with WriteMap performance of |master| drops 3x to 10000 >> msec >> or just >> 1 Mops, while for the |v.0.9.14| performance with WriteMap >> improvesto 2350 >> msec or 4.25 Mops. >> >> Is this the cost of convenience or it could be fixed so that >> WriteMap >> still "is faster and uses fewer mallocs" as the docs say? >> >> >> That's pretty much the cost of this patch, it has the biggest impact >> on >> WriteMap usage. In default mode, regular Writes are done to grow the >> file >> so the code path is basically unchanged from before. In WriteMap mode >> the >> file has to be grown explicitly, right before accessing a new page and >> apparently the VirtualAlloc call that does this is expensive. Since >> it's >> the equivalent of both a malloc and a write together, it's actually >> more >> expensive than the default mode. >> >> Please followup to the ITS so this conversation stays with the ticket. >> > > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ >
