--047d7bfcf1deec63470514165cd2 Content-Type: text/plain; charset=UTF-8 My Testing does show it is not worth it for the data size in the db.
Also there is a race condition using sendfile on linux that could happen. It is avoidable but it would require changes to the protocol to avoid it which I am not willing to do. So dismiss my patch with extreme prejudice On Wed, Apr 15, 2015 at 10:37 PM, Howard Chu <[email protected]> wrote: > [email protected] wrote: > >> Full_Name: James Rouzier >> Version: >> OS: >> URL: ftp://ftp.openldap.org/incoming/james-rouzier-150331.patch >> Submission from: (NULL) (192.222.129.11) >> >> >> I am working on a proof of concept of a simple key value store service >> using >> lmdb as the back end. >> I am using sendfile to avoid an extra copy. >> I accomplished this by creating function to extract get the mmap ptr from >> the >> env so I can calculate the offset of data. >> > > I'm quite reluctant to return the mmap pointer. In a WRITEMAP environment > that invites all manner of undetectable/unpreventable corruptions. > > Have you actually measured the performance difference between using > sendfile() here and plain write() or send()? The kernel doesn't physically > copy the written data from user space to kernel space, it just maps in a > particular page. Since LMDB data is already mmap'd, the kernel has very > little work to do since the relevant pages are already present in the > kernel's page tables. > > Ultimately sending data over TCP involves byte-by-byte iteration anyway, > since TCP checksums must be computed on all outgoing data. Doesn't seem to > me like sendfile can be a very significant optimization here. > >> >> I figured that function might be useful to the bigger community. >> >> If it does not fit in the bigger vision of lmdb it is fine not to include >> it. >> >> >> I, James Rouzier, hereby place the following modifications to OpenLDAP >> Software >> (and only these modifications) into the public domain. Hence, these >> modifications may be freely used and/or redistributed for any purpose >> with or >> without attribution and/or other notice. >> >> >> >> > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ > --047d7bfcf1deec63470514165cd2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_quote"><div dir=3D"ltr">My Testing doe= s show it is not worth it for the data size in the db.<br><br><div>Also the= re is a race condition using sendfile on linux that could happen.<br>It is = avoidable but it would require changes to=C2=A0the protocol to avoid it whi= ch I am not willing to do.<br><br>So dismiss my patch with extreme prejudic= e<br><br><br></div></div><div><div><div class=3D"gmail_extra"><br><div clas= s=3D"gmail_quote">On Wed, Apr 15, 2015 at 10:37 PM, Howard Chu <span dir=3D= "ltr"><<a href=3D"mailto:[email protected]"; target=3D"_blank">[email protected]<= /a>></span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:= 0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><a href=3D"mailto:r= [email protected]" target=3D"_blank">[email protected]</a> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Full_Name: James Rouzier<br> Version:<br> OS:<br> URL: <a href=3D"ftp://ftp.openldap.org/incoming/james-rouzier-150331.patch"= target=3D"_blank">ftp://ftp.openldap.org/incoming/james-rouzier-150331.pat= ch</a><br> Submission from: (NULL) (192.222.129.11)<br> <br> <br> I am working on a proof of concept of a simple key value store service usin= g<br> lmdb as the back end.<br> I am using sendfile to avoid an extra copy.<br> I accomplished this by creating function to extract get the mmap ptr from t= he<br> env so I can calculate the offset of data.<br> </blockquote> <br> I'm quite reluctant to return the mmap pointer. In a WRITEMAP environme= nt that invites all manner of undetectable/unpreventable corruptions.<br> <br> Have you actually measured the performance difference between using sendfil= e() here and plain write() or send()? The kernel doesn't physically cop= y the written data from user space to kernel space, it just maps in a parti= cular page. Since LMDB data is already mmap'd, the kernel has very litt= le work to do since the relevant pages are already present in the kernel= 9;s page tables.<br> <br> Ultimately sending data over TCP involves byte-by-byte iteration anyway, si= nce TCP checksums must be computed on all outgoing data. Doesn't seem t= o me like sendfile can be a very significant optimization here.<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <br> I figured that function might be useful to the bigger community.<br> <br> If it does not fit in the bigger vision of lmdb it is fine not to include i= t.<br> <br> <br> I, James Rouzier, hereby place the following modifications to OpenLDAP Soft= ware<br> (and only these modifications) into the public domain. Hence, these<br> modifications may be freely used and/or redistributed for any purpose with = or<br> without attribution and/or other notice.<br> <br> <br> <br><span><font color=3D"#888888"> </font></span></blockquote><span><font color=3D"#888888"> <br> <br> -- <br> =C2=A0 -- Howard Chu<br> =C2=A0 CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"= http://www.symas.com"; target=3D"_blank">http://www.symas.com</a><br> =C2=A0 Director, Highland Sun=C2=A0 =C2=A0 =C2=A0<a href=3D"http://highland= sun.com/hyc/" target=3D"_blank">http://highlandsun.com/hyc/</a><br> =C2=A0 Chief Architect, OpenLDAP=C2=A0 <a href=3D"http://www.openldap.org/p= roject/" target=3D"_blank">http://www.openldap.org/project/</a><br> </font></span></blockquote></div><br></div> </div></div></div><br></div> --047d7bfcf1deec63470514165cd2--
