On Sat, Feb 09, 2019 at 03:15:30PM +0100, Otto Moerbeek wrote:
> On Sat, Feb 09, 2019 at 12:39:37PM +0100, Peter J. Philipp wrote:
>
> > On Sat, Feb 09, 2019 at 12:01:39PM +0100, Otto Moerbeek wrote:
> > > Why is this a wall? Do your mmaps start failing? With what error code?
> >
> > Well 13G isn't the wall, but I had tried the entire /usr/share/dict/words as
> > A records which would have given more than 200K RRSET's which would have
> > blown up this SIZE considerably since the 30K RRSET's were 13G.
>
> So you're using around 433k per RRSET. That's a lot given that a
> typical RRSET in the wild often is smaller than 100 bytes (no k
> there). I understand to advantages of fixed size data structures, but
> in this case it's not the right way.
>
> -Otto
Thanks Otto. I doodled up a plan in the last few hours, here is the result:
https://centroid.eu/blog/c?article=1549722619
I'm not going to be rash about this, but I may start next week with this this
new concept. Do you think an RRSET like this would be space saving and still
be fast enough? I know I'm losing a bit of speed with all the TAILQ's but I
think I'll work this out with the tree(3) and queue(3) macros.
It's been a long way on getting the database right over the last 15 years on
this daemon. I started with BerkeleyDB backend, and replaced the BDB about
2 years ago with the tree(3). Now hopefully I'll get it right finally. This
year is already gonna rock in terms of milestones, hehe.
Best Regards,
-peter
> >
> > The mmaps failed but due to a stupidity in my own code I could not glance
> > the
> > real errno value (I'll show you why):
> >
> > ----->
> > /* does not exist, create it */
> >
> > map = (char *)mmap(NULL, SIZENODE, PROT_READ|PROT_WRITE,
> > MAP_PRI
> > VATE | MAP_ANON, -1, 0);
> > if (map == MAP_FAILED) {
> > errno = EINVAL;
> > return -1;
> > }
> >
> > <-----
> >
> > I should have logged it before reusing errno, I'll do some tests and get
> > back
> > to you. I am currently not at my workstation as I went and visited my
> > parents
> > since writing the original mail, and I won't be back until monday or so, so
> > any
> > changing of this code will have to wait until then.
> >
> > > You should be able (given ulimits) to mmap up to MAXDSIZ (32G on
> > > amd64) per process.
> >
> > I noticed that RLIMIT_DATA limits had given me problems too. Sizing them to
> > 16G had allowed me to work with the 30K RRSETs.
> >
> > > If you want to reduce SIZE, call munmap(). That's the only way. You
> > > cannot have virtual memory pages that are not accounted for in SIZE.
> > >
> > > -Otto
> >
> > Ahh ok thanks for clearing that up. It looks like I'll have to rewrite the
> > way I store internal data then, if I want to use a LOT of RRSETs in the
> > future.
> > May be better for me too.
> >
> > Thanks Otto!
> >
> > Best Regards,
> > -peter
> >
