On Sat, Feb 09, 2019 at 03:15:30PM +0100, Otto Moerbeek wrote: > On Sat, Feb 09, 2019 at 12:39:37PM +0100, Peter J. Philipp wrote: > > > On Sat, Feb 09, 2019 at 12:01:39PM +0100, Otto Moerbeek wrote: > > > Why is this a wall? Do your mmaps start failing? With what error code? > > > > Well 13G isn't the wall, but I had tried the entire /usr/share/dict/words as > > A records which would have given more than 200K RRSET's which would have > > blown up this SIZE considerably since the 30K RRSET's were 13G. > > So you're using around 433k per RRSET. That's a lot given that a > typical RRSET in the wild often is smaller than 100 bytes (no k > there). I understand to advantages of fixed size data structures, but > in this case it's not the right way. > > -Otto
Hi Otto & Misc, Good news I was programming all week long and managed to compact the database. I still have to test the works of it as I'm sure there is a few bugs lurking, but at first glance it looks like I was able to shrink the size of an RRSET down to ~1800 bytes, and it's pretty fast too. I added all of /usr/share/dict/words as A RR's and thus as their own RRSET to the existing 30K I had and the memory footprint was around 456M virtual size. I'm very happy I did this, and not the problematic query I had in the beginning of this thread. Thanks Otto! Regards, -peter
