On Sat, Feb 09, 2019 at 11:34:41AM +0100, Peter J. Philipp wrote:
> Hi misc@,
>
> I have a program, a DNS server. It has a database to hold internal data.
> Right now it's very inneficient in the way it uses memory. Let me explain.
>
> If you know what an RRSET is it's all the RR records under one name. Like in
> the OpenBSD.ORG name there is a SOA, NS, A RR's and so on. That's an RRSET.
>
> Internally in my program I reserve space for an RRSET to all the maximum
> records (around maximum 15 A RR's or so) and there is 15 or so types of
> RR's in support with my program. As you can guess the RRSET becomes huge.
>
> When designing the database for this I thought "hey what if I mmap this space
> and only those parts that get written to it will actually be used". And
> that's
> what I did. Only I run into a wall. With close to 30000 RRSET's I get this
> in top:
>
> Memory: Real: 1233M/3468M act/tot Free: 28G Cache: 1197M Swap: 0K/32G
>
> PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND
> 74465 _ddd 2 0 13G 163M sleep/1 select 0:01 0.00%
> delphinusdn
> 31148 _ddd 2 0 13G 1116K sleep/1 select 0:01 0.00%
> delphinusdn
> ..
>
> So you see the SIZE is 13 GB, but the actual resident memory is 163M. So now
> I come to my final question:
>
> Is there a way to claim back part of that 13G?
>
> If not is there a way to overcommit? I googled on this and many people think
> it's bad and I don't want to start a religious war right now.
>
> In the worst case scenario I'll have to rewrite my underlying database, and
> I'm
> a little lazy, if I do have to do that it won't be for another year+.
>
> I looked at madvise(2) and I'm a little confused as to what I can use for
> this.
>
> Best regards,
> -peter
>
Why is this a wall? Do your mmaps start failing? With what error code?
You should be able (given ulimits) to mmap up to MAXDSIZ (32G on
amd64) per process.
If you want to reduce SIZE, call munmap(). That's the only way. You
cannot have virtual memory pages that are not accounted for in SIZE.
-Otto
