FWIW, I use (on a 32-bit i686 appliance) a 33MiB ads/pron/warez blocklist of 1.2M domains in the form "local=/FQDN/" (that is, the domains do not exist at all for me; I'm OK with seeing whitespace). The virtual size (from 'ps aux') of the running dnsmasq is 175MiB. Resolution time is a hair slower, but still very acceptable.
Most of the domains in the list are longer than 10 bytes. There are no SHA1/SHA256 collisions in the first 10 bytes of the FQDN hashes; but there are in the first 9 bytes, so it's close. Does dnsmasq store the actual FQDN in RAM? If so, could it instead store the first 10 bytes of a hash of the domain? This would help reduce RAM usage by maybe 40%. Huge blocklists are going to require lots of RAM. If your system has limited RAM, you really have two options. (1) Plead with the device manufacturers to boost RAM to 1GiB. (2) Use an RPi to run dnsmasq; a personal challenge for me would be to run it transparently between my F/W and ISP. One question remains. What does dnsmasq do that increases RAM usage by about 160MiB for 25MiB of FQDNs? Though it may be completely reasonable, it seems a trifle much from a soaring vulture. Neal On Tue, 19 Nov 2024 18:31:42 +0200 Ercolino de Spiacico <bellocar...@hotmail.com> wrote: > In the context of Adblock, I noticed that our adblock script can handle > relatively well about 10MB of blockfile which is about 7.8% of the > device RAM (128MB), after that the resolution time increases > exponentially to the point where the DNS resolution times-out and more > importantly the device becomes unstable. > > I was trying to understand the root cause on why we couldn't have a > larger blockfile which is compiled in local=/example.com/# format. The > RAM, cache and buffer demand stay relatively low when the system becomes > unstable. > > In the aim to investigate this further, I run a test creating a bogus > blockfile of the size of about 100MB (and 3.6M lines/domains). Here are > the grep results to fetch info from it: > > root@router:/mnt/USB/adblock# time grep 'mytestdomain.com' > /mnt/USB/adblock/test2 > mytestdomain.com > real 0m 2.44s > user 0m 0.49s > sys 0m 0.43s > > root@router:/mnt/USB/adblock# cp /mnt/USB/adblock/test2 /tmp/test2 > > root@router:/mnt/USB/adblock# time grep 'mytestdomain.com' /tmp/test2 > mytestdomain.com > real 0m 0.65s > user 0m 0.39s > sys 0m 0.20s > > What I'm trying to demonstrate here is that a USB2 device can extract a > domain via grep in 2.44sec, and if that file was to be placed in RAM > (/tmp is mapped in RAM on devices with squashfs) it's just 0.65sec. > Admittedly /tmp compresses the content so the 100MB uses about 38MB, > still the point on performance is valid and tells me we could fit > 200-250MB blockfile if ever needed, looking at RAM capacity only. > > As a point of discussion/improvement, I believe dnsmasq uploads the > custom config (so the blockfile in this case) into RAM, why do we > experience poor resolution performance and system instability at just 10MB? > > Considering the system grep is so fast, could this be an alternative > method for dnsmasq to address locally defined domains? If given the > possibility, I would be very happy to map a file in RAM knowing that > this is handled differently from the "standard" conf-file. > > I suppose the first step would be to fully understand where the > limitation we currently have comes from. > > Then, I'm not suggesting we should re-invent the wheel, but perhaps > there's a margin for a new directive whose behavior is a simple grep > against a mapped file to be used as an authority for those domains? > Might be restricted to blocking only (returning NX or 0.0.0.0 or > 127.0.0.1)? Not sure what the secondary implications of such an idea > would be, but I'll be glad to hear some comments/opinions on this topic. > > > Thanks > > _______________________________________________ > Dnsmasq-discuss mailing list > Dnsmasq-discuss@lists.thekelleys.org.uk > https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss _______________________________________________ Dnsmasq-discuss mailing list Dnsmasq-discuss@lists.thekelleys.org.uk https://lists.thekelleys.org.uk/cgi-bin/mailman/listinfo/dnsmasq-discuss
