Send inn-workers mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.isc.org/mailman/listinfo/inn-workers
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of inn-workers digest..."
Today's Topics:
1. Re: [patch] sm -s -R and overview mechanisms (Julien ?LIE)
2. Re: [patch] sm -s -R and overview mechanisms (Bo Lindbergh)
3. Re: overview mechanisms (Bo Lindbergh)
4. Re: overview mechanisms (Julien ?LIE)
5. Re: overview mechanisms (Bo Lindbergh)
6. Re: overview mechanisms (Julien ?LIE)
7. Re: Please generate the ChangeLog file also for snapshots
(Julien ?LIE)
----------------------------------------------------------------------
Message: 1
Date: Mon, 23 Nov 2020 13:08:29 +0100
From: Julien ?LIE <[email protected]>
To: [email protected]
Subject: Re: [patch] sm -s -R and overview mechanisms
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8; format=flowed
Hi Bo,
> One build-and-install later, a size comparison becomes possible:
>> -rwxr-xr-x 1 root wheel 1700816 Nov 23 09:49
>> /usr/local/BerkeleyDB.18.1/lib/libdb-18.1.dylib
>> -rwxr-xr-x 1 root wheel 1735808 Oct 13 10:04
>> /usr/local/lib/libsqlite3.0.dylib
>
> Code size is not a strong argument against SQLite.
Was it at the time Berkeley DB choice for INN was made?
I'm also wondering what the pros are for a DBMS like SQLite or Berkeley
DB. For a new user, INSTALL does not necessarily help to choose. Do
you see any thoughts that should be added?
For a full-text feed nowadays, on modern hardware, would we see a real
difference between the 3 mechanisms in termes of speed or maximum number
of users at the same time?
Has anyone had a chance to benchmark the 3 of them?
Quoting INSTALL:
%%
There are three overview mechanisms to choose from:
tradindexed
It is very fast for readers, but it has to update two files for each
incoming article and can be quite slow to write.
buffindexed
It can keep up with a large feed more easily, since it uses large
buffers to store all overview information, but it's somewhat slower for
readers (although not as slow as the unified overview in INN 2.2). You
will need to create the buffers for it to use (very similar to creating
CNFS buffers) and list the available buffers in buffindexed.conf.
ovdb
It stores overview data in a Berkeley DB database; it's fast and very
robust, but may require more disk space.
%%
And according to ovdb documentation:
%%
The ovdb database may take up more disk space for a given spool than the
other overview methods. Plan on needing at least 1.1 KB for every
article in your spool (not counting crossposts). So, if you have 5
million articles, you'll need at least 5.5 GB of disk space for ovdb.
With compression enabled, this estimate changes to 0.7 KB per article.
Plus, you'll need additional space for transaction logs: at least 100 MB.
%%
--
Julien ?LIE
??Pourquoi apprendre ? calculer la surface d'un losange?? Au cours de ma
vie, je n'ai jamais compt? aucun losange parmi mes relations.??
(Jacques Sternberg)
------------------------------
Message: 2
Date: Mon, 23 Nov 2020 16:20:17 +0100
From: Bo Lindbergh <[email protected]>
To: [email protected]
Subject: Re: [patch] sm -s -R and overview mechanisms
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Quoth Julien ?LIE <[email protected]>:
>> One build-and-install later, a size comparison becomes possible:
>>> -rwxr-xr-x 1 root wheel 1700816 Nov 23 09:49
>>> /usr/local/BerkeleyDB.18.1/lib/libdb-18.1.dylib
>>> -rwxr-xr-x 1 root wheel 1735808 Oct 13 10:04
>>> /usr/local/lib/libsqlite3.0.dylib
>> Code size is not a strong argument against SQLite.
>
> Was it at the time Berkeley DB choice for INN was made?
Not applicable, since ovdb is older than the entire SQLite project.
> I'm also wondering what the pros are for a DBMS like SQLite or Berkeley DB.
> For a new user, INSTALL does not necessarily help to choose. Do you see any
> thoughts that should be added?
Just an anecdotal data point: ovdb was much faster than buffindexed
at expiration when I switched overview methods for my private server
back in 2010.
Now that I can build ovdb again, I'll definitely run benchmark tests
comparing ovdb and ovsqlite.
/Bo Lindbergh
------------------------------
Message: 3
Date: Mon, 23 Nov 2020 17:19:22 +0100
From: Bo Lindbergh <[email protected]>
To: [email protected]
Subject: Re: overview mechanisms
Message-ID: <[email protected]>
Content-Type: text/plain; charset=us-ascii
I myself said:
> Now that I can build ovdb again, I'll definitely run benchmark tests
> comparing ovdb and ovsqlite.
Quick and dirty test running "makehistory -O -x" on 544504 entries
with the cycbuffs and the overview data both stored on the same
external rotating rust device.
method real time CPU time disk space
======== ========= ======== ==========
ovdb 329.80 31.43 402M
ovsqlite 333.51 54.73 200M
Development progress: feature complete and ready for the hard part:
documentation.
/Bo Lindbergh
------------------------------
Message: 4
Date: Mon, 23 Nov 2020 22:33:51 +0100
From: Julien ?LIE <[email protected]>
To: [email protected]
Subject: Re: overview mechanisms
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8; format=flowed
Hi Bo,
> Quick and dirty test running "makehistory -O -x" on 544504 entries
> with the cycbuffs and the overview data both stored on the same
> external rotating rust device.
>
> method real time CPU time disk space
> ======== ========= ======== ==========
> ovdb 329.80 31.43 402M
> ovsqlite 333.51 54.73 200M
Thanks for the benchmark!
Are ovdb and INN built with zlib support, and "compress" set to true in
ovdb.conf?
--
Julien ?LIE
??Les grands artistes n'ont pas de patrie.?? (Alfred de Musset)
------------------------------
Message: 5
Date: Tue, 24 Nov 2020 01:42:10 +0100
From: Bo Lindbergh <[email protected]>
To: Julien ?LIE <[email protected]>
Cc: [email protected]
Subject: Re: overview mechanisms
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Quoth Julien ?LIE <[email protected]>:
>
> Are ovdb and INN built with zlib support, and "compress" set to true in
> ovdb.conf?
Yes.
If I turn off compression, disk usage increases to 494 MB for ovdb
and 469 MB for ovsqlite. I think the major difference is that
ovdb doesn't try to compress entries smaller than 600 bytes.
/Bo Lindbergh
------------------------------
Message: 6
Date: Tue, 24 Nov 2020 08:10:06 +0100
From: Julien ?LIE <[email protected]>
To: [email protected]
Subject: Re: overview mechanisms
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8; format=flowed
Hi Bo,
>> Are ovdb and INN built with zlib support, and "compress" set to true in
>> ovdb.conf?
>
> Yes.
>
> If I turn off compression, disk usage increases to 494 MB for ovdb
> and 469 MB for ovsqlite. I think the major difference is that
> ovdb doesn't try to compress entries smaller than 600 bytes.
It would indeed explain that. Maybe this hard-coded size should be
reduced? Overview space seems to be twice as much as it could be,
according to the comparison with SQLite.
--
Julien ?LIE
??Les grands artistes n'ont pas de patrie.?? (Alfred de Musset)
------------------------------
Message: 7
Date: Tue, 24 Nov 2020 08:59:14 +0100
From: Julien ?LIE <[email protected]>
To: [email protected]
Subject: Re: Please generate the ChangeLog file also for snapshots
Message-ID: <[email protected]>
Content-Type: text/plain; charset=windows-1252; format=flowed
Hi Marco,
> I suggest that you have a look at the systemd integration patch but wait
> to merge it, because socket activation does not work yet with xexec.
Do you happen to have a new patch for socket activation?
>> In innd/python.c, you should free(path) before returning.
> Noted, thank you.
> I think that this should be merged as well after some more testing.
In the no_unused_python patch, when Python filter files are not found,
logs are changed from error to notice:
https://salsa.debian.org/md/inn2/-/raw/master/debian/patches/no_unused_python
syslog(L_NOTICE, "python is not initialized");
syslog(L_NOTICE, "%s not installed", path);
Shouldn't the same level (notice) be also used for Perl filters?
In lib/perl.c, PERLsetup logs it as en error:
syslog(L_ERROR,"SERVER perl loading %s failed: %s",
startupfile, SvPV(errsv, PL_na));
--
Julien ?LIE
??Attention?! Dalles glissantes?!?? (Ast?rix)
------------------------------
Subject: Digest Footer
_______________________________________________
inn-workers mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/inn-workers
------------------------------
End of inn-workers Digest, Vol 125, Issue 10
********************************************
