Send inn-workers mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.isc.org/mailman/listinfo/inn-workers
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of inn-workers digest..."
Today's Topics:
1. Re: ovsqlite (Bo Lindbergh)
2. Re: ovsqlite (Bo Lindbergh)
3. Re: ovsqlite (zlib edge-cases) (Bo Lindbergh)
4. Re: ovsqlite (Bo Lindbergh)
5. Re: Overview methods and rebuild documentation (Bo Lindbergh)
6. Re: ovsqlite (Julien ?LIE)
7. Re: Overview methods and rebuild documentation (Julien ?LIE)
----------------------------------------------------------------------
Message: 1
Date: Wed, 23 Dec 2020 20:20:52 +0100
From: Bo Lindbergh <[email protected]>
To: [email protected]
Subject: Re: ovsqlite
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Quoth Julien ?LIE <[email protected]>:
> As a matter of fact, PRIu64 is only used for compressed ovsqlite overview
> data. And we would then use OVSQLITE_USE_DICTIONARY instead of
> USE_DICTIONARY (unconditionally defined otherwise in ovsqlite-server.c).
>
> Would it suit you? Or did you have another idea in mind for USE_DICTIONARY
> and PRIu64 availability?
USE_DICTIONARY exists just to make it easy to do with/without testing
to check if it saves measurable amounts of disk space (it does!).
It should probably be removed before the public release. Or it could be turned
into a configuration option: three compression levels (none, plain, dictionary).
As you noted, PRIu64 is only used once. There is also a single use of PRId64.
I'm preparing a patch that eliminates both of them and thus any need for
<inttypes.h>. Configuration simplification!
/Bo Lindbergh
------------------------------
Message: 2
Date: Wed, 23 Dec 2020 20:47:36 +0100
From: Bo Lindbergh <[email protected]>
To: [email protected]
Subject: Re: ovsqlite
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Quoth Julien ?LIE <[email protected]>:
> Article lines processed 3252700
> Articles dropped 57
> Overview index dropped 2673
>
> I usually have the same numbers for articles dropped and overview index
> dropped. Maybe the difference is normal for the first run on a new overview
> database built with makehistory? (I can't tell as I do not know what was
> dropped exactly.)
No idea. The expiration logic is mostly copied from ovdb. I tested it
by removing the oldest 1/10 of each group's articles with sm -d
before running expireover, and it seemed to remove exactly the right set
of article rows from the database. (This was with cnfs and no group-based
expiration.)
/Bo Lindbergh
------------------------------
Message: 3
Date: Wed, 23 Dec 2020 21:19:18 +0100
From: Bo Lindbergh <[email protected]>
To: [email protected]
Subject: Re: ovsqlite (zlib edge-cases)
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Quoth Julien ?LIE <[email protected]>:
> A suggestion for close_db():
>
> #ifdef HAVE_ZLIB
> if (use_compression) {
> inflateEnd(&inflation);
> deflateEnd(&deflation);
> buffer_free(flate);
> }
> #endif
Good idea! I'll include it in the upcoming patch.
The compression code might be excessively clever, so I'll add some clarifying
comments. Brief version:
1. deflation.avail_out is carefully set so that deflate fails to process
everything in one pass exactly when compression doesn't save any space.
2. inflation.avail_out is set to the uncompressed data size, so if inflate
fails to process everything in one pass, the compressed data must be
corrupted.
/Bo Lindbergh
------------------------------
Message: 4
Date: Wed, 23 Dec 2020 21:37:24 +0100
From: Bo Lindbergh <[email protected]>
To: [email protected]
Subject: Re: ovsqlite
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Quoth Julien ?LIE <[email protected]>:
> Note that I've renamed the --with-sqlite3 configure flag to --with-sqlite out
> of consistency with our other configure flags, that do not mention a version.
I'd say the 3 is part of the name rather than a version.
Include file name: sqlite3.h
Library file name: libsqlite3.a
> Do you prefer Bo or /Bo (like in your signature) ?
"Bo". The "/" is just a nonstandard separator.
> Do you want your e-mail to be added in the HISTORY section of POD files?
It's visible in the mailing list archive anyway, so go ahead.
> +Bo Lindbergh has implemented a new overview storage method based on
> +SQLite, known for its long-term stability and compatibility. Fast and
> +reliable, this new SQLite-based method is a perfect choice to store
> +overview data.
I don't know about "fast". It's slower than ovdb when writing, after all.
It's faster than ovdb when reading whole ranges, since it transfers
overview data between ovsqlite-server and nnrpd in 128-kilobyte chunks.
The rest looks good. Did you select a lower default transaction row limit?
/Bo Lindbergh
------------------------------
Message: 5
Date: Wed, 23 Dec 2020 22:05:21 +0100
From: Bo Lindbergh <[email protected]>
To: "[email protected]" <[email protected]>
Subject: Re: Overview methods and rebuild documentation
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Quoth Julien ?LIE <[email protected]>:
>
> => OK with that new "OVERVIEW REBUILD" section ?
Maybe mention
ctlinnd renumber ""
for when you want to be extra sure that the active file and the overview are
in sync?
/Bo Lindbergh
------------------------------
Message: 6
Date: Wed, 23 Dec 2020 22:06:03 +0100
From: Julien ?LIE <[email protected]>
To: [email protected]
Subject: Re: ovsqlite
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8; format=flowed
Hi Bo,
>> Note that I've renamed the --with-sqlite3 configure flag to
>> --with-sqlite out of consistency with our other configure flags, that do
>> not mention a version.
>
> I'd say the 3 is part of the name rather than a version.
> Include file name: sqlite3.h
> Library file name: libsqlite3.a
Hmm... Like we have --with-krb5, looking for libkrb5, for Kerberos V5.
But unlike --with-sasl, looking for libsasl2, for Cyrus SASL...
@Russ, do you plan on renaming sqlite.m4 to sqlite3.m4 in rra-c-util too?
>> Do you want your e-mail to be added in the HISTORY section of POD files?
>
> It's visible in the mailing list archive anyway, so go ahead.
OK, noted.
>> +Bo Lindbergh has implemented a new overview storage method based on
>> +SQLite, known for its long-term stability and compatibility. Fast and
>> +reliable, this new SQLite-based method is a perfect choice to store
>> +overview data.
>
> I don't know about "fast". It's slower than ovdb when writing, after all.
> It's faster than ovdb when reading whole ranges, since it transfers
> overview data between ovsqlite-server and nnrpd in 128-kilobyte chunks.
"Robust and faster at reading ranges of overview data, but somewhat
slower at writing" would be a better description?
> The rest looks good. Did you select a lower default transaction row limit?
I kept the default of 10,000 transactions. Anyway, my news server
receives far less articles than this, so the time limit of 10s always
applies.
I increased it during makehistory but I'm unsure it improves speed
(commits are probably faster than makehistory collecting the next bunch
of overview lines to add). Maybe that suggestion should not be kept
after all?
--
Julien ?LIE
??C'est de ma faute ? moi si les portes ne sont pas ? la hauteur de mes
menhirs???? (Ast?rix)
------------------------------
Message: 7
Date: Wed, 23 Dec 2020 22:10:53 +0100
From: Julien ?LIE <[email protected]>
To: [email protected]
Subject: Re: Overview methods and rebuild documentation
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8; format=flowed
Hi Bo,
>> => OK with that new "OVERVIEW REBUILD" section ?
>
> Maybe mention
>
> ctlinnd renumber ""
>
> for when you want to be extra sure that the active file and the overview are
> in sync?
As rc.news takes care of renumbering on the first run after a rebuild,
would it be in case someone starts INN without rc.news or a bug in
makehistory?
I bet it is the reason of the "extra sure" :-)
--
Julien ?LIE
??Ex nihilo nihil.?? (Perse)
------------------------------
Subject: Digest Footer
_______________________________________________
inn-workers mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/inn-workers
------------------------------
End of inn-workers Digest, Vol 126, Issue 17
********************************************
