Send inn-workers mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.isc.org/mailman/listinfo/inn-workers
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of inn-workers digest..."
Today's Topics:
1. Re: ovsqlite (Bo Lindbergh)
2. Re: QIO_BUFFERSIZE (Julien ?LIE)
3. Re: ovsqlite (Julien ?LIE)
4. Re: ovsqlite (Bo Lindbergh)
5. Re: ovsqlite (Julien ?LIE)
----------------------------------------------------------------------
Message: 1
Date: Tue, 22 Dec 2020 17:16:32 +0100
From: Bo Lindbergh <[email protected]>
To: "[email protected]" <[email protected]>
Subject: Re: ovsqlite
Message-ID: <[email protected]>
Content-Type: text/plain; charset="utf-8"
Quoth Julien ?LIE <[email protected]>:
> I'm now encountering two shift-negative-value errors (building with "make
> warnings"):
>
> ovsqlite/ovsqlite-server.c: Dans la fonction ? pack_length ?:
> ovsqlite/ovsqlite-server.c:482:27: erreur: d?calage vers la gauche d'une
> valeur n?gative [-Werror=shift-negative-value]
> 482 | *--walk = length | (~0<<(9-lenlen));
> | ^~
> ovsqlite/ovsqlite-server.c: Dans la fonction ? unpack_length ?:
> ovsqlite/ovsqlite-server.c:501:20: erreur: d?calage vers la gauche d'une
> valeur n?gative [-Werror=shift-negative-value]
> 501 | length = c&~(~0<<(8-lenlen));
> | ^~
Marking some constants as unsigned should fix that
(included in the attached patch).
> ovsqlite/ovsqlite-server.c: Dans la fonction ? open_db ?:
> ovsqlite/ovsqlite-server.c:591:9: erreur: ? SQLITE_PREPARE_PERSISTENT ? non
> d?clar? (premi?re utilisation dans cette fonction)
> 591 | SQLITE_PREPARE_PERSISTENT,
> | ^~~~~~~~~~~~~~~~~~~~~~~~~
>
> For this last one, it seems that SQLITE_PREPARE_PERSISTENT was introduced in
> SQLite 3.20.0.
> Should INN require that recent version (August 2017) at configure time?
> Or #if SQLITE_VERSION_NUMBER >= 3020000 added there?
> And also around sqlite3_prepare_v3 in sqlite-helper.c and use
> sqlite3_prepare_v2 before 3.20.0?
The prepare flags argument is useful but not essential. Don't change
the version requirement; add workarounds to sqlite-helper.[ch]
(patch attached).
> If I do not take into account the first 2 warnings, and manually use
> sqlite3_prepare_v2, the remaining of the build works fine with SQLite 3.8.7.1
> from Debian Jessie. Good news!
The minimum version requirement is based only on SQL language features.
I ought to download SQLite 3.8.2 and test against it at some point.
/Bo Lindbergh
-------------- next part --------------
A non-text attachment was scrubbed...
Name: delta2.diff.gz
Type: application/x-gzip
Size: 828 bytes
Desc: not available
URL:
<https://lists.isc.org/pipermail/inn-workers/attachments/20201222/92b086df/attachment-0001.bin>
------------------------------
Message: 2
Date: Tue, 22 Dec 2020 19:43:15 +0100
From: Julien ?LIE <[email protected]>
To: [email protected]
Subject: Re: QIO_BUFFERSIZE
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8; format=flowed
Hi all,
> #define QIO_BUFFERSIZE (32 * 1024)
Thanks for that patch Russ.
Just this message to tell that I've just rebuilt my overview data (yes,
en route to ovsqlite!).
I have 3,249,249 articles in my overview. During the rebuild, I only
got 2 errors of too long overview lines (> 32 KB - the new size of
QIO_BUFFERSIZE).
One is a spam (37028 bytes of overview for
<[email protected]> with
a giant subject header field) and the other one is a test I did in 2010
in a local newsgroup (45827 bytes of overview with an extra-long
User-Agent header field, advertised in extra overview.fmt fields).
So, the new QIO buffer size seems very good. At least for the groups I
carry (notably fr.*).
--
Julien ?LIE
??Ce serait un tunnel pour aller de la Gaule en Bretagne.?? (Ast?rix)
------------------------------
Message: 3
Date: Tue, 22 Dec 2020 20:37:17 +0100
From: Julien ?LIE <[email protected]>
To: "[email protected]" <[email protected]>
Subject: Re: ovsqlite
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8; format=flowed
Hi Bo,
>> ovsqlite/ovsqlite-server.c: Dans la fonction ? unpack_length ?:
>> ovsqlite/ovsqlite-server.c:501:20: erreur: d?calage vers la gauche d'une
>> valeur n?gative [-Werror=shift-negative-value]
>>? 501 |???? length = c&~(~0<<(8-lenlen));
>>????? |??????????????????? ^~
>
> Marking some constants as unsigned should fix that
> (included in the attached patch).
Yes, indeed.
>> For this last one, it seems that SQLITE_PREPARE_PERSISTENT was introduced in
>> SQLite 3.20.0.
>> Should INN require that recent version (August 2017) at configure time?
>> Or #if SQLITE_VERSION_NUMBER >= 3020000 added there?
>
> The prepare flags argument is useful but not essential.? Don't change
> the version requirement; add workarounds to sqlite-helper.[ch]
> (patch attached).
Works like a charm!
I've rebuilt my overview data earlier this day, to switch to ovsqlite :)
It is as straight-forward as:
1- install ovsqlite.conf and eventually change its default settings
(though ovsqlite also works without that file, and just complains in the
logs)
2- run "ovsqlite-server"
3- run "makehistory -O -x -F"
I have 3,249,249 articles in my overview (according to the latest run of
expireover).
As for sizes:
-rw-r--r-- 1 news news 5,1G d?c. 22 17:36 ovsqlite.db
-rw-r--r-- 1 news news 22M d?c. 22 17:36 ovsqlite.db-journal
My previous tradindexed overview was 2,9 GB in size (sum of all
subdirectories). It seems that ovsqlite with uncompressed overview data
takes almost twice as much space, but that's not a big deal for me.
For benchmarking comparison, I rebuilt it with compression on:
-rw-r--r-- 1 news news 1,6G d?c. 22 18:26 ovsqlite.db
-rw-r--r-- 1 news news 17M d?c. 22 18:26 ovsqlite.db-journal
ovsqlite then performs better than tradindexed.
The ratio between compressed and uncompressed ovsqlite data is about 1/3.
However, I think I have spotted an issue with ovsqlite.
I had 30 "cannot write overview data" errors from makehistory. To make
sure it ovsqlite-related, I did 4 rebuilds:
1/ uncompressed ovsqlite => 30 write errors
2/ compressed ovsqlite => 30 write errors (exactly the same)
3/ uncompressed ovsqlite, with a modified makehistory so as to keep
overview data => 30 write errors (exactly the same)
4/ tradindexed => 0 error
For instance, this one:
fr.rec.photo.numerique: 1493646864 0
@04020059073E000600000000000000000000@
=?iso-8859-1?Q?Forcer_r=E9solution_16:9_sous_XP_pour_affichage_sur_TV_LCD?=
"Alf92" <[email protected]> Wed, 30 Mar 2011 00:53:33 +0200
<[email protected]> 2545 34 Xref:
news.trigofacile.com
fr.rec.photo.numerique:87727 fr.rec.son-image.video.materiel:4634
microsoft.public.fr.windowsxp:86375 injection-info:
mx01.eternal-september.org; posting-host="1VpS/qOBiAXpLThHEQ8yMA";
logging-data="29841";
mail-complaints-to="[email protected]";
posting-account="U2FsdGVkX19016eiCDPk9wyjdfTtK+5G" Path:
news2.trigofacile.com!news.trigofacile.com!weretis.net!feeder4.news.weretis.net!news-transit.tcx.org.uk!eternal-september.org!feeder.eternal-september.org!.POSTED!not-for-mail
Xref: news.trigofacile.com fr.rec.photo.numerique:87727
fr.rec.son-image.video.materiel:4634 microsoft.public.fr.windowsxp:86375
This article appears in my tradindexed overview (in
fr.rec.son-image.video.materiel.DAT), just rebuilt.
However, it is not in my ovsqlite overview.
I think the issue comes from the fact that the fr.rec.photo.numerique
newsgroup no longer exists. Yet, one of the newsgroup it was
crossposted to still exists.
350|5350|5827|384|0|0|fr.rec.son-image.video.materiel|y
select * from artinfo where artnum=4634 and groupid=350;
=> no article
I also have an example of crosspost where the 1st group still exists,
but not the 2nd.
I've not checked all 30 write errors, but the ones I did are similar
crossposts to a removed newsgroup from active.
Full log (for future reference for me):
makehistory: cannot write overview data
"@030346523300000000000000001000000002@"
makehistory: cannot write overview data
"@030346523400000000000000001000000002@"
makehistory: cannot write overview data
"@04020059073900DB00000000000000000000@"
makehistory: cannot write overview data
"@04020059073900DC00000000000000000000@"
makehistory: cannot write overview data
"@04020059073900DD00000000000000000000@"
makehistory: cannot write overview data
"@04020059073900DE00000000000000000000@"
makehistory: cannot write overview data
"@04020059073900DF00000000000000000000@"
makehistory: cannot write overview data
"@04020059073900E000000000000000000000@"
makehistory: cannot write overview data
"@04020059073900E100000000000000000000@"
makehistory: cannot write overview data
"@04020059073900E200000000000000000000@"
makehistory: cannot write overview data
"@04020059073900E300000000000000000000@"
makehistory: cannot write overview data
"@04020059073900EC00000000000000000000@"
makehistory: cannot write overview data
"@04020059073900ED00000000000000000000@"
makehistory: cannot write overview data
"@04020059073900D600000000000000000000@"
makehistory: cannot write overview data
"@04020059073A00BF00000000000000000000@"
makehistory: cannot write overview data
"@04020059073E000600000000000000000000@"
makehistory: cannot write overview data
"@04020059073E000700000000000000000000@"
makehistory: cannot write overview data
"@04020059073E000800000000000000000000@"
makehistory: cannot write overview data
"@04020059073E000900000000000000000000@"
makehistory: cannot write overview data
"@0201590739421D5100000000000000000000@"
makehistory: cannot write overview data
"@020159073974356600000000000000000000@"
makehistory: cannot write overview data
"@0201590739C646D900000000000000000000@"
makehistory: cannot write overview data
"@020159073974357100000000000000000000@"
makehistory: cannot write overview data
"@020159073974356A00000000000000000000@"
makehistory: cannot write overview data
"@020159073A19651800000000000000000000@"
makehistory: cannot write overview data
"@020159073E7436B300000000000000000000@"
makehistory: cannot write overview data
"@020159073CD9F9E100000000000000000000@"
makehistory: cannot write overview data
"@05000000024B000000000000020000000000@"
makehistory: cannot write overview data
"@05000000024B00000000000001F900000000@"
makehistory: cannot write overview data
"@05000000024B000000000000020400000000@"
--
Julien ?LIE
??Ce serait un tunnel pour aller de la Gaule en Bretagne.?? (Ast?rix)
------------------------------
Message: 4
Date: Tue, 22 Dec 2020 22:25:08 +0100
From: Bo Lindbergh <[email protected]>
To: "[email protected]" <[email protected]>
Subject: Re: ovsqlite
Message-ID: <[email protected]>
Content-Type: text/plain; charset="utf-8"
Quoth Julien ?LIE <[email protected]>:
>
> I think the issue comes from the fact that the fr.rec.photo.numerique
> newsgroup no longer exists. Yet, one of the newsgroup it was crossposted to
> still exists.
Looking at the add methods of the other overview methods, they all seem
to handle unknown groups by silently returning success, which I hadn't noticed
before. Is there a document that I missed which describes the expected
semantics of overview methods? Should the add method handle duplicate
groupname:artnum entries by replacing rather than failing?
Patch: make the add method return success in case of an unknown group.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: delta3.diff.gz
Type: application/x-gzip
Size: 273 bytes
Desc: not available
URL:
<https://lists.isc.org/pipermail/inn-workers/attachments/20201222/c528e2d4/attachment-0001.bin>
------------------------------
Message: 5
Date: Tue, 22 Dec 2020 23:41:31 +0100
From: Julien ?LIE <[email protected]>
To: "[email protected]" <[email protected]>
Subject: Re: ovsqlite
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8; format=flowed
Hi Bo,
>> I think the issue comes from the fact that the fr.rec.photo.numerique
>> newsgroup no longer exists.? Yet, one of the newsgroup it was crossposted to
>> still exists.
>
> Looking at the add methods of the other overview methods, they all seem
> to handle unknown groups by silently returning success, which I hadn't
> noticed
> before.? Is there a document that I missed which describes the expected
> semantics of overview methods?? Should the add method handle duplicate
> groupname:artnum entries by replacing rather than failing?
>
> Patch: make the add method return success in case of an unknown group.
Oh, thanks for the patch. That's perfect!
I've rebuilt my overview, and only got 2 write errors from makehistory
(both of them were duplicate overview entries, for a reason I do not
know, and the article is correctly present in ovsqlite).
The 28 other write errors I had no longer show up.
As for the crosspost example to an unknown group:
Before:
350|5350|5827|384|0|0|fr.rec.son-image.video.materiel|y
select * from artinfo where artnum=4634 and groupid=350;
=> no article
Now::
350|4634|5827|388|0|0|fr.rec.son-image.video.materiel|y
select * from artinfo where artnum=4634 and groupid=350;
=> found
Thanks Bo for your fast patch and having made it work perfectly.
I'll report tomorrow the result of the expiry process.
Regarding your questions, I'm only aware of this one you probably
already know, and too high-level for what you are looking:
https://www.eyrie.org/~eagle/software/inn/docs/libstorage.html
--
Julien ?LIE
??Il ne faut jamais pleurer d'avoir perdu la lune car les larmes
emp?chent de voir les ?toiles.??
------------------------------
Subject: Digest Footer
_______________________________________________
inn-workers mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/inn-workers
------------------------------
End of inn-workers Digest, Vol 126, Issue 15
********************************************
