Send inn-workers mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.isc.org/mailman/listinfo/inn-workers
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of inn-workers digest..."
Today's Topics:
1. Re: ovsqlite (Bo Lindbergh)
2. Re: Overview methods and rebuild documentation (Bo Lindbergh)
3. Re: ovsqlite (Julien ?LIE)
4. Re: ovsqlite (zlib edge-cases) (Bo Lindbergh)
5. Re: ovsqlite (Bo Lindbergh)
6. Re: ovsqlite (zlib edge-cases) (Julien ?LIE)
7. Re: ovsqlite (Julien ?LIE)
----------------------------------------------------------------------
Message: 1
Date: Wed, 23 Dec 2020 22:54:35 +0100
From: Bo Lindbergh <[email protected]>
To: [email protected]
Subject: Re: ovsqlite
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Quoth Julien ?LIE <[email protected]>:
>
> "Robust and faster at reading ranges of overview data, but somewhat slower at
> writing" would be a better description?
Good enough.
> I kept the default of 10,000 transactions. Anyway, my news server receives
> far less articles than this, so the time limit of 10s always applies.
> I increased it during makehistory but I'm unsure it improves speed (commits
> are probably faster than makehistory collecting the next bunch of overview
> lines to add). Maybe that suggestion should not be kept after all?
In my makehistory tests using an external USB disk, throughput didn't
increase much for row limits above 10000.
Maybe leave the default at that and instead advise admins to adjust
the time limit according to how many articles they accept per second
during normal operation?
/Bo Lindbergh
------------------------------
Message: 2
Date: Wed, 23 Dec 2020 23:20:36 +0100
From: Bo Lindbergh <[email protected]>
To: Julien ?LIE <[email protected]>
Cc: [email protected]
Subject: Re: Overview methods and rebuild documentation
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Quoth Julien ?LIE <[email protected]>:
>
>> Maybe mention
>> ctlinnd renumber ""
>> for when you want to be extra sure that the active file and the overview are
>> in sync?
>
> As rc.news takes care of renumbering on the first run after a rebuild, would
> it be in case someone starts INN without rc.news or a bug in makehistory?
> I bet it is the reason of the "extra sure" :-)
OK, that sounds sufficiently paranoid already. Suggestion withdrawn. :-)
/Bo Lindbergh
------------------------------
Message: 3
Date: Wed, 23 Dec 2020 23:28:28 +0100
From: Julien ?LIE <[email protected]>
To: [email protected]
Subject: Re: ovsqlite
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8; format=flowed
Hi Bo,
>> "Robust and faster at reading ranges of overview data, but
>> somewhat slower at writing" would be a better description?
>
> Good enough.
Adopted!
>> I kept the default of 10,000 transactions. Anyway, my news server
>> receives far less articles than this, so the time limit of 10s
>> always applies.
>> I increased it during makehistory but I'm unsure it improves speed
>> (commits are probably faster than makehistory collecting the next
>> bunch of overview lines to add). Maybe that suggestion should not
>> be kept after all?
>
> In my makehistory tests using an external USB disk, throughput didn't
> increase much for row limits above 10000.
>
> Maybe leave the default at that and instead advise admins to adjust
> the time limit according to how many articles they accept per second
> during normal operation?
Having a look at aioe's innreport:
https://news.aioe.org/stats/innreport-reports/
Incoming articles seem to be less than 1000/hour, 15000/day for a
full-text feed.
1000/hour is less than 3 articles per 10s.
Like you say, we could advise admins to look at their daily reports and
adjust the time limit accordingly. The question is how will they choose
the number of articles in a transaction (30s for about 9 articles, 60s
for 18...).
--
Julien ?LIE
??It's documented in The Book, somewhere??? (Larry Wall)
------------------------------
Message: 4
Date: Wed, 23 Dec 2020 23:55:31 +0100
From: Bo Lindbergh <[email protected]>
To: [email protected]
Subject: Re: ovsqlite (zlib edge-cases)
Message-ID: <[email protected]>
Content-Type: text/plain; charset="us-ascii"
Yet another patch!
* <inttypes.h> is no longer needed. The one spot where converting a 64-bit
integer to text is necessary now uses sqlite3_snprintf.
* Include directives cleaned up based on a better understanding of what
include/clibrary.h provides.
* Resources are freed before the server exits. This should make leak detection
tools happier.
* Comments added to the compression code, explaining why only one call
to inflate/deflate is needed for each overview record.
/Bo Lindbergh
-------------- next part --------------
A non-text attachment was scrubbed...
Name: delta4.diff.gz
Type: application/x-gzip
Size: 1813 bytes
Desc: not available
URL:
<https://lists.isc.org/pipermail/inn-workers/attachments/20201223/ac0e6320/attachment-0001.bin>
------------------------------
Message: 5
Date: Thu, 24 Dec 2020 00:29:14 +0100
From: Bo Lindbergh <[email protected]>
To: [email protected]
Subject: Re: ovsqlite
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Quoth Julien ?LIE <[email protected]>:
>
> Having a look at aioe's innreport:
> https://news.aioe.org/stats/innreport-reports/
>
> Incoming articles seem to be less than 1000/hour, 15000/day for a full-text
> feed.
> 1000/hour is less than 3 articles per 10s.
> Like you say, we could advise admins to look at their daily reports and
> adjust the time limit accordingly. The question is how will they choose the
> number of articles in a transaction (30s for about 9 articles, 60s for 18...).
Eternal September doesn't have much more traffic either.
https://www.eternal-september.org/stats/index.html
Are there any INN sites that do?
/Bo Lindbergh
(Since it's December 24 in this time zone: ABCDEFGHIJKMNOPQRSTUVWXYZ!)
------------------------------
Message: 6
Date: Thu, 24 Dec 2020 09:04:32 +0100
From: Julien ?LIE <[email protected]>
To: "[email protected]" <[email protected]>
Subject: Re: ovsqlite (zlib edge-cases)
Message-ID: <[email protected]>
Content-Type: text/plain; charset=windows-1252; format=flowed
Hi Bo,
> Yet another patch!
>
> ?* <inttypes.h> is no longer needed.? The one spot where converting a
> 64-bit
> ?? integer to text is necessary now uses sqlite3_snprintf.
> ?* Include directives cleaned up based on a better understanding of what
> ?? include/clibrary.h provides.
> ?* Resources are freed before the server exits.? This should make leak
> detection
> ?? tools happier.
> ?* Comments added to the compression code, explaining why only one call
> ?? to inflate/deflate is needed for each overview record.
Thanks, applied!
Useful comments for the compression code, very much appreciated.
And the related check in sqlite.m4 removed.
--
Julien ?LIE
??? Qu'est-ce qu'on fait Ast?rix?? On coule??
? Non, on ?coule?!?? (Ast?rix)
------------------------------
Message: 7
Date: Thu, 24 Dec 2020 09:17:07 +0100
From: Julien ?LIE <[email protected]>
To: [email protected]
Subject: Re: ovsqlite
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8; format=flowed
Hi Bo,
>> Having a look at aioe's innreport:
>> https://news.aioe.org/stats/innreport-reports/
>>
>> Incoming articles seem to be less than 1000/hour, 15000/day for a
>> full-text feed.
>> 1000/hour is less than 3 articles per 10s. Like you say, we could
>> advise admins to look at their daily reports and adjust the time
>> limit accordingly. The question is how will they choose the
>> number of articles in a transaction (30s for about 9 articles, 60s
>> for 18...).
>
> Eternal September doesn't have much more traffic either.
> https://www.eternal-september.org/stats/index.html
> Are there any INN sites that do?
If someone uses INN with a binaries feed, yes, but I doubt it is the case.
Specific servers like news.gmane.io may receive more messages too.
https://www.giganews.com/2000-days-retention/
(in 2014)
"There are roughly 45 billion articles stored on our servers and on
average, we receive between 450 and 500 articles per second. In fact, in
the past 2000 days, 1 exabyte of data has been transferred."
Gosh, 500 articles per second for binaries!
So, should the default values be changed to reflect normal usage of text
feeds? (30 seconds and 1,000 articles for a transaction?)
--
Julien ?LIE
??Ils ont refus? une offre de Normand??!??? (Ast?rix)
------------------------------
Subject: Digest Footer
_______________________________________________
inn-workers mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/inn-workers
------------------------------
End of inn-workers Digest, Vol 126, Issue 18
********************************************
