Send inn-workers mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.isc.org/mailman/listinfo/inn-workers
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of inn-workers digest..."
Today's Topics:
1. Re: [Resolved?] Re: ovsqlite - bug? (Russ Allbery)
2. Re: [Resolved?] Re: ovsqlite - bug? (Kamil Jo?ca)
3. makehistory can generate duplicate lines was:[Re: ovsqlite -
bug?] (Kamil Jo?ca)
----------------------------------------------------------------------
Message: 1
Date: Sun, 20 Nov 2022 17:52:32 -0800
From: Russ Allbery <[email protected]>
To: [email protected]
Subject: Re: [Resolved?] Re: ovsqlite - bug?
Message-ID: <[email protected]>
Content-Type: text/plain; charset=utf-8
Julien ?LIE <[email protected]> writes:
> I don't think you should bother.
> You won't be able to make your history file consistent with your
> self-expiring CNFS buffers. I bet the articles have already expired
> from your current overview. They are just still here in the buffer, just
> waiting for being overwritten by the next cycle.
> What happens is that makehistory parses your spool and finds these old
> articles. It generates overview entries for them.
> Tomorrow, after the run of expireover (by news.daily), your overview date
> will be purged of these entries.
There's some sort of deeper bug here, though, since we shouldn't have two
copies of the same group and article number pair. I'm not entirely sure
*where* the duplicate is... the history file, perhaps? It's the same
storage token for both copies of the article, so it doesn't seem to be
duplicated in CNFS. But I'm also not sure how duplicates managed to be
written to the history file.
--
Russ Allbery ([email protected]) <https://www.eyrie.org/~eagle/>
Please send questions to the list rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.
------------------------------
Message: 2
Date: Mon, 21 Nov 2022 05:17:05 +0100
From: Kamil Jo?ca <[email protected]>
To: [email protected]
Subject: Re: [Resolved?] Re: ovsqlite - bug?
Message-ID: <[email protected]>
Content-Type: text/plain; charset=iso-8859-2
Julien ?LIE <[email protected]> writes:
[..]
>
> I don't think you should bother.
> You won't be able to make your history file consistent with your
> self-expiring CNFS buffers. I bet the articles have already expired
> from your current overview. They are just still here in the buffer,
> just waiting for being overwritten by the next cycle.
>
> What happens is that makehistory parses your spool and finds these old
> articles. It generates overview entries for them.
> Tomorrow, after the run of expireover (by news.daily), your overview
> date will be purged of these entries.
Maybe I was not accurate enough previously:
1. these articles exists in spool, (sm could extract them) but have no
history entries for some reason (crash?). And they rather far from expirinig.
2. please note, that generated overview lines (in my previous post) are
exact duplicates (even storage token is the same). For now I would
suspect that makehistory generates record twice for record not in
history, but it is only my guess.
Neverthless ovsqlite seems to did good work here.
KJ
--
http://wolnelektury.pl/wesprzyj/teraz/
------------------------------
Message: 3
Date: Mon, 21 Nov 2022 08:51:07 +0100
From: Kamil Jo?ca <[email protected]>
To: [email protected]
Subject: makehistory can generate duplicate lines was:[Re: ovsqlite -
bug?]
Message-ID: <[email protected]>
Content-Type: text/plain; charset=iso-8859-2
Kamil Jo?ca <[email protected]> writes:
[...]
>
> So the first thigs I should do is to make history consistent with spool
> and then test further.
> But for now I am pretty sure, that sqlite worked properly as, it forces
> uniqueness of (group,artnum ) in overview (and other methods not) -
> and this causes error messages.
> Sorry for bothering you.
>
I tried to rebuild history:
--8<---------------cut here---------------start------------->8---
kjonca@alfa:/var/lib/news%sudo -u news env PATH=$PATH:/usr/lib/news/bin
makehistory -f history.n
makehistory: dbzstore duplicate message-id
history.n:[2FFF2E79E03CA12C3E3C4F7E398EBAF7]@60072283 Success
makehistory: dbzstore duplicate message-id
history.n:[C1067871BE1F25B369114E7C94635D22]@121782076 Success
makehistory: dbzstore duplicate message-id
history.n:[6282400F8A293D047E9052468792AF18]@176267656 Success
makehistory: dbzstore duplicate message-id
history.n:[65B607B90549808E3D4A4E3E33DD6D4A]@226791333 Success
makehistory: dbzstore duplicate message-id
history.n:[E55B9C6F2E865C431F4466F1085D7596]@352606103 Success
makehistory: dbzstore duplicate message-id
history.n:[28464C771BBE6A7D471737C28E6C7F6A]@411231076 Success
makehistory: dbzstore duplicate message-id
history.n:[1F375DE27593F75A6022530840232809]@470885597 Success
--8<---------------cut here---------------end--------------->8---
You can see that there is 7 messages about duplicates ... (anyone surprised?)
I took a look in new history file and end with command:
--8<---------------cut here---------------start------------->8---
kjonca@alfa:/var/lib/news%grep -n -A2 -B2
'2FFF2E79E03CA12C3E3C4F7E398EBAF7\|C1067871BE1F25B369114E7C94635D22\|6282400F8A293D047E9052468792AF18\|65B607B90549808E3D4A4E3E33DD6D4A\|E55B9C6F2E865C431F4466F1085D7596\|28464C771BBE6A7D471737C28E6C7F6A\|1F375DE27593F75A6022530840232809'
history.n
1:[2FFF2E79E03CA12C3E3C4F7E398EBAF7] 1643424642~-~1538492029
@03FF30000000000000000000004400000008@
2-[50AFBD2AD3C16FA233D6E88D7D5B600C] 1643424642~-~1538492036
@03FF30000000000000000000005E00000007@
3-[4CA26D1C22BC1FCB52DCAA5B56F36666] 1643424642~-~1538492212
@03FF30000000000000000000005F00000007@
--
612979-[11F1E04DF20F6533C381DB6C649B374D] 1643427583~-~1566810064
@03FF3000000000000000001FFFE500000007@
612980-[D121941BCEA5E037523AE35801A53596] 1643427583~-~1566810077
@03FF3000000000000000001FFFE800000007@
612981:[2FFF2E79E03CA12C3E3C4F7E398EBAF7] 1643424642~-~1538492029
@03FF30000000000000000000004400000008@
612982:[C1067871BE1F25B369114E7C94635D22] 1643427583~-~1566810112
@03FF31000000000000000000004400000008@
612983-[82BFD6304D1012333F22C559B8AF46A3] 1643427583~-~1566810146
@03FF31000000000000000000009100000007@
612984-[B2EAA6504DB45DC7F33C214880D99A50] 1643427583~-~1566810185
@03FF31000000000000000000009200000007@
--
1242667-[4F893D038B278138E8AFB8E1C899BFFA] 1643431380~-~1595582940
@03FF3100000000000000001FFFF400000007@
1242668-[B22F99E507F22F30250D4A2D28742A19] 1643431380~-~1595582974
@03FF3100000000000000001FFFFD00000007@
1242669:[C1067871BE1F25B369114E7C94635D22] 1643427583~-~1566810112
@03FF31000000000000000000004400000008@
1242670:[6282400F8A293D047E9052468792AF18] 1643431380~-~1595583001
@03FF32000000000000000000004400000008@
1242671-[A0615E54374BB1263836C289CAB75EF9] 1643431380~-~1595583061
@03FF32000000000000000000004600000007@
1242672-[C07606242B08C0D7FC987945FE0CECA1] 1643431380~-~1595583073
@03FF32000000000000000000004700000007@
--
1798639-[BE079E8E4CA8D10A0ED4C8761C04EE9B] 1643433853~-~1623959587
@03FF3200000000000000001FFFF900000007@
1798640-[C52CD86396DB88B01887D56389ECC6E7] 1643433853~-~1623959629
@03FF3200000000000000001FFFFC00000007@
1798641:[6282400F8A293D047E9052468792AF18] 1643431380~-~1595583001
@03FF32000000000000000000004400000008@
1798642:[65B607B90549808E3D4A4E3E33DD6D4A] 1643433853~-~1623959630
@03FF33000000000000000000004400000008@
1798643-[7D4F42A4C19BC04239EBB1C1538E64AE] 1643433853~-~1623959631
@03FF33000000000000000000004800000007@
1798644-[F49DEA0F8095B1B96440EF800427A797] 1643433853~-~1623959632
@03FF33000000000000000000004B00000007@
--
2314183-[BC90903F831BFB1DE8CA39C98F8ED2C9] 1653058806~-~1653058751
@03FF3300000000000000001FFFF600000007@
2314184-[A505379E708A59F16DAA62541CCFDB67] 1653058866~-~1653058823
@03FF3300000000000000001FFFFE00000007@
2314185:[65B607B90549808E3D4A4E3E33DD6D4A] 1643433853~-~1623959630
@03FF33000000000000000000004400000008@
2314186-[1AA051033A3D96E8E79F6F42C2159419] 1643409332~-~1454972341
@03FF3400000000000000000EE11400000006@
2314187-[8FCA5D33F8688E4E75445850D7CCE398] 1643409332~-~1454972378
@03FF3400000000000000000EE11600000006@
--
2921682-[8735D546AC02468CEC974116E2291BBA] 1669003873~-~1669003801
@03FF3400000000000000000EE10E00000007@
2921683-[4A8E1E3096F262C460C7963971964912] 1669004073~-~1669003319
@03FF3400000000000000000EE11000000007@
2921684:[E55B9C6F2E865C431F4466F1085D7596] 1643412102~-~1465897595
@03FF35000000000000000000004400000007@
2921685-[00F1642A04FEAA3F23269EAF258BDC4A] 1643412102~-~1465897596
@03FF35000000000000000000004800000006@
2921686-[9025840433BA049A341F2D7BF71F44AE] 1643412102~-~1465897596
@03FF35000000000000000000004A00000006@
--
3597988-[D8DB3B14DB0DD47F9344179C17974EF7] 1643417459~-~1487957728
@03FF3500000000000000001FFFCF00000006@
3597989-[EB84472A4704BB56196FAE8D119B6102] 1643417459~-~1487957745
@03FF3500000000000000001FFFD100000006@
3597990:[E55B9C6F2E865C431F4466F1085D7596] 1643412102~-~1465897595
@03FF35000000000000000000004400000007@
3597991:[28464C771BBE6A7D471737C28E6C7F6A] 1643417459~-~1487957763
@03FF36000000000000000000004400000007@
3597992-[25B7BD351BFC0DCA37C9E8D143C74313] 1643417459~-~1487957771
@03FF36000000000000000000008800000006@
3597993-[A4C478830BCD26EDF9206CF4DC18CE29] 1643417459~-~1487957812
@03FF36000000000000000000008B00000006@
--
4196192-[E6B5FC05ED4C9BDB2DC0452382676E79] 1643421414~-~1512585723
@03FF3600000000000000001FFFD300000006@
4196193-[C2574DC662BA26F96D83668966312137] 1643421414~-~1512585730
@03FF3600000000000000001FFFD500000006@
4196194:[28464C771BBE6A7D471737C28E6C7F6A] 1643417459~-~1487957763
@03FF36000000000000000000004400000007@
4196195:[1F375DE27593F75A6022530840232809] 1643421414~-~1512585730
@03FF37000000000000000000004400000007@
4196196-[37B8A01C3D7A5849FC290DA5D3CF91BB] 1643421414~-~1512585769
@03FF37000000000000000000007900000006@
4196197-[578459C8F2A952BC0782C85CCC270CD5] 1643421414~-~1512585775
@03FF37000000000000000000007B00000006@
--
4804906-[9A0C78E7194A7BBA54EB5CB20962BCAA] 1643424642~-~1538491933
@03FF3700000000000000001FFFE700000006@
4804907-[55DB71AEF999C1BCD62FAE9BECC0ED6C] 1643424642~-~1538492145
@03FF3700000000000000001FFFE800000006@
4804908:[1F375DE27593F75A6022530840232809] 1643421414~-~1512585730
@03FF37000000000000000000004400000007@
4804909-[D712318D392955A986E84DCC54C5CC1D]
1651442596~1657056641~1651440641 @040000626F03000200000000000000000000@
4804910-[99690AB74014DD13A932527753D33814] 1646094796~-~1646093042
@040000621D69000200000000000000000000@
--8<---------------cut here---------------end--------------->8---
This is clear (at least for me) that for each cycbuff file (except this
named "4") makehistory, after scanning whole buffer, it additionally try
to write entry about "first" message in buffer (compare lines 1-612981,
612982-1242669 and so on)
Here is additional information - my cycbuff config:
--8<---------------cut here---------------start------------->8---
cycbuff:0:/var/spool/news/cycbuffs/0:8388608
cycbuff:1:/var/spool/news/cycbuffs/1:8388608
cycbuff:2:/var/spool/news/cycbuffs/2:8388608
cycbuff:3:/var/spool/news/cycbuffs/3:8388608
cycbuff:4:/var/spool/news/cycbuffs/4:8388608
cycbuff:5:/var/spool/news/cycbuffs/5:8388608
cycbuff:6:/var/spool/news/cycbuffs/6:8388608
cycbuff:7:/var/spool/news/cycbuffs/7:8388608
metacycbuff:0:0,1,2,3,4,5,6,7:SEQUENTIAL
--8<---------------cut here---------------end--------------->8---
note the "SEQUENTIAL" line - this means that all buffers except one are
"fully rotated" (if you know what I mean) i.e.:
--8<---------------cut here---------------start------------->8---
sudo -u news /usr/lib/news/bin/cnfsstat
Class 0 for groups matching "*"
Buffer 0, size: 8.00 GBytes, position: 272 kBytes 7.00 cycles
Newest: 2022-01-29 4:39:43, 296 days, 4:06:37 ago
Buffer 1, size: 8.00 GBytes, position: 272 kBytes 7.00 cycles
Newest: 2022-01-29 5:43:00, 296 days, 3:03:20 ago
Buffer 2, size: 8.00 GBytes, position: 272 kBytes 7.00 cycles
Newest: 2022-01-29 6:24:13, 296 days, 2:22:07 ago
Buffer 3, size: 8.00 GBytes, position: 272 kBytes 7.00 cycles
Newest: 2022-05-20 17:01:09, 184 days, 16:45:11 ago
Buffer 4, size: 8.00 GBytes, position: 3.72 GBytes 6.47 cycles
Newest: 2022-11-21 8:45:05, 0 days, 0:01:15 ago
Buffer 5, size: 8.00 GBytes, position: 272 kBytes 6.00 cycles
Newest: 2022-01-29 1:50:59, 296 days, 6:55:21 ago
Buffer 6, size: 8.00 GBytes, position: 272 kBytes 6.00 cycles
Newest: 2022-01-29 2:56:54, 296 days, 5:49:26 ago
Buffer 7, size: 8.00 GBytes, position: 272 kBytes 6.00 cycles
Newest: 2022-01-29 3:50:42, 296 days, 4:55:38 ago
--8<---------------cut here---------------end--------------->8---
(please ignore the dates, this spool was recreated from archive/backup)
So I think problem is with "makehistory" when comes to "fully rotated"
cycbuff.
KJ
--
http://wolnelektury.pl/wesprzyj/teraz/
------------------------------
Subject: Digest Footer
_______________________________________________
inn-workers mailing list
[email protected]
https://lists.isc.org/mailman/listinfo/inn-workers
------------------------------
End of inn-workers Digest, Vol 145, Issue 5
*******************************************
