Re: Storage Array Problems
Bob Proulx writes: > Since they were logged as being discarded they are never going to be > delivered. I would go slow and send one or two initially and note the > message-ids of those messages. If they do not show up through the > list in a reasonable time please send a note to the debbugs team > mailing list with the message-ids so we can look for them. Luckily Prot's resent messages cited my discarded messages, so in the end I didn't think it necessary to resend mine :). Thanks for all the information and your help, -- Basil
Re: Storage Array Problems
Protesilaos Stavrou wrote: > "Basil L. Contovounesios" wrote: > > Bob Proulx writes: > >> Are all of those messages yours? They all have the same unique string > >> pattern. > > > > This pattern is generated by an Emacs MUA. The @tcd.ie ones are mine, > > and the @protesilaos.com ones are Prot's (CCed). I think I received the > > messages locally, but they're clearly missing from > > https://bugs.gnu.org/45068 and possibly other places too. Should I just > > resend the missing messages? > > Hello! I noticed that they were missing, but assumed that the sync > takes some time. > > Please re-send them or tell me how I can do it from here. When I was provided with a message-id by Lars for one of his missing messages I was able to grep around and find that message and the others in the logs. The logs said those message-ids had been discarded. That's all I know. Sorry. The group of those all together just stood out as looking unusual to my eye and therefore I mentioned it. I don't know if there is a systematic failure that needs to be fixed or if it was simply human error due to the systems problems and the large spam backlog. One of the contributing factors may have been related to the storage array problems yesterday. When a system can't read or write files the process trying to do so gets "blocked waiting for I/O" and pauses in an uninterruptable wait state. (In the Linux kernel a ps listing shows this uninterruptible state as the "D" state.) Since most OS functions get cached in the file system buffer cache in RAM the OS on most systems were still able to function at some level of functionality. As far as I know none of the systems outright crashed. But these processes blocked waiting for I/O from the networked storage server did pile up. I saw that fencepost had a system load of more than 1100! The FSF admins worked almost all day long Sunday morning through late afternoon to restore the storage array. As you can imagine it was a high stress situation for them. Meanwhile after the initial couple of hours the rest of the systems were mostly restored to normal operation and they were able to drain down their high cpu load averages. Those uninterruptible processes completed their I/O reads and writes upon which they were blocked and were able to exit. However after being blocked for a long time some processes that have timeouts will time out and be killed for taking too long to complete. The large mail backlog that occurred yesterday which meant that humans looking at the mailman web page hold queue were looking at dozens and dozens of messages most of which were spam because the anti-spam "cancel bot" was also backlogged. That's almost worst case for a human looking at mail messages and trying to pick out the non-spam messages from the sea of spam. But I really have no idea about any particular message and am just guessing. I also don't know the deep details of the storage array problems either. Perhaps the FSF admins will write up a blog note about it. That would be interesting to me. From what I could tell there was a coupled failure of multiple controller nodes causing the array to lose redundancy. At least one of the arrays went offline completely. They had to carefully reset and restore redundancy quorom of the disk storage and the controller nodes. Other than the initial hour when things were completely offline the subsequent restoration was all done online and running while the system was functioning in a degraded raid mode. Which is pretty cool when you think of it! > [ I am using Emacs+Gnus and this setup has been stable for a fairly long time > ] Emacs+Gnus worked great. No problems there at all. The only reason that Emacs+Gnus got mentioned was that it created a message-id format that I did not recognize and therefore asked if those were all from Lars. Basil told me those were from Emacs. Which is great. No problems there at all. Bob
RE: Group keyrings
Hello, Can someone remove me from this distribution list. Thank you. -Original Message- From: Savannah-users On Behalf Of Ineiev Sent: March 1, 2021 2:23 PM To: savannah-users@gnu.org Subject: Re: Group keyrings WARNING: External Email - This email originated outside of Jefferson. DO NOT CLICK links or attachments unless you recognize the sender and are expecting the email. The information contained in this transmission contains privileged and confidential information. It is intended only for the use of the person named above. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. CAUTION: Intended recipients should NOT use email communication for emergent or urgent health care matters.
Re: Storage Array Problems
Basil L. Contovounesios wrote: > Bob Proulx writes: > > Are all of those messages yours? They all have the same unique string > > pattern. > > This pattern is generated by an Emacs MUA. Oh! Thank you for that tidbit of information. I am unfamiliar with that signature and thought it might have been applied custom. (One might notice that message-ids on my messages are custom for example.) > The @tcd.ie ones are mine, and the @protesilaos.com ones are Prot's > (CCed). I think I received the messages locally, but they're > clearly missing from https://bugs.gnu.org/45068 and possibly other > places too. Should I just resend the missing messages? Since they were logged as being discarded they are never going to be delivered. I would go slow and send one or two initially and note the message-ids of those messages. If they do not show up through the list in a reasonable time please send a note to the debbugs team mailing list with the message-ids so we can look for them. help-debb...@gnu.org help-debbugs AT gnu DOT org (The obfuscators often get in the way of actually sending email addresses to people who read the email on web pages.) The main debbugs team members do not monitor the Savannah mailing lists. As a standard operating procedure we normally hold all unknown senders (unknown in this case is unknown to the Mailman mailing list management software, you might be Margaret Hamilton in real life but still an unknown sender to Mailman here), we normally hold all unknown senders for human review upon the initial contact. After review the message is approved and the sender is then added to the known list of senders for that mailing list. This is so it is only the initial review is needed for specific human checking. Subsequent mail is completely automated after that point with no human delays added. I found it an unusual pattern to see that large group of message-ids that all had the same syntax form all were discarded that morning over that time of 8am to 11am. Which is why I asked about them. Something might have gone wrong somewhere, potentially between a chair and keyboard for that matter. Bob signature.asc Description: PGP signature
Re: Storage Array Problems
Bob Proulx writes: > I would send just one or two initially. Please keep note of the > message-ids of them. See if they show up. If not please notify us of > the message-ids so that we can look to see their disposition. I've already resent them all and they all showed up fine. > For debbugs please send to the debbugs team mailing list. > > help-debb...@gnu.org > help-debbugs AT gnu DOT org > > The main debbugs folks don't monitor the Savannah mailing lists. Right; thanks.
Re: Storage Array Problems
Lars Ingebrigtsen wrote: > Thanks for checking. I'll resend the discarded messages, then. I would send just one or two initially. Please keep note of the message-ids of them. See if they show up. If not please notify us of the message-ids so that we can look to see their disposition. For debbugs please send to the debbugs team mailing list. help-debb...@gnu.org help-debbugs AT gnu DOT org The main debbugs folks don't monitor the Savannah mailing lists. One of the contributing factors may have been the large mail backlog that occurred yesterday which meant that humans looking at the mailman web page hold queue were looking at dozens and dozens of messages most of which were spam because the anti-spam "cancel bot" was also backlogged. That's almost worst case for a human looking at things and trying to pick out the non-spam messages. But I really have no idea about this particular message and am just guessing. Bob
Re: Group keyrings
Hello; On Tue, Feb 09, 2021 at 04:03:25PM +, Ineiev wrote: ... > Probably, it would be better if each group had a public area > where its admins (rather than every member) could post only keys > used for releases, like GnuPG does [1]. I've pushed a patch for it > to the group-keyring branch [2]. I've just installed the changes on Savannah, including updated documentation, https://savannah.gnu.org/maintenance/UsingGpg/ https://savannah.gnu.org/maintenance/DownloadArea/ Please check if anything needs fixing; after that, we probably should make an announcement in the Savannah News area. Thank you! signature.asc Description: PGP signature
Re: Storage Array Problems
Bob Proulx writes: > Lars Ingebrigtsen wrote: >> <8735xgnw6y@gnus.org> >> >> And here's the logs from my MTA for this message: >> >> 2021-02-28 14:44:41 1lGMNC-0008OQ-HF <= la...@gnus.org >> H=cm-84.212.220.105.getinternet.no (xo) [84.212.220.105] P=esmtpsa >> X=TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no A=plain_server:larsi S=3501 >> id=8735xgnw6y@gnus.org >> 2021-02-28 14:45:16 1lGMNC-0008OQ-HF => 46...@debbugs.gnu.org R=dnslookup >> T=remote_smtp H=debbugs.gnu.org [209.51.188.43] C="250 OK >> id=1lGMNS-00025F-TJ" >> (Times are in +0100 (CET).) >> >> This message has not shown up on the debbugs bug tracker here: >> >> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=46781 > > I find that message has been deleted at the Mailman part of the > pipeline. Time in US/Eastern -0500. I do not find it in the > listhelper anti-spam automation. > > Feb 28 08:45:45 2021 (9480) Message discarded, msgid: > <8735xgnw6y@gnus.org> > > I see that the message id pattern is unusual. And there are many in > that group. > > Feb 28 08:44:11 2021 (9480) Message discarded, msgid: > <877dmsnwa7@gnus.org> > Feb 28 08:45:45 2021 (9480) Message discarded, msgid: > <8735xgnw6y@gnus.org> > Feb 28 08:49:27 2021 (9480) Message discarded, msgid: > <87lfb8l2wr@tcd.ie> > Feb 28 08:50:36 2021 (9480) Message discarded, msgid: > <87y2f8mheo@gnus.org> > Feb 28 08:51:12 2021 (9480) Message discarded, msgid: > <87wnusmhe1@gnus.org> > Feb 28 08:54:46 2021 (9480) Message discarded, msgid: > <87sg5gmh7j@gnus.org> > Feb 28 08:55:07 2021 (9480) Message discarded, msgid: > <87r1l0mh74@gnus.org> > Feb 28 08:57:54 2021 (9480) Message discarded, msgid: > <87eeh0xpm6@protesilaos.com> > Feb 28 09:12:38 2021 (9480) Message discarded, msgid: > <87lfb8mgdu@gnus.org> > Feb 28 09:14:34 2021 (9480) Message discarded, msgid: > <87a6roxou1@protesilaos.com> > Feb 28 09:14:55 2021 (9480) Message discarded, msgid: > <87h7lwmgaj@gnus.org> > Feb 28 09:15:32 2021 (9480) Message discarded, msgid: > <87mtvomgea@gnus.org> > Feb 28 09:24:10 2021 (9480) Message discarded, msgid: > <87czwkmfuk@gnus.org> > Feb 28 09:24:31 2021 (9480) Message discarded, msgid: > <87blc4mfub@gnus.org> > Feb 28 10:20:43 2021 (9480) Message discarded, msgid: > <87im6cfcex@tcd.ie> > Feb 28 10:58:31 2021 (9480) Message discarded, msgid: > <8735xgxk0z@protesilaos.com> > > Are all of those messages yours? They all have the same unique string > pattern. This pattern is generated by an Emacs MUA. The @tcd.ie ones are mine, and the @protesilaos.com ones are Prot's (CCed). I think I received the messages locally, but they're clearly missing from https://bugs.gnu.org/45068 and possibly other places too. Should I just resend the missing messages? Thanks, -- Basil
Re: Storage Array Problems
On 2021-03-01, 08:03 +, "Basil L. Contovounesios" wrote: > Bob Proulx writes: > >> Lars Ingebrigtsen wrote: >>> <8735xgnw6y@gnus.org> >>> >>> And here's the logs from my MTA for this message: >>> >>> 2021-02-28 14:44:41 1lGMNC-0008OQ-HF <= la...@gnus.org >>> H=cm-84.212.220.105.getinternet.no (xo) [84.212.220.105] P=esmtpsa >>> X=TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no A=plain_server:larsi S=3501 >>> id=8735xgnw6y@gnus.org >>> 2021-02-28 14:45:16 1lGMNC-0008OQ-HF => 46...@debbugs.gnu.org R=dnslookup >>> T=remote_smtp H=debbugs.gnu.org [209.51.188.43] C="250 OK >>> id=1lGMNS-00025F-TJ" >>> (Times are in +0100 (CET).) >>> >>> This message has not shown up on the debbugs bug tracker here: >>> >>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=46781 >> >> I find that message has been deleted at the Mailman part of the >> pipeline. Time in US/Eastern -0500. I do not find it in the >> listhelper anti-spam automation. >> >> Feb 28 08:45:45 2021 (9480) Message discarded, msgid: >> <8735xgnw6y@gnus.org> >> >> I see that the message id pattern is unusual. And there are many in >> that group. >> >> Feb 28 08:44:11 2021 (9480) Message discarded, msgid: >> <877dmsnwa7@gnus.org> >> Feb 28 08:45:45 2021 (9480) Message discarded, msgid: >> <8735xgnw6y@gnus.org> >> Feb 28 08:49:27 2021 (9480) Message discarded, msgid: >> <87lfb8l2wr@tcd.ie> >> Feb 28 08:50:36 2021 (9480) Message discarded, msgid: >> <87y2f8mheo@gnus.org> >> Feb 28 08:51:12 2021 (9480) Message discarded, msgid: >> <87wnusmhe1@gnus.org> >> Feb 28 08:54:46 2021 (9480) Message discarded, msgid: >> <87sg5gmh7j@gnus.org> >> Feb 28 08:55:07 2021 (9480) Message discarded, msgid: >> <87r1l0mh74@gnus.org> >> Feb 28 08:57:54 2021 (9480) Message discarded, msgid: >> <87eeh0xpm6@protesilaos.com> >> Feb 28 09:12:38 2021 (9480) Message discarded, msgid: >> <87lfb8mgdu@gnus.org> >> Feb 28 09:14:34 2021 (9480) Message discarded, msgid: >> <87a6roxou1@protesilaos.com> >> Feb 28 09:14:55 2021 (9480) Message discarded, msgid: >> <87h7lwmgaj@gnus.org> >> Feb 28 09:15:32 2021 (9480) Message discarded, msgid: >> <87mtvomgea@gnus.org> >> Feb 28 09:24:10 2021 (9480) Message discarded, msgid: >> <87czwkmfuk@gnus.org> >> Feb 28 09:24:31 2021 (9480) Message discarded, msgid: >> <87blc4mfub@gnus.org> >> Feb 28 10:20:43 2021 (9480) Message discarded, msgid: >> <87im6cfcex@tcd.ie> >> Feb 28 10:58:31 2021 (9480) Message discarded, msgid: >> <8735xgxk0z@protesilaos.com> >> >> Are all of those messages yours? They all have the same unique string >> pattern. > > This pattern is generated by an Emacs MUA. The @tcd.ie ones are mine, > and the @protesilaos.com ones are Prot's (CCed). I think I received the > messages locally, but they're clearly missing from > https://bugs.gnu.org/45068 and possibly other places too. Should I just > resend the missing messages? > > Thanks, Hello! I noticed that they were missing, but assumed that the sync takes some time. Please re-send them or tell me how I can do it from here. [ I am using Emacs+Gnus and this setup has been stable for a fairly long time ] Thank you! Prot -- Protesilaos Stavrou protesilaos.com
Re: Storage Array Problems
Bob Proulx writes: > I find that message has been deleted at the Mailman part of the > pipeline. Time in US/Eastern -0500. I do not find it in the > listhelper anti-spam automation. > > Feb 28 08:45:45 2021 (9480) Message discarded, msgid: > <8735xgnw6y@gnus.org> Thanks for checking. I'll resend the discarded messages, then. [...] > Are all of those messages yours? They all have the same unique string > pattern. No, only the @gnus.org messages are mine.
