Re: VERY late notification emails
And its probably down again. Last notification I received was at 2022-04-20 16:37:08 UTC Vít Dne 20. 04. 22 v 17:04 Vít Ondruch napsal(a): Dne 19. 04. 22 v 20:35 Kevin Fenzi napsal(a): On Tue, Apr 19, 2022 at 07:48:56PM +0200, Vít Ondruch wrote: I think this is broken again. Last notification I have received was at 03:50:49 UTC. Yep. Restarted it. Thank you. It works as of today. In other news, we now have the upstream python3 version deployed in staging and are testing it/fixing issues. After f36 is out we will likely upgrade prod with this version and hopefully improve stability/maintainability. Great news! Looking forward to it. Vít kevin ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
Dne 19. 04. 22 v 20:35 Kevin Fenzi napsal(a): On Tue, Apr 19, 2022 at 07:48:56PM +0200, Vít Ondruch wrote: I think this is broken again. Last notification I have received was at 03:50:49 UTC. Yep. Restarted it. Thank you. It works as of today. In other news, we now have the upstream python3 version deployed in staging and are testing it/fixing issues. After f36 is out we will likely upgrade prod with this version and hopefully improve stability/maintainability. Great news! Looking forward to it. Vít kevin ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
On Tue, Apr 19, 2022 at 07:48:56PM +0200, Vít Ondruch wrote: > I think this is broken again. Last notification I have received was at > 03:50:49 UTC. Yep. Restarted it. In other news, we now have the upstream python3 version deployed in staging and are testing it/fixing issues. After f36 is out we will likely upgrade prod with this version and hopefully improve stability/maintainability. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
I think this is broken again. Last notification I have received was at 03:50:49 UTC. Vít Dne 22. 03. 22 v 12:56 Vít Ondruch napsal(a): Actually, just moments ago 11:44 UTC I have received more messages, this time stamped 2022-03-21 17:35:45 UTC. So while the service is not completely dead, the delay is getting bigger. Vít Dne 22. 03. 22 v 12:34 Vít Ondruch napsal(a): No notifications again. The last notification I have received was today on 6:38 UTC originally sent 2022-03-21 15:55:21 UTC. Vít Dne 19. 03. 22 v 22:50 Kevin Fenzi napsal(a): On Thu, Mar 17, 2022 at 10:29:37AM +0100, Vít Ondruch wrote: Dne 07. 03. 22 v 19:12 Kevin Fenzi napsal(a): I watched it over the weekend, and now it's only 3 days behind. ;( Hopefully it will finish catching up... I'm continuing to watch it. It eventually caught up and I have received the notifications on time up until 2022-03-16 15:49:09 UTC. After that moment, the next two notification were delayed ~10 minutes, and the following notification from 2022-03-16 16:31:13 UTC I have received on Thu, 17 Mar 2022 01:54:03 -0700 (PDT), i.e. 2022-03-17 08:54:03 UTC. So is 2022-03-16 15:49:09 UTC the approximate time when the service crashed and 2022-03-17 08:54:03 UTC the time of restart? I don't think I restarted it then... 09 UTC is around 2am my local time. :) I did restart it this morning, so it was not processing right for about 4 hours this morning. It's already caught back up I think. kevin ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
Actually, just moments ago 11:44 UTC I have received more messages, this time stamped 2022-03-21 17:35:45 UTC. So while the service is not completely dead, the delay is getting bigger. Vít Dne 22. 03. 22 v 12:34 Vít Ondruch napsal(a): No notifications again. The last notification I have received was today on 6:38 UTC originally sent 2022-03-21 15:55:21 UTC. Vít Dne 19. 03. 22 v 22:50 Kevin Fenzi napsal(a): On Thu, Mar 17, 2022 at 10:29:37AM +0100, Vít Ondruch wrote: Dne 07. 03. 22 v 19:12 Kevin Fenzi napsal(a): I watched it over the weekend, and now it's only 3 days behind. ;( Hopefully it will finish catching up... I'm continuing to watch it. It eventually caught up and I have received the notifications on time up until 2022-03-16 15:49:09 UTC. After that moment, the next two notification were delayed ~10 minutes, and the following notification from 2022-03-16 16:31:13 UTC I have received on Thu, 17 Mar 2022 01:54:03 -0700 (PDT), i.e. 2022-03-17 08:54:03 UTC. So is 2022-03-16 15:49:09 UTC the approximate time when the service crashed and 2022-03-17 08:54:03 UTC the time of restart? I don't think I restarted it then... 09 UTC is around 2am my local time. :) I did restart it this morning, so it was not processing right for about 4 hours this morning. It's already caught back up I think. kevin ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
On Tue, Mar 22, 2022 at 12:35 PM Vít Ondruch wrote: > > No notifications again. The last notification I have received was today > on 6:38 UTC originally sent 2022-03-21 15:55:21 UTC. At this point, I wonder if it wouldn't be more productive to just add a cron job to restart that service once a day, rather than constantly needing to do babysitting ... Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
No notifications again. The last notification I have received was today on 6:38 UTC originally sent 2022-03-21 15:55:21 UTC. Vít Dne 19. 03. 22 v 22:50 Kevin Fenzi napsal(a): On Thu, Mar 17, 2022 at 10:29:37AM +0100, Vít Ondruch wrote: Dne 07. 03. 22 v 19:12 Kevin Fenzi napsal(a): I watched it over the weekend, and now it's only 3 days behind. ;( Hopefully it will finish catching up... I'm continuing to watch it. It eventually caught up and I have received the notifications on time up until 2022-03-16 15:49:09 UTC. After that moment, the next two notification were delayed ~10 minutes, and the following notification from 2022-03-16 16:31:13 UTC I have received on Thu, 17 Mar 2022 01:54:03 -0700 (PDT), i.e. 2022-03-17 08:54:03 UTC. So is 2022-03-16 15:49:09 UTC the approximate time when the service crashed and 2022-03-17 08:54:03 UTC the time of restart? I don't think I restarted it then... 09 UTC is around 2am my local time. :) I did restart it this morning, so it was not processing right for about 4 hours this morning. It's already caught back up I think. kevin ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
On Thu, Mar 17, 2022 at 10:29:37AM +0100, Vít Ondruch wrote: > > Dne 07. 03. 22 v 19:12 Kevin Fenzi napsal(a): > > I watched it over the weekend, and now it's only 3 days behind. ;( > > Hopefully it will finish catching up... I'm continuing to watch it. > > > It eventually caught up and I have received the notifications on time up > until 2022-03-16 15:49:09 UTC. After that moment, the next two notification > were delayed ~10 minutes, and the following notification from 2022-03-16 > 16:31:13 UTC I have received on Thu, 17 Mar 2022 01:54:03 -0700 (PDT), i.e. > 2022-03-17 08:54:03 UTC. > > So is 2022-03-16 15:49:09 UTC the approximate time when the service crashed > and 2022-03-17 08:54:03 UTC the time of restart? I don't think I restarted it then... 09 UTC is around 2am my local time. :) I did restart it this morning, so it was not processing right for about 4 hours this morning. It's already caught back up I think. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
Dne 07. 03. 22 v 19:12 Kevin Fenzi napsal(a): I watched it over the weekend, and now it's only 3 days behind. ;( Hopefully it will finish catching up... I'm continuing to watch it. It eventually caught up and I have received the notifications on time up until 2022-03-16 15:49:09 UTC. After that moment, the next two notification were delayed ~10 minutes, and the following notification from 2022-03-16 16:31:13 UTC I have received on Thu, 17 Mar 2022 01:54:03 -0700 (PDT), i.e. 2022-03-17 08:54:03 UTC. So is 2022-03-16 15:49:09 UTC the approximate time when the service crashed and 2022-03-17 08:54:03 UTC the time of restart? Vít OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
On Sat, Mar 05, 2022 at 10:58:53AM +, Dan Čermák wrote: > > > On March 5, 2022 9:03:54 AM UTC, Gary Buhrmaster > wrote: > >On Mon, Feb 28, 2022 at 5:25 PM Kevin Fenzi wrote: > > > >> No, there's things we can do and are trying to do. ;) > > > >I seem to remember that one of the issues > >identified was (for those of us using gmail > >for the notifications) was that google could > >end up throttling emails. > > > >I have a vague recollection that google has > >numerous recommendations to mitigate > >against throttling (including DKIM and SPF), > >in addition to a method in which one can > >reduce certain delays for "buik" email > >and also monitor for issues ((I think via > >postmaster.google.com?) > > > >I am going to presume that these resources > >have been investigated and you have looked > >at the google monitoring information. Can > >you share why the emails are getting delayed > >according to Google? Yes, we do all those things. Google limits incoming emails to accounts. Here's the google "workspace" page on it (but I think there's also possibly lower limits for 'free' accounts: https://support.google.com/a/answer/1366776?hl=en So basically from time to time people (especially those on scm-commits mailing list) go over those limits and google just rejects email to them until it cools down. There's often also people who go over their google storage quota and it rejects emails for them until they delete things. > It's definitely not just Google, I keep getting delayed emails as well and my > emails are not hosted by a big provider. These are two seperate discussions. FMN is slow/behind, and google sometimes drops/delays/rejects FMN emails. I watched it over the weekend, and now it's only 3 days behind. ;( Hopefully it will finish catching up... I'm continuing to watch it. We are also close to having the python3 version up in staging. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
On March 5, 2022 9:03:54 AM UTC, Gary Buhrmaster wrote: >On Mon, Feb 28, 2022 at 5:25 PM Kevin Fenzi wrote: > >> No, there's things we can do and are trying to do. ;) > >I seem to remember that one of the issues >identified was (for those of us using gmail >for the notifications) was that google could >end up throttling emails. > >I have a vague recollection that google has >numerous recommendations to mitigate >against throttling (including DKIM and SPF), >in addition to a method in which one can >reduce certain delays for "buik" email >and also monitor for issues ((I think via >postmaster.google.com?) > >I am going to presume that these resources >have been investigated and you have looked >at the google monitoring information. Can >you share why the emails are getting delayed >according to Google? It's definitely not just Google, I keep getting delayed emails as well and my emails are not hosted by a big provider. Cheers, Dan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
On Mon, Feb 28, 2022 at 5:25 PM Kevin Fenzi wrote: > No, there's things we can do and are trying to do. ;) I seem to remember that one of the issues identified was (for those of us using gmail for the notifications) was that google could end up throttling emails. I have a vague recollection that google has numerous recommendations to mitigate against throttling (including DKIM and SPF), in addition to a method in which one can reduce certain delays for "buik" email and also monitor for issues ((I think via postmaster.google.com?) I am going to presume that these resources have been investigated and you have looked at the google monitoring information. Can you share why the emails are getting delayed according to Google? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
Am 28.02.22 um 14:45 schrieb Richard Shaw: I almost wrote this a week ago but decided not to as it's been recently discussed but this is really annoying. 6 days later is more than useless. FWIW: I just received a bunch (ca. 10) of notifications with 5 days of delay. Don't know if somebody just restarted something, but in its present shape, things still seem to be dysfunctional. Ralf ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
On Tue, Mar 01, 2022 at 10:42:38AM +0100, Vít Ondruch wrote: > > Dne 28. 02. 22 v 18:24 Kevin Fenzi napsal(a): > > On Mon, Feb 28, 2022 at 07:45:27AM -0600, Richard Shaw wrote: > > > I almost wrote this a week ago but decided not to as it's been recently > > > discussed but this is really annoying. 6 days later is more than useless. > > > > > > Previously it was blamed, at least partially, on the mass rebuild, but > > > clearly that should no longer be an issue by now? > > Well, I noted a number of reasons for it... one of them was that it > > sometimes crashes, but appears to be processing. This happened a few > > days ago and it was just restarted this morning. ;( > > > Is it down again? Or does it trying to catch up? If the status of the queue > was visible somewhere at least. I am asking, because the latest notification > I have ATM is from 22nd. Well, thats not really too helpfull a question. I mean, it might have been down when you asked, but now I am replying a day later and it's not, so I have no idea. I've got a session open to it now and am babysitting it. (restarting when I see failed services, restarting it if I don't see any messages for an hour or so, etc). > > > I know I can turn them off, but I actually LIKE the messages if they were > > > delivered promptly. > > > > > > Is there really nothing we can do about this? > > No, there's things we can do and are trying to do. ;) > > > > But of course more help welcome! > > > > We have a python3 port of it nearly ready to go, but I think it's > > CI/tests are not working, and we want to make sure those all work before > > we deploy it. I'll see if I can get more exact status... > > > But what happened that from reliable service is completely unreliable one? > Up until here: > > https://pagure.io/fedora-infrastructure/issue/10370 > > It was working mostly OK (although I have reported once incident earlier > https://pagure.io/fedora-infrastructure/issue/9932). But since that time it > is useless: > > https://pagure.io/fedora-infrastructure/issue/10414 I think it's a combination of things: * it's still querying the old account system, so all the new users who enable emails from it, get nothing and cause it to generate a bunch of emails that get rejected. * It's on fedmsg and most apps are using fedora-messaging. We have a gateway relaying from one to the other, but it doesn't handle new messages or changes in messages at all well. * We are generating a lot more messages than we used to. With more and more packages, mass rebuilds, eln builds, main branch to epel9 merges with hundreds of commits, etc, etc. The volume is just a lot higher. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
On Tue, Mar 01, 2022 at 08:04:14PM -, Peter Oliver wrote: > > I almost wrote this a week ago but decided not to as it's been recently > > discussed but this is really annoying. 6 days later is more than useless. > > ... > > Is there really nothing we can do about this? > > I think getting https://github.com/fedora-infra/fmn/pull/294 merged would > help a little, > because it moves the timestamp out of the message body and into the email > Date header. > The notifications will still be just as late, but at least you'll be able to > see that from your email index without opening each message. Yes, it could help some, but of course first we need to actually be running that version. Thats the python3 version, we are still running the old python2 version. ;( I'm working on getting the python3 version deployed to staging now, so hopefully soon we can get it in production. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
On 28. 02. 22 18:24, Kevin Fenzi wrote: On Mon, Feb 28, 2022 at 07:45:27AM -0600, Richard Shaw wrote: I almost wrote this a week ago but decided not to as it's been recently discussed but this is really annoying. 6 days later is more than useless. Previously it was blamed, at least partially, on the mass rebuild, but clearly that should no longer be an issue by now? Well, I noted a number of reasons for it... one of them was that it sometimes crashes, but appears to be processing. This happened a few days ago and it was just restarted this morning. ;( I know I can turn them off, but I actually LIKE the messages if they were delivered promptly. Is there really nothing we can do about this? No, there's things we can do and are trying to do. ;) But of course more help welcome! We have a python3 port of it nearly ready to go, but I think it's CI/tests are not working, and we want to make sure those all work before we deploy it. I'll see if I can get more exact status... CI is now working and the tests are passing. But I wasn't able to actually found out how the new version FMN should be released. Michal There's also been a lot of talk about re-writing it. Ryan just posted recently asking for folks use cases and pain points for that. I'll see about making it restart every day perhaps to make sure it's actually processing. kevin ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
> I almost wrote this a week ago but decided not to as it's been recently > discussed but this is really annoying. 6 days later is more than useless. > ... > Is there really nothing we can do about this? I think getting https://github.com/fedora-infra/fmn/pull/294 merged would help a little, because it moves the timestamp out of the message body and into the email Date header. The notifications will still be just as late, but at least you'll be able to see that from your email index without opening each message. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
Dne 28. 02. 22 v 18:24 Kevin Fenzi napsal(a): On Mon, Feb 28, 2022 at 07:45:27AM -0600, Richard Shaw wrote: I almost wrote this a week ago but decided not to as it's been recently discussed but this is really annoying. 6 days later is more than useless. Previously it was blamed, at least partially, on the mass rebuild, but clearly that should no longer be an issue by now? Well, I noted a number of reasons for it... one of them was that it sometimes crashes, but appears to be processing. This happened a few days ago and it was just restarted this morning. ;( Is it down again? Or does it trying to catch up? If the status of the queue was visible somewhere at least. I am asking, because the latest notification I have ATM is from 22nd. I know I can turn them off, but I actually LIKE the messages if they were delivered promptly. Is there really nothing we can do about this? No, there's things we can do and are trying to do. ;) But of course more help welcome! We have a python3 port of it nearly ready to go, but I think it's CI/tests are not working, and we want to make sure those all work before we deploy it. I'll see if I can get more exact status... But what happened that from reliable service is completely unreliable one? Up until here: https://pagure.io/fedora-infrastructure/issue/10370 It was working mostly OK (although I have reported once incident earlier https://pagure.io/fedora-infrastructure/issue/9932). But since that time it is useless: https://pagure.io/fedora-infrastructure/issue/10414 Vít There's also been a lot of talk about re-writing it. Ryan just posted recently asking for folks use cases and pain points for that. I'll see about making it restart every day perhaps to make sure it's actually processing. kevin ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure OpenPGP_signature Description: OpenPGP digital signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
On Mon, Feb 28, 2022 at 11:25 AM Kevin Fenzi wrote: > On Mon, Feb 28, 2022 at 07:45:27AM -0600, Richard Shaw wrote: > > I almost wrote this a week ago but decided not to as it's been recently > > discussed but this is really annoying. 6 days later is more than useless. > > > > Previously it was blamed, at least partially, on the mass rebuild, but > > clearly that should no longer be an issue by now? > > Well, I noted a number of reasons for it... one of them was that it > sometimes crashes, but appears to be processing. This happened a few > days ago and it was just restarted this morning. ;( > > Ahh. Ok. At least that explains it! > > > I know I can turn them off, but I actually LIKE the messages if they were > > delivered promptly. > > > > Is there really nothing we can do about this? > > No, there's things we can do and are trying to do. ;) > > But of course more help welcome! > > We have a python3 port of it nearly ready to go, but I think it's > CI/tests are not working, and we want to make sure those all work before > we deploy it. I'll see if I can get more exact status... > > There's also been a lot of talk about re-writing it. Ryan just posted > recently asking for folks use cases and pain points for that. > > I'll see about making it restart every day perhaps to make sure it's > actually processing. I've used this in a SystemD service file for a chromium based dashboard on a RPi: # Restart every 8 hours RuntimeMaxSec=28800 Thanks, Richard ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: VERY late notification emails
On Mon, Feb 28, 2022 at 07:45:27AM -0600, Richard Shaw wrote: > I almost wrote this a week ago but decided not to as it's been recently > discussed but this is really annoying. 6 days later is more than useless. > > Previously it was blamed, at least partially, on the mass rebuild, but > clearly that should no longer be an issue by now? Well, I noted a number of reasons for it... one of them was that it sometimes crashes, but appears to be processing. This happened a few days ago and it was just restarted this morning. ;( > > I know I can turn them off, but I actually LIKE the messages if they were > delivered promptly. > > Is there really nothing we can do about this? No, there's things we can do and are trying to do. ;) But of course more help welcome! We have a python3 port of it nearly ready to go, but I think it's CI/tests are not working, and we want to make sure those all work before we deploy it. I'll see if I can get more exact status... There's also been a lot of talk about re-writing it. Ryan just posted recently asking for folks use cases and pain points for that. I'll see about making it restart every day perhaps to make sure it's actually processing. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
VERY late notification emails
I almost wrote this a week ago but decided not to as it's been recently discussed but this is really annoying. 6 days later is more than useless. Previously it was blamed, at least partially, on the mass rebuild, but clearly that should no longer be an issue by now? I know I can turn them off, but I actually LIKE the messages if they were delivered promptly. Is there really nothing we can do about this? Thanks, Richard -- Forwarded message - From: Date: Sun, Feb 27, 2022 at 6:09 PM Subject: hobbes1069's mingw-fltk-1.3.8-1.fc36 completed To: Notification time stamped 2022-02-21 13:42:00 UTC hobbes1069's mingw-fltk-1.3.8-1.fc36 completed http://koji.fedoraproject.org/koji/buildinfo?buildID=1921324 ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
