These are great questions! 

Dec 26 we released 1.4.1 (again by merging a PR opened by dependabot). 
Dec 23 we moved from 1.3.1 to 1.4.0

So seems like all these bumps were just a week or less each. However in 
about two weeks we didn't have any incidents.



In other news, today (on 1.4.1) we had a similar incident where a process 
was stuck at 100% cpu, but after about 2-3 minutes it recovered. We had 
some requests lost due to timeout on heroku router (30 seconds).

I'm thinking about doing a few changes now:

1) move to 1.4.2 (since 1.4.1 exhibits the same behavior)
2) use --expose-tracebacks 
3) use --channel-timeout=29 (to make sure waitress times out before the 
socket is closed by heroku router)

If anyone has a good insight on how to enable further request logging 
(beyond the path that heroku already gives me) that would be great.

--peter

On Thursday, January 9, 2020 at 10:14:41 PM UTC-5, Bert JW Regeer wrote:
Another question:


How long were you using 1.4.1 before you upgrade to 1.4.2? Did you use 
1.4.0 for any period of time before that? I am trying to narrow down what 
the potential changeset could be.


Thanks,
Bert JW Regeer



On Jan 9, 2020, at 07:51, 'Peter Lada' via pylons-discuss <
[email protected] <javascript:>> wrote:

Github dependabot has opened a PR for me to upgrade to 1.4.2 (thanks for 
the release), and I merged it yesterday around 1730. 


At around 1930 one of the 6 dynos (heroku, 1X instance, single CPU, 0.5GB 
RAM) has come to get pegged at 1.0 load and timed out every subsequent 
request (heroku router cuts connection after 30s).


At 1945 it happened to another dyno. 


At around 2000 I restarted the dynos and the problem got rectified, 
probably temporarily.


I've reverted to 1.4.1 and the issue has not surfaced since (12 plus hours).


Has anyone else used 1.4.2 in production? Any issues?

Sadly I cannot provide more info, beyond the 1,5,15-minute load avg graphs 
as the logs just show timed out requests and no other info.


--peter
Formsort.com <http://formsort.com/>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an 
email to [email protected] <javascript:>.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/54aa81bf-b935-4afc-b71c-f52d1fb15516%40googlegroups.com
 
<https://groups.google.com/d/msgid/pylons-discuss/54aa81bf-b935-4afc-b71c-f52d1fb15516%40googlegroups.com?utm_medium=email&utm_source=footer>
.
<Screen Shot 2020-01-09 at 10.42.23.png>

-- 
You received this message because you are subscribed to the Google Groups 
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/pylons-discuss/27441d74-50d5-4882-80b5-e081102cf776%40googlegroups.com.

Reply via email to