[Mon Feb 16 10:50:00.885860 2015] [wsgi:error] [pid 7129:tid
139757846304512] ('RUNNING GARBAGE COLLECTOR', 1424083800.88584)
[Mon Feb 16 04:50:10.268822 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.268801)
[Mon Feb 16 04:50:10.271923 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.27191)
[Mon Feb 16 04:50:10.286823 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.286808)
[Mon Feb 16 04:50:10.373270 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.373256)
[Mon Feb 16 04:50:10.374693 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.374681)
[Mon Feb 16 04:50:10.376588 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.376579)
[Mon Feb 16 04:50:10.385338 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.385328)
[Mon Feb 16 04:50:10.400635 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.400624)
[Mon Feb 16 04:50:10.402888 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.402874)
[Mon Feb 16 04:50:10.405370 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.405357)
[Mon Feb 16 04:50:10.406246 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.406237)
[Mon Feb 16 04:50:10.408119 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.408108)
[Mon Feb 16 04:50:10.421543 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.421529)
[Mon Feb 16 04:50:10.422140 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.42213)
[Mon Feb 16 04:50:10.422850 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.42284)
[Mon Feb 16 04:50:10.423643 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.423633)
[Mon Feb 16 04:50:10.424818 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.424808)
[Mon Feb 16 04:50:10.425857 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.425847)
[Mon Feb 16 04:50:10.427138 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.427128)
[Mon Feb 16 04:50:10.430634 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.430623)
[Mon Feb 16 04:50:10.431942 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.431932)
[Mon Feb 16 04:50:10.432624 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.432615)
[Mon Feb 16 04:50:10.433388 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.433378)
[Mon Feb 16 04:50:10.442762 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.442749)
[Mon Feb 16 04:50:10.443735 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.443726)
[Mon Feb 16 04:50:10.444698 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.444687)
[Mon Feb 16 04:50:10.448658 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.448648)
[Mon Feb 16 04:50:10.450024 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.450013)
[Mon Feb 16 04:50:10.450990 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.450978)
[Mon Feb 16 04:50:10.451722 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.45171)
[Mon Feb 16 04:50:10.454838 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.454828)
[Mon Feb 16 04:50:10.460671 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.460656)
[Mon Feb 16 04:50:10.462460 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.462445)
[Mon Feb 16 04:50:10.479175 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.479142)
[Mon Feb 16 04:50:10.482813 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.482795)
[Mon Feb 16 04:50:10.486749 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.486737)
[Mon Feb 16 04:50:10.487875 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.487864)
[Mon Feb 16 04:50:10.489945 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.489933)
[Mon Feb 16 04:50:10.497272 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.497261)
[Mon Feb 16 04:50:10.507668 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.507654)
[Mon Feb 16 04:50:10.515107 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.515095)
[Mon Feb 16 04:50:10.515717 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.515708)
[Mon Feb 16 04:50:10.516797 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.516787)
[Mon Feb 16 04:50:10.519079 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.519069)
[Mon Feb 16 04:50:10.520464 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.520454)
[Mon Feb 16 04:50:10.521919 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.521908)
[Mon Feb 16 04:50:10.525004 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.524993)
[Mon Feb 16 04:50:10.525766 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.525755)
[Mon Feb 16 04:50:10.526721 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.526711)
[Mon Feb 16 04:50:10.527989 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.527975)
[Mon Feb 16 04:50:10.528724 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.528713)
[Mon Feb 16 04:50:10.529560 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.529548)
[Mon Feb 16 04:50:10.530651 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.530639)
[Mon Feb 16 04:50:10.531180 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.531171)
[Mon Feb 16 04:50:10.532407 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.532398)
[Mon Feb 16 04:50:10.533920 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.533908)
[Mon Feb 16 04:50:10.535317 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.535306)
[Mon Feb 16 04:50:10.537026 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.537013)
[Mon Feb 16 04:50:10.539757 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.539746)
[Mon Feb 16 04:50:10.541385 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.541375)
[Mon Feb 16 04:50:10.542175 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.542165)
[Mon Feb 16 04:50:10.543859 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.543847)
[Mon Feb 16 04:50:10.546160 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.54615)
[Mon Feb 16 04:50:10.548149 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.548138)
[Mon Feb 16 04:50:10.553919 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.553908)
[Mon Feb 16 04:50:10.555593 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.555583)
[Mon Feb 16 04:50:10.557226 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.557216)
[Mon Feb 16 04:50:10.561522 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.561513)
[Mon Feb 16 04:50:10.566261 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.566251)
[Mon Feb 16 04:50:10.570285 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.570274)
[Mon Feb 16 04:50:10.573015 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.573005)
[Mon Feb 16 04:50:10.573579 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.573569)
[Mon Feb 16 04:50:10.574279 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.57427)
[Mon Feb 16 04:50:10.575118 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.575108)
[Mon Feb 16 04:50:10.576658 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.576648)
[Mon Feb 16 04:50:10.577981 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.577971)
[Mon Feb 16 04:50:10.579068 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.579058)
[Mon Feb 16 04:50:10.585690 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.58568)
[Mon Feb 16 04:50:10.587592 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.587582)
[Mon Feb 16 04:50:10.588456 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.588446)
[Mon Feb 16 04:50:10.589260 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.589248)
[Mon Feb 16 04:50:10.591402 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.591391)
[Mon Feb 16 04:50:10.594528 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.594518)
[Mon Feb 16 04:50:10.599544 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.599533)
[Mon Feb 16 04:50:10.601261 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.601251)
[Mon Feb 16 04:50:10.603244 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.603234)
[Mon Feb 16 04:50:10.605063 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.605053)
[Mon Feb 16 04:50:10.606870 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.60686)
[Mon Feb 16 04:50:10.608601 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.60859)
[Mon Feb 16 04:50:10.613593 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.613583)
[Mon Feb 16 04:50:10.616680 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.61667)
[Mon Feb 16 04:50:10.617933 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.617923)
[Mon Feb 16 04:50:10.622714 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.622702)
[Mon Feb 16 04:50:10.629913 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.629897)
[Mon Feb 16 04:50:10.632170 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.632157)
[Mon Feb 16 04:50:10.642137 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.642118)
[Mon Feb 16 04:50:10.652818 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.6528)
[Mon Feb 16 04:50:10.657358 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.657343)
[Mon Feb 16 04:50:10.665102 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.66509)
[Mon Feb 16 04:50:10.668676 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.668665)
[Mon Feb 16 04:50:10.672610 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.6726)
[Mon Feb 16 04:50:10.700031 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.700009)
[Mon Feb 16 04:50:10.706635 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.706613)
[Mon Feb 16 04:50:10.715080 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.715062)
[Mon Feb 16 04:50:10.718759 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.718746)
[Mon Feb 16 04:50:10.724405 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.724387)
[Mon Feb 16 04:50:10.727816 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.727801)
[Mon Feb 16 04:50:10.731334 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.731323)
[Mon Feb 16 04:50:10.734882 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.734871)
[Mon Feb 16 04:50:10.738411 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.738399)
[Mon Feb 16 04:50:10.742037 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.742026)
[Mon Feb 16 04:50:10.752276 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.752261)
[Mon Feb 16 04:50:10.797924 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.797895)
[Mon Feb 16 04:50:10.829280 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.82926)
[Mon Feb 16 04:50:10.869706 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083810.869683)
[Mon Feb 16 04:50:13.167475 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083813.167457)
[Mon Feb 16 04:50:13.173208 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083813.173188)
[Mon Feb 16 04:50:13.198143 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083813.19812)
[Mon Feb 16 04:50:13.201056 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083813.201047)
[Mon Feb 16 04:50:13.203918 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083813.203908)
[Mon Feb 16 04:50:13.213175 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083813.213157)
[Mon Feb 16 04:50:13.216908 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083813.216898)
[Mon Feb 16 04:50:13.220571 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083813.220561)
[Mon Feb 16 04:50:13.224233 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083813.224218)
[Mon Feb 16 04:50:13.227812 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083813.227801)
[Mon Feb 16 04:50:13.233307 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083813.233297)
[Mon Feb 16 04:50:13.242947 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083813.242936)
[Mon Feb 16 04:50:13.256530 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083813.25652)
[Mon Feb 16 04:50:15.246868 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083815.246842)
[Mon Feb 16 04:50:15.252987 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083815.252944)
[Mon Feb 16 04:50:15.281071 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083815.28105)
[Mon Feb 16 04:50:15.284121 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083815.284111)
[Mon Feb 16 04:50:15.293618 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083815.293608)
[Mon Feb 16 04:50:15.297436 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083815.297424)
[Mon Feb 16 04:50:15.301093 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083815.301083)
[Mon Feb 16 04:50:15.305027 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083815.305015)
[Mon Feb 16 04:50:15.316323 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083815.316312)
[Mon Feb 16 04:50:15.321785 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083815.321773)
[Mon Feb 16 04:50:15.329168 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083815.329149)
[Mon Feb 16 04:50:15.335019 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083815.335005)
[Mon Feb 16 04:50:17.494926 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083817.494908)
[Mon Feb 16 04:50:17.501100 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083817.501082)
[Mon Feb 16 04:50:17.524260 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083817.524237)
[Mon Feb 16 04:50:17.527176 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083817.527163)
[Mon Feb 16 04:50:17.536892 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083817.536872)
[Mon Feb 16 04:50:17.540900 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083817.540881)
[Mon Feb 16 04:50:17.544676 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083817.544664)
[Mon Feb 16 04:50:17.548432 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083817.548411)
[Mon Feb 16 04:50:17.552149 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083817.552129)
[Mon Feb 16 04:50:17.557825 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083817.557804)
[Mon Feb 16 04:50:17.562587 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083817.56257)
[Mon Feb 16 04:50:17.567694 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083817.567684)
[Mon Feb 16 04:50:21.178826 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.178808)
[Mon Feb 16 04:50:21.185300 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.185285)
[Mon Feb 16 04:50:21.194050 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.19404)
[Mon Feb 16 04:50:21.210312 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.210301)
[Mon Feb 16 04:50:21.223734 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.223724)
[Mon Feb 16 04:50:21.236545 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.236535)
[Mon Feb 16 04:50:21.269957 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.269943)
[Mon Feb 16 04:50:21.300233 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.300215)
[Mon Feb 16 04:50:21.314732 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.314717)
[Mon Feb 16 04:50:21.329057 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.329047)
[Mon Feb 16 04:50:21.344399 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.344389)
[Mon Feb 16 04:50:21.358038 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.358028)
[Mon Feb 16 04:50:21.382785 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.382775)
[Mon Feb 16 04:50:21.408774 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.408763)
[Mon Feb 16 04:50:21.487170 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.487139)
[Mon Feb 16 04:50:21.500866 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.500856)
[Mon Feb 16 04:50:21.513610 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.513599)
[Mon Feb 16 04:50:21.526114 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.526104)
[Mon Feb 16 04:50:21.542886 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.542875)
[Mon Feb 16 04:50:21.562428 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.562411)
[Mon Feb 16 04:50:21.577404 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.577385)
[Mon Feb 16 04:50:21.592142 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.592131)
[Mon Feb 16 04:50:21.626810 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.6268)
[Mon Feb 16 04:50:21.641754 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.641744)
[Mon Feb 16 04:50:21.666358 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.666349)
[Mon Feb 16 04:50:21.705918 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.7059)
[Mon Feb 16 04:50:21.727509 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.727491)
[Mon Feb 16 04:50:21.767890 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.767873)
[Mon Feb 16 04:50:21.785718 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.785702)
[Mon Feb 16 04:50:21.815524 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.815503)
[Mon Feb 16 04:50:21.844980 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.844961)
[Mon Feb 16 04:50:21.892451 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083821.892441)
[Mon Feb 16 04:50:22.056950 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083822.056932)
[Mon Feb 16 04:50:22.091341 2015] [wsgi:error] [pid 7129:tid
139757296285440] ('RUNNING GARBAGE COLLECTOR', 1424083822.091332)
[Mon Feb 16 04:50:22.178069 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083822.178051)
[Mon Feb 16 04:50:22.183973 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083822.183962)
[Mon Feb 16 04:50:22.201901 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083822.201881)
[Mon Feb 16 04:50:22.204731 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083822.204723)
[Mon Feb 16 04:50:22.208458 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083822.208448)
[Mon Feb 16 04:50:22.218118 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083822.218106)
[Mon Feb 16 04:50:22.222159 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083822.22215)
[Mon Feb 16 04:50:22.226123 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083822.226113)
[Mon Feb 16 04:50:22.230080 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083822.23007)
[Mon Feb 16 04:50:22.235619 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083822.235608)
[Mon Feb 16 04:50:22.240913 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083822.240902)
[Mon Feb 16 04:50:22.244122 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083822.24411)
[Mon Feb 16 04:50:22.258841 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083822.258825)
[Mon Feb 16 04:50:41.466300 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083841.466282)
[Mon Feb 16 04:50:41.489648 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083841.489627)
[Mon Feb 16 04:50:41.492478 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083841.492469)
[Mon Feb 16 04:50:41.496313 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083841.496303)
[Mon Feb 16 04:50:41.506000 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083841.505989)
[Mon Feb 16 04:50:41.510034 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083841.510022)
[Mon Feb 16 04:50:41.514008 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083841.513998)
[Mon Feb 16 04:50:41.517985 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083841.517976)
[Mon Feb 16 04:50:41.523027 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083841.523017)
[Mon Feb 16 04:50:41.527916 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083841.527906)
[Mon Feb 16 04:50:41.531557 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083841.531547)
[Mon Feb 16 04:50:41.542040 2015] [wsgi:error] [pid 7129:tid
139757545793280] ('RUNNING GARBAGE COLLECTOR', 1424083841.542022)
[Mon Feb 16 04:50:43.276978 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083843.276958)
[Mon Feb 16 04:50:43.301118 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083843.301097)
[Mon Feb 16 04:50:43.303988 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083843.303978)
[Mon Feb 16 04:50:43.307828 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083843.307817)
[Mon Feb 16 04:50:43.317347 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083843.317337)
[Mon Feb 16 04:50:43.321355 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083843.321345)
[Mon Feb 16 04:50:43.325343 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083843.325333)
[Mon Feb 16 04:50:43.329243 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083843.329232)
[Mon Feb 16 04:50:43.334295 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083843.33428)
[Mon Feb 16 04:50:43.340423 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083843.340413)
[Mon Feb 16 04:50:43.357043 2015] [wsgi:error] [pid 7129:tid
139757388543744] ('RUNNING GARBAGE COLLECTOR', 1424083843.357031)
On Monday, February 16, 2015 at 12:37:32 PM UTC+2, Graham Dumpleton wrote:
>
> Put this in your WSGI script file (wsgi.py).
>
> import time
> import threading
>
> class Monitor(object):
>
> initialized = False
> lock = threading.Lock()
>
> count = 0
>
> @classmethod
> def initialize(cls):
> with Monitor.lock:
> if not cls.initialized:
> cls.initialized = True
> cls.rollover()
>
> @staticmethod
> def rollover():
> print('RUNNING GARBAGE COLLECTOR', time.time())
>
> class Object(object):
> pass
>
> o1 = Object()
> o2 = Object()
>
> o1.o = o2
> o2.o = o1
>
> o1.t = Monitor()
>
> del o1
> del o2
>
> def __del__(self):
> global count
> Monitor.count += 1
> Monitor.rollover()
>
> Monitor.initialize()
>
> Then monitor the log file and see if it periodically outputs 'RUNNING
> GARBAGE COLLECTOR' or whether it stops being output after a while.
>
> Graham
>
> On 16/02/2015, at 9:29 PM, Paul Royik <[email protected] <javascript:>>
> wrote:
>
> I'm sorry. Django 1.7.1
>
> On Monday, February 16, 2015 at 12:19:33 PM UTC+2, Graham Dumpleton wrote:
>
> I asked what version of Django are you running? Not Python.
>
> Graham
>
> On 16/02/2015, at 9:12 PM, Paul Royik <[email protected]> wrote:
>
> I'm using Python 2.7.9
> So, what solution do you propose?
>
> Is there any way to kill a thread? Because things now are working worse
> than wih unstoppable thread. I'm hitting request-timeout.
> Also memory grows on every request.
> Maybe there is way to kill thread on external calculations?
>
> On Monday, February 16, 2015 at 12:04:23 PM UTC+2, Graham Dumpleton wrote:
>
> What version of Django are you running? Older Django versions have a bug
> in them which can cause the Python garbage collector to block and no longer
> run. Memory usage will go up because Python objects are not reclaimed
> properly. Not out of the question that other third party libraries could
> cause this also and can see a correlation between it and the fact that you
> are hitting queue time where request threads are blocking on a thread mutex.
>
> The MaxRequestWorkers warning is because you ran out of capacity in the
> daemon processes due to all your long running requests and/or hung
> requests. The fact you are hitting the request timeout means that the
> daemon process likely has stopped taking more requests and eventually as
> new requests back up, the Apache child worker process eventually complain
> as they also run out of capacity in their capacity of proxying requests.
>
> So it is an outcome of the problems you are having. You still need to work
> out the underlying problem.
>
> Graham
>
> On 16/02/2015, at 8:53 PM, Paul Royik <[email protected]> wrote:
>
> I also got following error.
> server reached MaxRequestWorkers setting, consider raising the
> MaxRequestWorkers setting
>
> On Sunday, February 15, 2015 at 10:42:19 PM UTC+2, Paul Royik wrote:
>
> As I discovered, memory grows on every request, not only calculator, even
> in admin. Situation is very close to this:
> http://stackoverflow.com/questions/2293333/django-memory-usage-going-up-with-every-request
> I hit 3 GB. It is the first time.
>
> On Sunday, February 15, 2015 at 1:59:06 PM UTC+2, Paul Royik wrote:
>
> It grows. Below is list between subsequent requests.
> 78988
> 85503
> 92873
> 100237
>
> On Sunday, February 15, 2015 at 1:36:15 PM UTC+2, Graham Dumpleton wrote:
>
> It being empty is fine.
>
> At least is not caused by uncollectable objects.
>
> Next would be to trying printing out periodically:
>
> len(gc.get_objects())
>
> and see if it grows over time.
>
> This is not conclusive either as meaning anything, but if it does keep
> growing, still useful to know.
>
> Graham
>
> On 15/02/2015, at 10:32 PM, Paul Royik <[email protected]> wrote:
>
> Don't know why but gc.garbage is empty list.
>
> On Sunday, February 15, 2015 at 1:04:15 PM UTC+2, Graham Dumpleton wrote:
>
> If the objects cannot be reclaimed due to Python objects cycles where
> objects have __del__() methods, they simply are not able to be reclaimed.
>
> You can try:
>
> import gc
> gc.collect()
>
> but will likely have little effect.
>
> I would suggest you also read:
>
> http://christian.hofstaedtler.name/blog/2013/01/gc-garbage.html
>
> Graham
>
> On 15/02/2015, at 10:00 PM, Paul Royik <[email protected]> wrote:
>
> Can I force cleaning memory after algorithm has finished?
>
> On Sunday, February 15, 2015 at 12:53:30 PM UTC+2, Graham Dumpleton wrote:
>
>
> On 15/02/2015, at 9:41 PM, Paul Royik <[email protected]> wrote:
>
> I reduced possibility by restricting very long and complex expressions.
>
> Code, that I use is simple.
>
> class TimeLimit(object):
> def __init__(self, timeout=60):
> self.timeout = timeout
> self.end = None
>
> def check_timeout(self):
> if self.end and time.time() > self.end:
> raise TimeoutException
> else:
> self.start()
>
> def start(self):
> if not self.end:
> self.end = time.time() + self.timeout
>
> class MainClass(object):
> def __init__(self):
> self.timer = TimeLimit()
> algo(timer)
>
>
> def algo(timer):
> do_something()
> external()
> timer.check_timeout()
> algo(timer)
>
>
> Which ultimately is no different to what was being done with the decorator.
>
> If external() can run a lot longer than the timeout, either way is not
> going to be much use. It is only going to be of much use if external() is
> always a short call and on subsequent calls you are simply calling it with
> different arguments.
>
> I debugged code thoroughly and it works as expected. And as shown memory
> is not drastically increased during execution, but later. Is that possible
> that reasong in apache conf? Or I need to look for problem in code? I have
> no clue where to search as everything works. Is it possible to see memory
> usage somehow on localhost (I'm under Windows)?
>
>
> It is unlikely to be the Apache configuration. It is more likely that
> after finishing the execution of the algorithm, the algorithm isn't
> clearing out any working data structures it creates when it is doing
> calculations and they are persisting in memory. Subsequent calls simply add
> more memory each time and so it grows in that way.
>
> There are various reasons that Python objects cannot be reclaimed and will
> keep using memory.
>
> There aren't really great tools for tracking that sort of stuff down.
>
> Graham
>
> On Sunday, February 15, 2015 at 12:31:36 PM UTC+2, Graham Dumpleton wrote:
>
>
> On 15/02/2015, at 9:26 PM, Paul Royik <[email protected]> wrote:
>
> Yes. External library could run long (although I reduced this possibility).
>
>
> How have you reduced possibility?
>
> So, what should I do with locking threads? Is it resolvable?
>
>
> No idea. It isn't my code. I can only highlight what looks suspicious in
> the log output.
>
> I'm just now playing with code on stackoverflow. People suggested to
> create class instead of decorator and pass its instance in all functions. I
> did as they proposed and it worked.
>
>
> Since you don't provide the code for that, I can't comment.
>
> However, with the flow of time I see increase in memory.
> So, now nobody access site.
>
> I ran command ps -u simamura -o pid,rss,command | awk '{print $0}{sum+=$2}
> END {print "Total", sum/1024, "MB"}' and see 200 MB.
> Then I invoke integral calculator, while it works above command shows 200
> MB.
> Calculator finished, I check memory - still 200 MB. In 10-15 minutes
> (remember nobody access the site) I see 240 MB.
>
> In result, memory increased from 130 MB (memory immediately after server
> restart) to 435 MB in 10 hours.
>
> How can I resolve this issue? I feel that in another 12 hours webfaction
> will kill the process, because I memomy will be over limit.
>
>
> Have you tried adding print() debug statements to your code to track
> through what is being called and to determine whether your code is doing
> what is expected?
>
> Graham
>
> On Sunday, February 15, 2015 at 12:12:39 PM UTC+2, Graham Dumpleton wrote:
>
>
> On 15/02/2015, at 9:03 PM, Paul Royik <[email protected]> wrote:
>
> How do you know that it runs too long?
>
>
> In explained the log output previously and what was happening.
>
> If you go back far enough in the logs even before what you provided you
> will see some messages like:
>
> [Fri Feb 06 12:39:06.254056 2015] [wsgi:info] [pid 9731:tid
> 140178879313664] mod_wsgi (pid=9731): Daemon process request time limit
> exceeded, stopping process 'localhost:20241'.
> [Fri Feb 06 12:39:06.254108 2015] [wsgi:info] [pid 9731:tid
> 140179136616192] mod_wsgi (pid=9731): Shutdown requested 'localhost:20241'.
> [Fri Feb 06 12:39:06.286339 2015] [wsgi:info] [pid 9731:tid
> 140179136616192] mod_wsgi (pid=9731): Dumping stack trace for active Python
> threads.
> [Fri Feb 06 12:39:06.286359 2015] [wsgi:info] [pid 9731:tid
> 140179136616192] mod_wsgi (pid=9731): Thread 140178316318464 executing file
> "/usr/local/lib/python2.7/threading.py", line 147, in acquire
> [Fri Feb 06 12:39:06.286363 2015] [wsgi:info] [pid 9731:tid
> 140179136616192] mod_wsgi (pid=9731): called from file
> "/home/simamura/lib/python2.7/sympy/core/compatibility.py", line 851, in
> wrapper,
> [Fri Feb 06 12:39:06.286366 2015] [wsgi:info] [pid 9731:tid
> 140179136616192] mod_wsgi (pid=9731): called from file
> "/home/simamura/lib/python2.7/sympy/core/cache.py", line 89, in wrapper,
> [Fri Feb 06 12:39:06.286369 2015] [wsgi:info] [pid 9731:tid
> 140179136616192] mod_wsgi (pid=9731): called from file
> "/home/simamura/lib/python2.7/sympy/core/operations.py", line 127, in
> _matches_commutative,
> [Fri Feb 06 12:39:06.286372 2015] [wsgi:info] [pid 9731:tid
> 140179136616192] mod_wsgi (pid=9731): called from file
> "/home/simamura/lib/python2.7/sympy/core/mul.py", line 808, in matches,
> [Fri Feb 06 12:39:06.286375 2015] [wsgi:info] [pid 9731:tid
> 140179136616192] mod_wsgi (pid=9731): called from file
> "/home/simamura/lib/python2.7/sympy/core/add.py", line 358, in
> _matches_simple,
> [Fri Feb 06 12:39:06.286378 2015] [wsgi:info] [pid 9731:tid
> 140179136616192] mod_wsgi (pid=9731): called from file
> "/home/simamura/lib/python2.7/sympy/core/operations.py", line 127, in
> _matches_commutative,
>
> So it says that request time limit was exceeded.
>
> This is followed by stack dumps for what all the request threads were
> doing.
>
> In the part you did show, they both are stuck in:
>
> [Sat Feb 14 19:58:00.261118 2015] [wsgi:info] [pid 12528:tid
> 139862016030464] mod_wsgi (pid=12528): Thread 139861715523328 executing
> file "/usr/local/lib/python2.7/threading.py", line 147, in acquire
> [Sat Feb 14 19:58:00.261120 2015] [wsgi:info] [pid 12528:tid
> 139862016030464] mod_wsgi (pid=12528): called from file
> "/home/simamura/lib/python2.7/sympy/core/compatibility.py", line 851, in
> wrapper,
> [Sat Feb 14 19:58:00.261122 2015] [wsgi:info] [pid 12528:tid
> 139862016030464] mod_wsgi (pid=12528): called from file
> "/home/simamura/lib/python2.7/sympy/core/cache.py", line 89, in wrapper,
> [Sat Feb 14 19:58:00.261125 2015] [wsgi:info] [pid 12528:tid
> 139862016030464] mod_wsgi (pid=12528): called from file
> "/home/simamura/lib/python2.7/sympy/core/mul.py", line 373, in _gather,
> [Sat Feb 14 19:58:00.261127 2015] [wsgi:info] [pid 12528:tid
> 139862016030464] mod_wsgi (pid=12528): called from file
> "/home/simamura/lib/python2.7/sympy/core/mul.py", line 98, in flatten,
>
> That is, they are both waiting on a thread lock.
>
> This suggests to me that the code isn't possibly dealing with locking
> properly, or a lock is held for a very long time while in some certain
> section and blocking other threads from running and so why they are delayed
> and take so long.
>
> No more than 300 seconds as I wrote in conf file (socket-timeout and
> request-timeout).
>
> I can not control it thoroughly.
> def algo():
> do_something()
> invoke_external_library() # this could run slighlty longer, so timeout
> is not precise
> check_timeout()
> algo()
>
>
> So the thing that takes the unpredictable amount of time and you need to
> interrupt is not even your code?
>
> Graham
>
> On Sunday, February 15, 2015 at 11:36:05 AM UTC+2, Graham Dumpleton wrote:
>
> So we are back the fact that your requests are still running too long and
> however you have set up the timeout check isn't working.
>
> Have you added print() debug statements in your code to validate that the
> die on timeout check is even being run regularly?
>
> To be able to distinguish each request thread, you can include the output
> of threading.currentThread() on the print statements.
>
> Graham
>
> On 15/02/2015, at 8:19 PM, Paul Royik <[email protected]> wrote:
>
> [Sat Feb 14 19:58:00.261309 2015] [wsgi:info] [pid 12528:tid
> 139862016030464] mod_wsgi (pid=12528): called from file
> "/home/simamura/lib/python2.7/mod_wsgi/server/__init__.py", line 1137, in
> handle_request.
> [Sat Feb 14 19:58:05.245850 2015] [wsgi:info] [pid 12528:tid
> 139861541500672] mod_wsgi (pid=12528): Aborting process 'localhost:20241'.
> [Sat Feb 14 19:58:05.245889 2015] [wsgi:info] [pid 12528:tid
> 139861541500672] mod_wsgi (pid=12528): Exiting process 'localhost:20241'.
> [Sun Feb 15 01:58:05.352861 2015] [wsgi:error] [pid 11272:tid
> 139862014424832] [client 127.0.0.1:48193] Truncated or oversized response
> headers received from daemon process 'localhost:20241':
> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
> [Sun Feb 15 01:58:05.352890 2015] [wsgi:error] [pid 11272:tid
> 139862015223552] [client 127.0.0.1:48563] Truncated or oversized response
> headers received from daemon process 'localhost:20241':
> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
> [Sun Feb 15 01:58:05.352958 2015] [wsgi:error] [pid 11272:tid
> 139861948524288] [client 127.0.0.1:48240] Truncated or oversized response
> headers received from daemon process 'localhost:20241':
> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
> [Sun Feb 15 01:58:05.352992 2015] [wsgi:error] [pid 11272:tid
> 139862015489792] [client 127.0.0.1:48440] Truncated or oversized response
> headers received from daemon process 'localhost:20241':
> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
> [Sun Feb 15 01:58:05.353008 2015] [wsgi:error] [pid 11666:tid
> 139862014957312] [client 127.0.0.1:48329] Truncated or oversized response
> headers received from daemon process 'localhost:20241':
> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
> [Sun Feb 15 01:58:05.353015 2015] [wsgi:error] [pid 11272:tid
> 139862014691072] [client 127.0.0.1:48620] Truncated or oversized response
> headers received from daemon process 'localhost:20241':
> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
> [Sun Feb 15 01:58:05.353081 2015] [wsgi:error] [pid 11272:tid
> 139861949323008] [client 127.0.0.1:48778] Truncated or oversized response
> headers received from daemon process 'localhost:20241':
> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
> [Sun Feb 15 01:58:05.353081 2015] [wsgi:error] [pid 11666:tid
> 139861948790528] [client 127.0.0.1:48735] Truncated or oversized response
> headers received from daemon process 'localhost:20241':
> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
> [Sun Feb 15 01:58:05.353096 2015] [wsgi:error] [pid 11666:tid
> 139862015223552] [client 127.0.0.1:48792] Truncated or oversized response
> headers received from daemon process 'localhost:20241':
> /home/simamura/webapps/django_math/express/htdocs/calculators, referer:
> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/
> [Sun Feb 15 01:58:05.972993 2015] [wsgi:info] [pid 24792:tid
> 139862016030464] mod_wsgi (pid=12528): Process 'localhost:20241' has died,
> deregister and restart it.
> [Sun Feb 15 01:58:06.081051 2015] [wsgi:info] [pid 24792:tid
> 139862016030464] mod_wsgi (pid=12528): Process 'localhost:20241' has been
> deregistered and will no longer be monitored.
> [Sun Feb 15 01:58:06.081333 2015] [wsgi:info] [pid 23028:tid
> 139862016030464] mod_wsgi (pid=23028): Starting process 'localhost:20241'
> with threads=10.
> [Sun Feb 15 01:58:06.163910 2015] [wsgi:info] [pid 23028:tid
> 139862016030464] mod_wsgi (pid=23028): Python home /usr/local.
> [Sun Feb 15 01:58:06.163937 2015] [wsgi:info] [pid 23028:tid
> 139862016030464] mod_wsgi (pid=23028): Initializing Python.
> [Sun Feb 15 01:58:06.534740 2015] [wsgi:info] [pid 23028:tid
> 139862016030464] mod_wsgi (pid=23028): Attach interpreter ''.
> [Sun Feb 15 01:58:06.549849 2015] [wsgi:info] [pid 23028:tid
> 139862016030464] mod_wsgi (pid=23028): Imported 'mod_wsgi'.
> [Sun Feb 15 01:58:06.549942 2015] [wsgi:info] [pid 23028:tid
> 139862016030464] mod_wsgi (pid=23028, process='localhost:20241',
> application=''): Loading WSGI script
> '/home/simamura/webapps/django_math/express/handler.wsgi'.
> [Sun Feb 15 02:06:36.476939 2015] [core:info] [pid 11272:tid
> 139861948524288] [client 127.0.0.1:56156] AH00128: File does not exist:
> /home/simamura/webapps/math_root/browserconfig.xml
>
>
> And that's all.
>
> On Sunday, February 15, 2015 at 10:11:25 AM UTC+2, Graham Dumpleton wrote:
>
> Provide me with the logging from a period before the restart until a
> period after.
>
> The log can be very informative if you know what to look for.
>
> Graham
>
> On 15/02/2015, at 6:43 PM, Paul Royik <[email protected]> wrote:
>
> I actually have custom error 500 page. This means, that when Django throws
> error, custom page should be shown.
> But it is not the case. Standard white-background apache page is shown.
> So, I think that problem is not in Django exception. Otherwise Django
> would show custom page, which is not the case.
> In logs I saw only that server was restarted.
> Can I somehow tune logging, so that real error is written?
>
> On Sunday, February 15, 2015 at 2:44:50 AM UTC+2, Graham Dumpleton wrote:
>
>
> On 15/02/2015, at 10:53 AM, Paul Royik <[email protected]> wrote:
>
> > I already tried this approach.
> >
> > It seems good.
> > But when I tested it on server, by simultaneously executing page on two
> different computers, it gave me 500 error.
> >
> > How this can be explained? Again something with apache? Logs didn't show
> anything. But I noticed again serious memory usage. And this happens only
> when I use code for limiting time.
>
>
> Some tips for you about ensuring you get the best help.
>
> Stop discarding the message content for the message you are replying to.
> It can be frustrating to have to keep going back to old messages to see
> what was said originally and what you may be following up to, especially
> when on a phone. When asking questions on mailing list, let the person you
> are asking questions of decide what can be discarded from the message chain
> as they will know better what should be kept in the message to provide easy
> access to important information for context.
>
> When you reply to say you are have a new error, provide the actual code
> you were using exactly at the time you had the error. Saves the person who
> is helping you having to go back and ask for it if it is unclear what code
> you were using.
>
> Your messages here and on StackOverflow show that you play around with the
> example code I am giving you and I have reduced confidence you were running
> with the code I suggested at the time of the problem you are now relating
> to.
>
> If you are getting 500 errors and nothing is being logged, it is because
> Django is capturing the exception and converting it to a generic 500 error
> response page. Configure Django to send you emails with the details of the
> exceptions. If you are on a local system, then set DEBUG=True in the Django
> settings so the details of the error are shown in response that goes back
> to the browser.
>
> As to the code, I did test it this time, including under mod_wsgi with 20
> request threads and hitting it with concurrent and successive requests with
> a benchmarking tool. I saw no issues and it appeared to behave as I would
> expect.
>
> So in a test2.py file I had:
>
> import time
> import functools
> import threading
>
> def time_limit(seconds):
> def decorator(func):
>
>
> ...
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
