I know nothing about our infraestructure but, is possible run proyects in development, like Sugar Network in a different server/vm than critical services like git?
Gonzalo On Sat, Apr 12, 2014 at 6:07 AM, Sebastian Silva <sebast...@fuentelibre.org>wrote: > Here I just got home. Sorry for the inconvenience I might have caused. > > Bernie, do you know which log was/is growing out of hand? > > Here's a report on everything I know about the issue. > We've been experiencing some performance degradation and also some > downtime in Sugar Network services (this is documented at > http://tareas.somosazucar.org/hxp/issue71 ). > We've seen a burst in users since deployment OS images with Sugar Network > features ( http://network.sugarlabs.org/stats-viewer/ growing pretty fast > user_total). > There is a notification feature that is polling the sugar network node > service. > This was causing the allocation and exhaustion of resources (open files). > Crashes got to a frequency of every hour or so. > It's code I don't understand really well, but I went ahead and patched the > Sugar Network with: > http://tareas.somosazucar.org/hxp/file66/sn_disable_notifications.patch > This made the SN much snappier and it stopped crashing. However logs were > saving a traceback several times per second. I thought I had contained the > log issue but apparently I missed some other logs (I guess apache logs but > they seem clean now). > > I took a glance at jita and could not find the growing log. > > Let me know where I can help mitigation. > > Regards > Sebastian > > > El vie, 11 de abr 2014 a las 7:51 PM, Bernie Innocenti < > ber...@sugarlabs.org> escribió: > > I was notified that git.sugarlabs.org was showing errors. After some head > scraping I realized that the root filesystem on jita was full. I looked > around and found giant request logs containing millions of requests > apparently originating from XOs located in Peru. We've been DDOSed by our > own creature :-) Anyway, the machine also had a giant, very fragmented > mysql database that I'm currently cleaning up. Gitorious will be back > online in less than 1 hour. Contact me on IRC if this is blocking your > work, I can postpone the maintenance. > -- > Bernie Innocenti Sugar Labs Infrastructure Team > http://wiki.sugarlabs.org/go/Infrastructure_Team_______________________________________________ > Sugar-devel mailing list > Sugar-devel@lists.sugarlabs.org > http://lists.sugarlabs.org/listinfo/sugar-devel > > > _______________________________________________ > Sugar-devel mailing list > Sugar-devel@lists.sugarlabs.org > http://lists.sugarlabs.org/listinfo/sugar-devel > > -- Gonzalo Odiard SugarLabs - Software for children learning
_______________________________________________ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel