Re: [Sugar-devel] [Systems] Error on trac reading a bug #4863

[Bernie Innocenti]

By the way, found this relevant article today on Hacker News:

  http://sirupsen.com/production-docker/

The comments are also interesting:

  https://news.ycombinator.com/item?id=9961537


On 28/07/15 19:48, Bernie Innocenti wrote:

On 07/28/2015 09:27 AM, Samuel Cantero wrote:

Hi All,

On Tue, Jul 28, 2015 at 6:52 AM, Sam P. mailto:sam@sam.today>> wrote:

 Hi Bernie,

 On Tue, Jul 28, 2015 at 2:12 PM Bernie Innocenti mailto:ber...@codewiz.org>> wrote:

 1. We need monitoring for containers health with notifications when
 parameters go off thresholds. Similar to what Munin does.


 I'm not sure how that would help us.  A container running out of
 resources doesn't crash - parsoid runs almost using all the
 allocated memory and parsoid+trac+others hit their cpu quotas
 sometimes and just keep running.


A container running out of memory dies. By default, the memory limit
value for the container is set to twice as much as the memory parameter
we specify while starting a container. The limit value is the sum of
memory and swap.

We do not have swap accounting enabled yet on freedom, but for sure we
are going to have it at the next reboot.


In the best Google tradition, freedom doesn't have any swap space at all :-)



You can test it with the stress image (as we did before for the CPU):

sudo docker run --rm -m 128m --name testmem scg/stress --vm 1 --vm-bytes
256M

If you allocate over twice the amount of memory we assign you will get
the following error:

stress: info: [1] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
stress: FAIL: [1] (415) <-- worker 7 got signal 9
stress: WARN: [1] (417) now reaping child worker processes
stress: FAIL: [1] (421) kill error: No such process
stress: FAIL: [1] (451) failed run completed in 3s

By default, Docker kills processes in a container if an out-of-memory
(OOM) error occurs. We can change this behaviour on a container but we
are not doing it now.


See my response to Sam P.: killing the entire container is almost always
the right response to an OOM.



 Maybe it would be more useful to monitor client metrics, like can we
 reach the service over http?

I guess It would be useful. We can do it with nagios.


Cool, does it support Docker? I used Nagios a long time ago at the FSF
and it was pretty good. The reason I was using munin at SL is that it
was easier to setup and I didn't know Nagios back then. Moreover, Munin
seems a bit of a dead project nowadays, so I'd be happy to switch.

Can we run the Nagios frontend in a container too?



 That is a feature built into docker.  I've enabled it in
 container.yml (as a global default) and restarted trac with the new
 parameters.

Great. Maybe we need to enable historical resource usage in cadvisor.
This would help us to understand if a container has crashed because it
has run out of resources.


Cool, another useful feature I didn't know about. Yes, please, let's
enable resource graphing. Is there an http frontend to look at the numbers?



 Thanks,
 Sam



 3. Since high-profile services have been deployed using Docker for a
 while, software for doing (1) and (2) certainly exists already :-)


 On 27/07/15 22:32, Samuel Cantero wrote:
 > I didn't shut it down. The trac container was already off when
 I checked
 > it.
 >
 > On Mon, Jul 27, 2015 at 9:40 PM, Sam P.  >> wrote:
 >
 > Did you check the logs before you shut it down?
 >
 >
 > On Tue, 28 Jul 2015 9:38 am Samuel Cantero
 mailto:s...@sugarlabs.org>
 > >> wrote:
 >
 > Hi Gonzalo,
 >
 > The trac container was down. I just recently turned it on.
 >
 > Somebody turned it off for some reason? If it not the
 case,
 > maybe the container has reached its maximum memory or
 processor
 > limit. The docker container stops in that case.
 >
 > Greetings,
 >
 >
 >
 > On Mon, Jul 27, 2015 at 5:01 PM, Gonzalo Odiard
 > mailto:godi...@sugarlabs.org>
 >>
 wrote:
 >
 > Today, I get a error
 >
 > 502 Bad Gateway
 > nginx/1.8.0
 >
 > when try access trac.
 >
 > Gonzalo
 >
 > On Sat, Jul 25, 2015 at 8:40 PM, Gonzalo Odiard
 > mailto:godi...@sugarlabs.org> >> wrote:
 >
 > Thanks Sams :)
 >
 > On Sat, Jul 25, 2015 at 3:5

Re: [Sugar-devel] [Systems] Error on trac reading a bug #4863

[Bernie Innocenti]

On 07/28/2015 09:27 AM, Samuel Cantero wrote:
> Hi All,
> 
> On Tue, Jul 28, 2015 at 6:52 AM, Sam P.  > wrote:
> 
> Hi Bernie,
> 
> On Tue, Jul 28, 2015 at 2:12 PM Bernie Innocenti  > wrote:
> 
> 1. We need monitoring for containers health with notifications when
> parameters go off thresholds. Similar to what Munin does.
> 
> 
> I'm not sure how that would help us.  A container running out of
> resources doesn't crash - parsoid runs almost using all the
> allocated memory and parsoid+trac+others hit their cpu quotas
> sometimes and just keep running.
> 
> 
> A container running out of memory dies. By default, the memory limit
> value for the container is set to twice as much as the memory parameter
> we specify while starting a container. The limit value is the sum of
> memory and swap. 
> 
> We do not have swap accounting enabled yet on freedom, but for sure we
> are going to have it at the next reboot.

In the best Google tradition, freedom doesn't have any swap space at all :-)


> You can test it with the stress image (as we did before for the CPU):
> 
> sudo docker run --rm -m 128m --name testmem scg/stress --vm 1 --vm-bytes
> 256M
> 
> If you allocate over twice the amount of memory we assign you will get
> the following error:
> 
> stress: info: [1] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
> stress: FAIL: [1] (415) <-- worker 7 got signal 9
> stress: WARN: [1] (417) now reaping child worker processes
> stress: FAIL: [1] (421) kill error: No such process
> stress: FAIL: [1] (451) failed run completed in 3s
> 
> By default, Docker kills processes in a container if an out-of-memory
> (OOM) error occurs. We can change this behaviour on a container but we
> are not doing it now.

See my response to Sam P.: killing the entire container is almost always
the right response to an OOM.


> Maybe it would be more useful to monitor client metrics, like can we
> reach the service over http?
> 
> I guess It would be useful. We can do it with nagios. 

Cool, does it support Docker? I used Nagios a long time ago at the FSF
and it was pretty good. The reason I was using munin at SL is that it
was easier to setup and I didn't know Nagios back then. Moreover, Munin
seems a bit of a dead project nowadays, so I'd be happy to switch.

Can we run the Nagios frontend in a container too?


> That is a feature built into docker.  I've enabled it in
> container.yml (as a global default) and restarted trac with the new
> parameters.
> 
> Great. Maybe we need to enable historical resource usage in cadvisor.
> This would help us to understand if a container has crashed because it
> has run out of resources.

Cool, another useful feature I didn't know about. Yes, please, let's
enable resource graphing. Is there an http frontend to look at the numbers?


> Thanks,
> Sam
>  
> 
> 
> 3. Since high-profile services have been deployed using Docker for a
> while, software for doing (1) and (2) certainly exists already :-)
> 
> 
> On 27/07/15 22:32, Samuel Cantero wrote:
> > I didn't shut it down. The trac container was already off when
> I checked
> > it.
> >
> > On Mon, Jul 27, 2015 at 9:40 PM, Sam P.  > >> wrote:
> >
> > Did you check the logs before you shut it down?
> >
> >
> > On Tue, 28 Jul 2015 9:38 am Samuel Cantero
> mailto:s...@sugarlabs.org>
> > >> wrote:
> >
> > Hi Gonzalo,
> >
> > The trac container was down. I just recently turned it on.
> >
> > Somebody turned it off for some reason? If it not the
> case,
> > maybe the container has reached its maximum memory or
> processor
> > limit. The docker container stops in that case.
> >
> > Greetings,
> >
> >
> >
> > On Mon, Jul 27, 2015 at 5:01 PM, Gonzalo Odiard
> > mailto:godi...@sugarlabs.org>
> >>
> wrote:
> >
> > Today, I get a error
> >
> > 502 Bad Gateway
> > nginx/1.8.0
> >
> > when try access trac.
> >
> > Gonzalo
> >
> > On Sat, Jul 25, 2015 at 8:40 PM, Gonzalo Odiard
> >    >> wrote:
> >
> > Thanks Sams :)
> >
> > On Sat, Jul 25, 2015 at 3:58 PM, Samuel Cantero
> > mail

Re: [Sugar-devel] [Systems] Error on trac reading a bug #4863

[Bernie Innocenti]

On 07/28/2015 06:52 AM, Sam P. wrote:
> Hi Bernie,
> 
> On Tue, Jul 28, 2015 at 2:12 PM Bernie Innocenti  > wrote:
> 
> 1. We need monitoring for containers health with notifications when
> parameters go off thresholds. Similar to what Munin does.
> 
> 
> I'm not sure how that would help us.  A container running out of
> resources doesn't crash - parsoid runs almost using all the allocated
> memory and parsoid+trac+others hit their cpu quotas sometimes and just
> keep running.
> 
> Maybe it would be more useful to monitor client metrics, like can we
> reach the service over http?

I wouldn't expect processes to die when they hit their quota of cpu
time, network and disk I/O bandwidth. But the only possible outcome of
an OOM on a Linux system is killing the process group. The reason why is
that most processes are not prepared to handle malloc() returning NULL
and they would just crash or misbehave. Restarting everything is better
than leaving half of the processes limping around in the container in a
crippled state in which all user queries would fail anyway (no database,
no frontend http server...).

When the filesystem fills, the appropriate action depends on whether the
container contains volatile or permanent data. Frontend containers
typically have no local data and can just be restarted from a clean
image. Storage is only used to store logs, coredumps and stuff like that.


> 2. Containers should autorestart if they crash for any reason (with the
> usual measures to deal with services which crash repeatedly).
> 
> 
> That is a feature built into docker.  I've enabled it in container.yml
> (as a global default) and restarted trac with the new parameters.

Cool.

Hey, thank you very much for being always so responsive in fixing
things. Much appreciated.

> Thanks,
> Sam
>  
> 
> 
> 3. Since high-profile services have been deployed using Docker for a
> while, software for doing (1) and (2) certainly exists already :-)
> 
> 
> On 27/07/15 22:32, Samuel Cantero wrote:
> > I didn't shut it down. The trac container was already off when I
> checked
> > it.
> >
> > On Mon, Jul 27, 2015 at 9:40 PM, Sam P.  > >> wrote:
> >
> > Did you check the logs before you shut it down?
> >
> >
> > On Tue, 28 Jul 2015 9:38 am Samuel Cantero  
> > >> wrote:
> >
> > Hi Gonzalo,
> >
> > The trac container was down. I just recently turned it on.
> >
> > Somebody turned it off for some reason? If it not the case,
> > maybe the container has reached its maximum memory or
> processor
> > limit. The docker container stops in that case.
> >
> > Greetings,
> >
> >
> >
> > On Mon, Jul 27, 2015 at 5:01 PM, Gonzalo Odiard
> > mailto:godi...@sugarlabs.org>
> >> wrote:
> >
> > Today, I get a error
> >
> > 502 Bad Gateway
> > nginx/1.8.0
> >
> > when try access trac.
> >
> > Gonzalo
> >
> > On Sat, Jul 25, 2015 at 8:40 PM, Gonzalo Odiard
> > mailto:godi...@sugarlabs.org>
> >> wrote:
> >
> > Thanks Sams :)
> >
> > On Sat, Jul 25, 2015 at 3:58 PM, Samuel Cantero
> > mailto:s...@sugarlabs.org>
> >> wrote:
> >
> > Hi All,
> >
> > I just recently check this email. The good news is
> > that Sam P already dockerized this service and now
> > we have the 1.0.7 Trac version.
> >
> > Greetings,
> >
> > On Fri, Jul 24, 2015 at 11:06 AM, Gonzalo Odiard
> >  
> >  >> wrote:
> >
> > I am pretty sure dnarvaez updated the trac
> > instance a time ago.
> >
> > On Fri, Jul 24, 2015 at 12:03 PM, Bernie
> > Innocenti  
> >  >> wrote:
> >
> > On 07/24/2015 10:34 AM, Gonzalo Odiard
> wrote:
> > > When try to read
> bughttp://bugs.sugarlabs.org/ticket/4863
> 
> >   

Re: [Sugar-devel] Unit Testing Turtle Block Js

[Gonzalo Odiard]

This is a new field in the Sugar space, I am afraid we don't have previous
experiences.
Maybe you can do a research of the options to do unit tests on js,
and share in the mailing list?

Gonzalo

On Thu, Jul 23, 2015 at 3:37 PM, Amit Kumar Jha 
wrote:

> Hello,
>
> I am Amit Kumar Jha working on the project Turtle Blocks for in-line
> programming under the guidance of Walter Bender.
>
> We decided to make code more robust by including some unit tests, since
> this is my first experience in writing unit test in JS, I would really be
> glad to be guided by someone who may have experience in unit testing in
> Javascript.
>
> --
> Regards,
> Amit Kumar Jha
> (artista)
>
> ___
> Sugar-devel mailing list
> Sugar-devel@lists.sugarlabs.org
> http://lists.sugarlabs.org/listinfo/sugar-devel
>
>


-- 
Gonzalo Odiard

SugarLabs - Software [for | by] children learning
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel

Re: [Sugar-devel] [Systems] Error on trac reading a bug #4863

[Sam P.]

Hi Bernie,

On Tue, Jul 28, 2015 at 2:12 PM Bernie Innocenti  wrote:

> 1. We need monitoring for containers health with notifications when
> parameters go off thresholds. Similar to what Munin does.
>

I'm not sure how that would help us.  A container running out of resources
doesn't crash - parsoid runs almost using all the allocated memory and
parsoid+trac+others hit their cpu quotas sometimes and just keep running.

Maybe it would be more useful to monitor client metrics, like can we reach
the service over http?


>
> 2. Containers should autorestart if they crash for any reason (with the
> usual measures to deal with services which crash repeatedly).
>

That is a feature built into docker.  I've enabled it in container.yml (as
a global default) and restarted trac with the new parameters.

Thanks,
Sam


>
> 3. Since high-profile services have been deployed using Docker for a
> while, software for doing (1) and (2) certainly exists already :-)
>
>
> On 27/07/15 22:32, Samuel Cantero wrote:
> > I didn't shut it down. The trac container was already off when I checked
> > it.
> >
> > On Mon, Jul 27, 2015 at 9:40 PM, Sam P.  > > wrote:
> >
> > Did you check the logs before you shut it down?
> >
> >
> > On Tue, 28 Jul 2015 9:38 am Samuel Cantero  > > wrote:
> >
> > Hi Gonzalo,
> >
> > The trac container was down. I just recently turned it on.
> >
> > Somebody turned it off for some reason? If it not the case,
> > maybe the container has reached its maximum memory or processor
> > limit. The docker container stops in that case.
> >
> > Greetings,
> >
> >
> >
> > On Mon, Jul 27, 2015 at 5:01 PM, Gonzalo Odiard
> > mailto:godi...@sugarlabs.org>> wrote:
> >
> > Today, I get a error
> >
> > 502 Bad Gateway
> > nginx/1.8.0
> >
> > when try access trac.
> >
> > Gonzalo
> >
> > On Sat, Jul 25, 2015 at 8:40 PM, Gonzalo Odiard
> > mailto:godi...@sugarlabs.org>>
> wrote:
> >
> > Thanks Sams :)
> >
> > On Sat, Jul 25, 2015 at 3:58 PM, Samuel Cantero
> > mailto:s...@sugarlabs.org>> wrote:
> >
> > Hi All,
> >
> > I just recently check this email. The good news is
> > that Sam P already dockerized this service and now
> > we have the 1.0.7 Trac version.
> >
> > Greetings,
> >
> > On Fri, Jul 24, 2015 at 11:06 AM, Gonzalo Odiard
> >  > > wrote:
> >
> > I am pretty sure dnarvaez updated the trac
> > instance a time ago.
> >
> > On Fri, Jul 24, 2015 at 12:03 PM, Bernie
> > Innocenti  > > wrote:
> >
> > On 07/24/2015 10:34 AM, Gonzalo Odiard wrote:
> > > When try to read bughttp://
> bugs.sugarlabs.org/ticket/4863
> > > get the following error:"Genshi
> UnicodeEncodeError error while rendering
> > > template (unknown template location)"
> > >
> > > A quick google search show this bug [1]
> > >
> > > The error is originated by simply enter a
> link like this
> > > ""
> > > to a ticket. According to this [2] the bug
> is already solved on Trac 1.0.2
> > >
> > > Then:
> > > Developers: Please don't add links between
> <> until the error is solved.
> > > Systems: I don't know what version of trac
> we have running, could be
> > > possible update?
> >
> > Who's maintaining Trac these days? The wiki
> > points at dnarvaez and me,
> > but the former is MIA and the latter has
> > installed it in 2008 and not
> > touched it ever since:
> >
> > https://wiki.sugarlabs.org/go/Service/bugs
> >
> > Perhaps one of the two Sams would like to
> > upgrade Trac in-place or move
> > it into a docker container?
> >
> > --
> >   _ // Bernie Innocenti
> >   \X/ http://codewiz.org
> >
> >
> >
> >
> > --
> > Gonzalo Odiard
> >
> >