Re: auto start not stable

[Qihao Bao]

Hi Lei,

Yes, you are right. I reproduced the issue that JPL-backend failed to auto
start.

Good news is that i found another way to run auto-start command lines for
JPL-Backend, which is use root user to run scripts as another user, instead
of switching to another user account.
I assume that is more stable for summer school.

I also update the docker images.

If you have time, please give a try. Restart the VM and see if it works.


Thanks.



Br.
Qihao

On Wed, Jul 20, 2016 at 1:17 AM, Pan, Lei (398K) <lei....@jpl.nasa.gov>
wrote:

> Hi Qihao:
>
> Here is the content of
> /etc/init.d/launch-cmac-docker.sh
> So the line does have “/bin/bash”
> at the end. Please give it a try! Thanks, -Lei
>
>
> !/bin/bash
>
> export HOME="/home/ubuntu"
>
> cd $HOME
>
> . $HOME/.bash_profile
>
> ### sudo service docker start
>
> ### /bin/su - ubuntu
>
> echo `whoami` > $HOME/mytest.log
>
> ### ls > $HOME/mytest.log
>
> exec /usr/bin/docker run -d -t -p 9033:9033 -p 9034:9034 -p 8122:22 -p
> 8180:8880 -p 8090:8890 -p 8023:8023 -v /mnt/xvdf/data_2015:/mnt/data_2015
> -i cmusvsc/jpl-summerschool2016:v1.0 /bin/bash
>
> echo "from docker run 2" >> $HOME/mytest.log
>
>
>
>
> From: Qihao Bao <qihao...@cmu.edu<mailto:qihao...@cmu.edu>>
> Date: Tuesday, July 19, 2016 at 4:41 PM
> To: JPL <lei....@jpl.nasa.gov<mailto:lei....@jpl.nasa.gov>>
> Cc: Jia Zhang <jia.zh...@sv.cmu.edu<mailto:jia.zh...@sv.cmu.edu>>,
> Seungwon Lee <seungwon....@jpl.nasa.gov<mailto:seungwon....@jpl.nasa.gov>>,
> Benyang Tang <benyang.t...@jpl.nasa.gov<mailto:benyang.t...@jpl.nasa.gov>>,
> "Zhai, Chengxing (398K)" <chengxing.z...@jpl.nasa.gov<mailto:
> chengxing.z...@jpl.nasa.gov>>, "dev@cmda.incubator.apache.org<mailto:
> dev@cmda.incubator.apache.org>" <dev@cmda.incubator.apache.org<mailto:
> dev@cmda.incubator.apache.org>>
> Subject: Re: auto start not stable
>
> Hi Lei,
>
> What's the command line that you used to start docker image inside of a VM?
> I guess that's the issue.
>
> I remembered i used something similar to
> ++++++++++++++++++++++++++++++++++++++++++
> docker run -d -t -p 9033:9033 -p 9034:9034 -p 8122:22 -p 8180:8880 -p
> 8090:8890 -p 8023:8023 -v /mnt/xvdf/data_2015:/mnt/data_2015  -i
> cmusvsc/jpl-summerschool2016:v1.0 /bin/bash
> ++++++++++++++++++++++++++++++++++++++++++
>
> The /bin/bash is needed to start a bash in the docker, which is necessary
> for auto start...
>
> Qihao
>
> On Tue, Jul 19, 2016 at 4:28 PM, Pan, Lei (398K) <lei....@jpl.nasa.gov
> <mailto:lei....@jpl.nasa.gov>> wrote:
> Hi Qihao:
>
> I have burned an EC2 image and am now testing
> the auto start of our services. I found it to be unstable.
>
> There are two steps in auto start: (1) the start of the
> Docker container when the EC2 instance is launched;
> and (2) the start of the CMU and JPL services, especially
> the CMDA backend, when the Docker container is up.
>
> The (1) step is still being tested, but at least every time
> I reboot the EC2 instance, I get Docker player, though
> sometimes it takes a long time.
>
> The (2) is not stable. What I found is that when I manually
> run the Docker instance, I do not always see the CMDA
> backend started. Sometimes if I log in as svc to the
> container, I may trigger the gunicorn process to be started,
> other times no matter how many times I log in, I do not
> see gunicorn proceses.
>
> You can debug the issue using the following steps:
> (1) Get on to the EC2 instance, from which I burned the image:
> ssh -i climate-default.pem
> ubu...@ec2-52-53-198-227.us-west-1.compute.amazonaws.com<mailto:
> ubu...@ec2-52-53-198-227.us-west-1.compute.amazonaws.com>
> (2) stop the running container if any, by
> docker ps
> and
> docker stop <container ID>
> (3) run docker using the line found in
>
> /etc/init.d/launch-cmac-docker.sh
>
> which is the script used to automatically launch docker
>
> (4) Point your browser to:
>
> <http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/><
> http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/><
> http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/>
>
> http://ec2-52-53-198-227.us-west-1.compute.amazonaws.com:9033/
>
> Go to any cmda service, such as twoDimMap, and try to run it.
>
> (5) If you see “error!”, you can try to log in to the Docker container
>
> ssh -p 8122 s...@ec2-52-53-198-227.us-west-1.compute.amazonaws.com<mailto:
> s...@ec2-52-53-198-227.us-west-1.compute.amazonaws.com>
>
> and type:
>
> ps -aelf|grep python
>
> to see if the gunicorn processes are there.
>
> If they are there, then you can go back to your browser
>
> to request cmda services again.
>
>
> What I have seen is that sometimes after logging in, I
>
> can see the gunicorn processes and the cmda services
>
> are then up, but other times even if I log in many times,
>
> I do not see the gunicorn processes. This is the instability
>
> I mean. This puzzles me because you put in the line “sh start.sh”
>
> in /home/svc/.bashrc, and it should be called every time I log
>
> in as svc, correct?
>
>
> Before asking you to check on this, I actually tried to use a different
>
> way — upstart — to launch the cmda services. But unlike in the full
>
> linux env, upstart does not seem to work in Docker. The service script
>
> that I put under /etc/init/ did not work.
>
>
> Thanks,
>
> -Lei
>
>
>
>
>
>