Hi Graeme,

I'm using following command to run the livetest -

rake test[example.com] TESTS="Basic*" SIGNUP_CODE=secret PROXY=10.109.190.9
ELLIS=10.109.190.10

here PROXY ip is where the bono container is launched and ELLIS ip is where
the ellis container is launched.

The bono service seems to be running in the container -

root@9837c4dab241:/# ps -eaf | grep bono
root       122     1  0 Sep19 ?        00:00:10
/usr/share/clearwater/clearwater-cluster-manager/env/bin/python
/usr/share/clearwater/bin/clearwater-cluster-manager
--mgmt-local-ip=10.0.1.2 --sig-local-ip=10.0.1.2 --local-site=site1
--remote-site= --remote-cassandra-seeds= --signaling-namespace=
--uuid=18c7daf3-a098-47ae-962f-a3d57c0cff6f --etcd-key=clearwater
--etcd-cluster-key=bono --log-level=3
--log-directory=/var/log/clearwater-cluster-manager
--pidfile=/var/run/clearwater-cluster-manager.pid
root       124     1  0 Sep19 ?        00:00:00 /bin/bash /etc/init.d/bono
run
root       139   124  0 Sep19 ?        00:00:00 /bin/bash
/usr/share/clearwater/bin/run-in-signaling-namespace start-stop-daemon
--start --quiet --exec /usr/share/clearwater/bin/bono --chuid bono --chdir
/etc/clearwater -- --domain=example.com --localhost=10.0.1.2,10.0.1.2
--alias=10.0.1.2 --pcscf=5060,5058 --webrtc-port=5062
--routing-proxy=scscf.sprout,5052,50,600 --ralf=ralf:10888 --sas=0.0.0.0,
bono@10.0.1.2 --dns-server=127.0.0.11 --worker-threads=4
--analytics=/var/log/bono --log-file=/var/log/bono --log-level=2
bono       140   139  0 Sep19 ?        00:11:15
/usr/share/clearwater/bin/bono --domain=example.com
--localhost=10.0.1.2,10.0.1.2 --alias=10.0.1.2 --pcscf=5060,5058
--webrtc-port=5062 --routing-proxy=scscf.sprout,5052,50,600
--ralf=ralf:10888 --sas=0.0.0.0,bono@10.0.1.2 --dns-server=127.0.0.11
--worker-threads=4 --analytics=/var/log/bono --log-file=/var/log/bono
--log-level=2
root       322   293  0 17:48 ?        00:00:00 grep --color=auto bono
root@9837c4dab241:/#
root@9837c4dab241:/# netstat -planut | grep 5060
tcp        0      0 10.0.1.2:5060           0.0.0.0:*               LISTEN
     -
udp        0      0 10.0.1.2:5060           0.0.0.0:*
    -
root@9837c4dab241:/#


But the connection to bono is failing from livetest container as you had
predicted.

root@40efba73deb5:~/clearwater-live-test# nc -v -z 10.109.190.9 5060
nc: connect to 10.109.190.9 port 5060 (tcp) failed: Connection refused
root@40efba73deb5:~/clearwater-live-test#


On checking the bono log, I see a series of errors like below
in beginning of the log -

19-09-2016 15:41:44.612 UTC Status utils.cpp:591: Log level set to 2
19-09-2016 15:41:44.612 UTC Status main.cpp:1388: Access logging enabled to
/var/log/bono
19-09-2016 15:41:44.613 UTC Warning main.cpp:1435: SAS server option was
invalid or not configured - SAS is disabled
19-09-2016 15:41:44.613 UTC Warning main.cpp:1511: A registration expiry
period should not be specified for P-CSCF
19-09-2016 15:41:44.613 UTC Status snmp_agent.cpp:117: AgentX agent
initialised
19-09-2016 15:41:44.613 UTC Status load_monitor.cpp:105: Constructing
LoadMonitor
19-09-2016 15:41:44.613 UTC Status load_monitor.cpp:106:    Target latency
(usecs)   : 100000
19-09-2016 15:41:44.613 UTC Status load_monitor.cpp:107:    Max bucket size
         : 1000
19-09-2016 15:41:44.613 UTC Status load_monitor.cpp:108:    Initial token
fill rate/s: 100.000000
19-09-2016 15:41:44.613 UTC Status load_monitor.cpp:109:    Min token fill
rate/s    : 10.000000
19-09-2016 15:41:44.613 UTC Status dnscachedresolver.cpp:144: Creating
Cached Resolver using servers:
19-09-2016 15:41:44.613 UTC Status dnscachedresolver.cpp:154:     127.0.0.11
19-09-2016 15:41:44.613 UTC Status sipresolver.cpp:60: Created SIP resolver
19-09-2016 15:41:44.637 UTC Status stack.cpp:419: Listening on port 5058
19-09-2016 15:41:44.637 UTC Status stack.cpp:419: Listening on port 5060
19-09-2016 15:41:44.638 UTC Status stack.cpp:855: Local host aliases:
19-09-2016 15:41:44.638 UTC Status stack.cpp:862:  10.0.1.2
19-09-2016 15:41:44.638 UTC Status stack.cpp:862:  172.18.0.2
19-09-2016 15:41:44.638 UTC Status stack.cpp:862:  10.0.1.2
19-09-2016 15:41:44.638 UTC Status stack.cpp:862:  10.0.1.2
19-09-2016 15:41:44.638 UTC Status stack.cpp:862:
19-09-2016 15:41:44.639 UTC Status httpresolver.cpp:52: Created HTTP
resolver
19-09-2016 15:41:44.641 UTC Status httpconnection.cpp:114: Configuring HTTP
Connection
19-09-2016 15:41:44.641 UTC Status httpconnection.cpp:115:   Connection
created for server ralf:10888
19-09-2016 15:41:44.641 UTC Status httpconnection.cpp:116:   Connection
will use a response timeout of 500ms
19-09-2016 15:41:44.642 UTC Status connection_pool.cpp:72: Creating
connection pool to scscf.sprout:5052
19-09-2016 15:41:44.642 UTC Status connection_pool.cpp:73:   connections =
50, recycle time = 600 +/- 120 seconds
19-09-2016 15:41:44.649 UTC Status bono.cpp:3314: Create list of PBXes
19-09-2016 15:41:44.649 UTC Status pluginloader.cpp:63: Loading plug-ins
from /usr/share/clearwater/sprout/plugins
19-09-2016 15:41:44.649 UTC Status pluginloader.cpp:158: Finished loading
plug-ins
19-09-2016 15:41:44.652 UTC Warning (Net-SNMP): Warning: Failed to connect
to the agentx master agent ([NIL]):
19-09-2016 15:41:44.653 UTC Error pjsip:  tcpc0x14f3df8 TCP connect()
error: Connection refused [code=120111]
19-09-2016 15:41:44.653 UTC Error pjsip:  tcpc0x14f5c38 TCP connect()
error: Connection refused [code=120111]


I have observed that in the bono container 5060 port is not listening in
all interfaces while 5062 port is listening in all interfaces.

root@9837c4dab241:/var/log/bono# netstat -planut | grep LISTEN
tcp        0      0 10.0.1.2:4000           0.0.0.0:*               LISTEN
     -
tcp        0      0 10.0.1.2:5058           0.0.0.0:*               LISTEN
     -
tcp        0      0 127.0.0.11:43395        0.0.0.0:*               LISTEN
     -
tcp        0      0 10.0.1.2:5060           0.0.0.0:*               LISTEN
     -
tcp        0      0 0.0.0.0:5062            0.0.0.0:*               LISTEN
     -
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN
     -
tcp        0      0 10.0.1.2:3478           0.0.0.0:*               LISTEN
     -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
     9/sshd
tcp6       0      0 :::22                   :::*                    LISTEN
     9/sshd
root@9837c4dab241:/var/log/bono#


Is it right to have 5060 port listen on only local port? Please help me to
debug the issue.

Thanks,
Sarbajit


On Wed, Sep 21, 2016 at 10:01 PM, Graeme Robertson (projectclearwater.org) <
gra...@projectclearwater.org> wrote:

> Hi Sarbajit,
>
>
>
> I’ve had another look at this, and actually I think clearwater-live-test
> checks it can connect to Bono before it tries to provision numbers from
> Ellis, and it’s actually that connection that’s failing – apologies!
>
>
>
> Can you do similar checks for the Bono container, i.e. connect to the Bono
> container and run ps -eaf | grep bono and run nc -z -v <ip> 5060 from
> your live test container (where <ip> is the IP of your Bono)?
>
>
>
> One other thought – what command are you using to run the tests? You’ll
> need to set the PROXY option to your Bono IP and the ELLIS option to your
> Ellis IP.
>
>
>
> Thanks,
>
> Graeme
>
>
>
> *From:* Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org]
> *On Behalf Of *Sarbajit Chatterjee
> *Sent:* 21 September 2016 16:41
> *To:* clearwater@lists.projectclearwater.org
> *Subject:* Re: [Project Clearwater] Deploy Clearwater in a Swarm cluster
> using docker-compose
>
>
>
> Thanks Graeme for your reply. Here are the command outputs that you had
> asked -
>
>
>
> root@e994b17b4563:/# ps -eaf | grep ellis
>
> root       177     1  0 Sep19 ?        00:00:10 /usr/share/clearwater/
> clearwater-cluster-manager/env/bin/python 
> /usr/share/clearwater/bin/clearwater-cluster-manager
> --mgmt-local-ip=10.0.1.7 --sig-local-ip=10.0.1.7 --local-site=site1
> --remote-site= --remote-cassandra-seeds= --signaling-namespace=
> --uuid=18c7daf3-a098-47ae-962f-a3d57c0cff6f --etcd-key=clearwater
> --etcd-cluster-key=ellis --log-level=3 
> --log-directory=/var/log/clearwater-cluster-manager
> --pidfile=/var/run/clearwater-cluster-manager.pid
>
> root       180     1  0 Sep19 ?        00:00:00 /bin/sh /etc/init.d/ellis
> run
>
> ellis      185   180  0 Sep19 ?        00:00:05
> /usr/share/clearwater/ellis/env/bin/python -m metaswitch.ellis.main
>
> root       287   253  0 15:18 ?        00:00:00 grep --color=auto ellis
>
> root@e994b17b4563:/#
>
> root@e994b17b4563:/# ps -eaf | grep nginx
>
> root       179     1  0 Sep19 ?        00:00:00 nginx: master process
> /usr/sbin/nginx -g daemon off;
>
> www-data   186   179  0 Sep19 ?        00:00:16 nginx: worker process
>
> www-data   187   179  0 Sep19 ?        00:00:00 nginx: worker process
>
> www-data   188   179  0 Sep19 ?        00:00:16 nginx: worker process
>
> www-data   189   179  0 Sep19 ?        00:00:16 nginx: worker process
>
> root       289   253  0 15:19 ?        00:00:00 grep --color=auto nginx
>
> root@e994b17b4563:/#
>
> root@e994b17b4563:/# netstat -planut | grep nginx
>
> tcp6       0      0 :::80                   :::*                    LISTEN
>      179/nginx -g daemon
>
> root@e994b17b4563:/#
>
>
>
>
>
> I think both ellis and nginx are running fine inside the container. I can
> also open the ellis login page from a web browser. I also checked the MySQL
> DB in ellis container. I can see livetest user entry in users table and
> 1000 rows in numbers table.
>
>
>
> I can also connect to ellis (host IP 10.109.190.10) from my livetest
> container -
>
>
>
> root@40efba73deb5:~/clearwater-live-test# nc -v -z 10.109.190.10 80
>
> Connection to 10.109.190.10 80 port [tcp/http] succeeded!
>
> root@40efba73deb5:~/clearwater-live-test#
>
>
>
>
>
> Is this happening because Clearwater containers are spread across multiple
> hosts? What other areas I should check?
>
>
>
>
>
> Thanks,
>
> Sarbajit
>
>
>
>
>
> On Wed, Sep 21, 2016 at 6:09 PM, Graeme Robertson (projectclearwater.org)
> <gra...@projectclearwater.org> wrote:
>
> Hi Sarbajit,
>
>
>
> I don’t think we’ve never tried deploying Project Clearwater in a Docker
> Swarm cluster, but I don’t see any reason why it couldn’t work. The tests
> are failing very early – they’re not able to connect to Ellis on port 80. I
> can think of a couple of reasons for this – either Ellis isn’t running or
> the Ellis port mapping hasn’t worked for some reason.
>
>
>
> Can you connect to the Ellis container and run ps –eaf | grep ellis and ps
> –eaf | grep nginx to confirm that NGINX and Ellis are running? Can you
> also run sudo netstat -planut | grep nginx or something equivalent to
> check that NGINX is listening on port 80? If there’s a problem with either
> NGINX or Ellis we probably need to look in the logs at /var/log/nginx/ or
> /var/log/ellis/ on the Ellis container.
>
>
>
> If however that all looks fine, then it sounds like the port mapping has
> failed for some reason. Can you run nc -z <ip> 80 from the box you’re
> running the live tests on? This will scan for anything listening at
> <ip>:80 and will return successfully if it finds anything.
>
>
>
> Thanks,
>
> Graeme
>
>
> ------------------------------
>
> *From:* Clearwater [mailto:clearwater-boun...@lists.projectclearwater.org
> <clearwater-boun...@lists.projectclearwater.org>] *On Behalf Of *Sarbajit
> Chatterjee
> *Sent:* 20 September 2016 15:05
> *To:* clearwater@lists.projectclearwater.org
> *Subject:* [Project Clearwater] Deploy Clearwater in a Swarm cluster
> using docker-compose
>
>
>
> Hello,
>
>
>
> I am following the instructions from https://github.com/
> Metaswitch/clearwater-docker. I can successfully deploy it on a single
> Docker node but, the compose file does not work with Swarm cluster.
>
>
>
> I did try to modify the compose file like this -
>
>
>
>
>
> version: '2'
>
> services:
>
>   etcd:
>
>     image: quay.io/coreos/etcd:v2.2.5
>
>     command: >
>
>       -name etcd0
>
>       -advertise-client-urls http://etcd:2379,http://etcd:4001
>
>       -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001
>
>       -initial-advertise-peer-urls http://etcd:2380
>
>       -listen-peer-urls http://0.0.0.0:2380
>
>       -initial-cluster etcd0=http://etcd:2380
>
>       -initial-cluster-state new
>
>   bono:
>
>     image: swarm-node:5000/clearwaterdocker_bono
>
>     ports:
>
>       - 22
>
>       - "3478:3478"
>
>       - "3478:3478/udp"
>
>       - "5060:5060"
>
>       - "5060:5060/udp"
>
>       - "5062:5062"
>
>   sprout:
>
>     image: swarm-node:5000/clearwaterdocker_sprout
>
>     networks:
>
>       default:
>
>         aliases:
>
>           - scscf.sprout
>
>           - icscf.sprout
>
>     ports:
>
>       - 22
>
>   homestead:
>
>     image: swarm-node:5000/clearwaterdocker_homestead
>
>     ports:
>
>       - 22
>
>   homer:
>
>     image: swarm-node:5000/clearwaterdocker_homer
>
>     ports:
>
>       - 22
>
>   ralf:
>
>     image: swarm-node:5000/clearwaterdocker_ralf
>
>     ports:
>
>       - 22
>
>   ellis:
>
>     image: swarm-node:5000/clearwaterdocker_ellis
>
>     ports:
>
>       - 22
>
>       - "80:80"
>
>
>
>
>
> where swarm-node:5000 is the local docker registry and it hosts the
> pre-built images of Clearwater containers. Even though the deployment
> succeeded, clearwater-livetests are failing with following error -
>
>
>
>
>
> Basic Registration (TCP) - Failed
>
>   Errno::ECONNREFUSED thrown:
>
>    - Connection refused - connect(2)
>
>      - 
> /usr/local/rvm/gems/ruby-1.9.3-p551/gems/quaff-0.7.3/lib/sources.rb:41:in
> `initialize'
>
>
>
>
>
> Any suggestions on how I can deploy Clearwater on a Swarm cluster?
>
>
>
> Thanks,
>
> Sarbajit
>
>
>
>
> _______________________________________________
> Clearwater mailing list
> Clearwater@lists.projectclearwater.org
> http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.
> projectclearwater.org
>
>
>
> _______________________________________________
> Clearwater mailing list
> Clearwater@lists.projectclearwater.org
> http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.
> projectclearwater.org
>
>
_______________________________________________
Clearwater mailing list
Clearwater@lists.projectclearwater.org
http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org

Reply via email to