Yep, you’re right that’s the root cause. But after trying several times, it still can’t load any plugins. Do you have any way to load these plugins manually ?
I wrote an C language file ‘test’, and ran it as ‘sprout’ user. Seems it works. root@sprout-1-1pp1p:~# su sprout -c '/var/log/sprout/test /usr/share/clearwater/sprout/plugins' Open Directory /usr/share/clearwater/sprout/plugins Error：Success . .. sprout_bgcf.so sprout_icscf.so sprout_mmtel_as.so sprout_scscf.so Open Directory /usr/share/clearwater/sprout/plugins Error：Success Open Directory /usr/share/clearwater/sprout/plugins Error：Success root@sprout-1-1pp1p:~# Read source code, the logic is very simple in https://github.com/Metaswitch/sprout/blob/dev/src/pluginloader.cpp#L68 . What may cause this issue ? Selinux?? or Filesystem? >From this page http://linux.die.net/man/3/readdir : “Currently, only some file systems (among them: Btrfs, ext2, ext3, and ext4) have full support for returning the file type in d_type” Currently rhel7 is using xfs as it’s default file system. And the container is using it !!! root@sprout-1-1pp1p:~# df -T -h Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/docker-253:0-67648440-e939ea7b2b39ce24a000b0e068e6feba0626504abe 909818797e9f48ce7bdd91 xfs 100G 370M 100G 1% / tmpfs tmpfs 16G 0 16G 0% /dev shm tmpfs 64M 0 64M 0% /dev/shm tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup tmpfs tmpfs 16G 40K 16G 1% /run/secrets /dev/mapper/rhel_mac002590f16385-root xfs 45G 29G 17G 64% /etc/hosts tmpfs tmpfs 16G 8.0K 16G 1% /run/secrets/kubernetes.io/serviceaccount tmpfs tmpfs 16G 0 16G 0% /proc/kcore tmpfs tmpfs 16G 0 16G 0% /proc/timer_stats root@sprout-1-1pp1p:~# Well, do we have some simple solution for this ? Great thanks J Jia Xuan China Mobile Research Institute 发件人: jiaxuan [mailto:[email protected]] 发送时间: 2016年3月1日 10:07 收件人: 'Matt Williams (projectclearwater.org)' 抄送: '[email protected]'; 'Chris Elford' 主题: 答复: [Project Clearwater] Test Clearwater on Openshift Matt: root@sprout-1-u9v45:~# cd /usr/share/clearwater/sprout root@sprout-1-u9v45:/usr/share/clearwater/sprout# ls -l total 4 drwxr-xr-x. 2 root root 4096 Dec 11 09:58 lib drwxr-xr-x. 2 root root 96 Mar 1 02:00 plugins root@sprout-1-u9v45:/usr/share/clearwater/sprout# cd plugins/ root@sprout-1-u9v45:/usr/share/clearwater/sprout/plugins# ls -l total 608 -rw-r--r--. 1 root root 171456 Dec 7 16:00 sprout_bgcf.so -rw-r--r--. 1 root root 212808 Dec 7 16:00 sprout_icscf.so -rw-r--r--. 1 root root 113960 Dec 7 16:00 sprout_mmtel_as.so -rw-r--r--. 1 root root 122504 Dec 7 16:00 sprout_scscf.so root@sprout-1-u9v45:/usr/share/clearwater/sprout/plugins# 发件人: Matt Williams (projectclearwater.org) [mailto:[email protected]] 发送时间: 2016年3月1日 1:31 收件人: jiaxuan 抄送: [email protected]; Chris Elford 主题: RE: [Project Clearwater] Test Clearwater on Openshift Jia, Thanks for these diagnostics. To me, the interesting bit is (in sprout_*.txt): 29-02-2016 11:20:03.678 UTC Status pluginloader.cpp:63: Loading plug-ins from /usr/share/clearwater/sprout/plugins 29-02-2016 11:20:03.679 UTC Status pluginloader.cpp:148: Finished loading plug-ins Sprout is architected as a base executable with plug-ins (often referred to as "Sproutlets") on top of it to implement function. The Sproutlets are loaded as shared objects (i.e. .so files) from the /usr/share/clearwater/sprout/plugins directory. The log above indicates that it doesn't find any plugins. Plugins are contained in separate Debian packages, such as sprout-icscf, sprout-scscf and sprout-bgcf (for each of the functions that Sprout provides). I've checked the cw_package_info.txt file that you provided, and this shows that these packages are installed, so the shared objects should be present. So I think the key question is whether these shared objects are present. On the Sprout node, please can you - run "ls -l /usr/share/clearwater/sprout/plugins" to list the contents of this directory - run "ls -l /usr/share/clearwater/sprout" to check that the permissions on the containing directory? Please let me know. Thanks, Matt -- Matt Williams Lead Architect, Project Clearwater +44 (0) 20 8366 1177 From: jiaxuan [mailto:[email protected]] Sent: 29 February 2016 12:06 To: Matt Williams <[email protected]> Cc: [email protected]; Chris Elford <[email protected]> Subject: 答复: [Project Clearwater] Test Clearwater on Openshift Hi Matt, Chris, Firstly , thanks for answering my questions. That makes me more confident to move docker-compose into Openshift platform. Well, Openshift is using RHEL7 as its host OS. But the container is still using ubuntu system, as far as I know the base image is Ubuntu14.04. I change nothing in Dockerfile. I am sorry I can’t find the log about ‘docker build’ and ‘docker compose’. But all the call succeeds. So I copy these images into Openshift Platform, and set some environment values, make all the models can be connected by requirements(https://raw.githubusercontent.com/Metaswitch/clearwater-docker/ master/minimal-distributed.yaml). For openshift platform: 1. All the subscribers are from Ellis web UI. From the client(x-lite or zoiper), I can see all the subscribers are registered. 2. The request ‘INVITE’ has been sent from bono to sprout. After at least 10 seconds, bono reply 408 to UE. I attach the log file which you wanted, please check . Great thanks. Jia Xuan 发件人: Matt Williams [mailto:[email protected]] 发送时间: 2016年2月26日 20:42 收件人: jiaxuan 抄送: [email protected]; Chris Elford 主题: RE: [Project Clearwater] Test Clearwater on Openshift Jia, I think I'm a bit more positive that we can make this work than Chris is - my understanding is that OpenShift is a Platform as a Service (Paas) built on Docker, and we've successfully run Clearwater under Docker before, so unless OpenShift disables something we rely on, Clearwater should run on OpenShift. His concern was that you were trying to port Clearwater to run on Red Hat Enterprise Linux (RHEL), rather than under Ubuntu. However, I think it should be possible to run Clearwater in its native Ubuntu, even if the host OS is RHEL. Please can you confirm which guest OS your containers are running? ...and have you made any changes to the clearwater-docker scripts? When Chris asked about "subscribers", he was referring to the telephone numbers you're calling between. How did you pick the numbers 6505550219 and 6505550782, for example? Did you create them through the Ellis web UI? Are these the only 2 subscribers you've created? The diagnostics we've got so far haven't been conclusive, but there are three things that would be useful. ・ The complete logs from the output from the "docker build" and "docker compose" commands, showing all the packages being installed to build the images and then the images being deployed, just in case there's something odd going on there. ・ On the Sprout node, stop the sprout process (with "sudo service sprout stop"), wait 30s so that it has definitely restarted and then attempt a call, and send us the logs from that - it's possible that something is going wrong on start-up. ・ Clearwater has a built-in tool to gather the key diagnostics together. On the Sprout node, run "/usr/share/clearwater/bin/gather_diags" and then gather the diagnostic dump from /var/clearwater-diags-monitor/dumps/*.tar.gz and send it to us. Please let us know. Thanks, Matt -- Matt Williams Lead Architect, Project Clearwater +44 (0) 20 8366 1177 From: Clearwater [mailto:[email protected]] On Behalf Of Chris Elford Sent: 26 February 2016 11:59 To: jiaxuan <[email protected]> Cc: [email protected] Subject: Re: [Project Clearwater] Test Clearwater on Openshift I notice below that you say you are using Openshift and Kubernetes. As far as I know, nobody is currently running Project Clearwater using these platforms, and I would not expect it to work without some development work to port it to this new platform. I think that we need to understand what you did to allow you to install Project Clearwater on this new platform before we can provide any help. Yours, Chris From: jiaxuan [mailto:[email protected]] Sent: 26 February 2016 09:02 To: Chris Elford <[email protected]> Cc: [email protected] Subject: 答复: [Project Clearwater] Test Clearwater on Openshift Sorry， I have no idea about what’s ‘subscriber’. It’s almost same with the environment which is setup by docker-compose (https://github.com/Metaswitch/clearwater-docker). But all call fail. I found a PUBLISH message : The content-Length is zero. PUBLISH sip:[email protected];transport=TCP SIP/2.0^M Via: SIP/2.0/TCP 175.128.0.113:55316;branch=z9hG4bK-524287-1---4b2b34f926a6c876;rport^M Max-Forwards: 70^M Route: <sip:10.1.0.217:5054;transport=TCP;lr;orig>^M Contact: <sip:[email protected]:55316;transport=TCP>^M To: "2323"<sip:[email protected];transport=TCP>^M From: "2323"<sip:[email protected];transport=TCP>;tag=ea19e074^M Call-ID: ZuNcniFDAQplqlNX3qyFJQ..^M CSeq: 319 PUBLISH^M Expires: 600^M Allow: INVITE, ACK, CANCEL, BYE, NOTIFY, REFER, MESSAGE, OPTIONS, INFO, SUBSCRIBE^M Supported: replaces, norefersub, extended-refer, timer, outbound, path, X-cisco-serviceuri^M User-Agent: Z 3.9.32144 r32121^M Event: presence^M Allow-Events: presence, kpml^M Content-Length: 0^M ^M While checking the docker-compose env, I got Max-Forwards: 70^M Route: <sip:172.17.0.5:5054;transport=TCP;lr;orig>^M Contact: <sip:[email protected]:51158;transport=TCP>^M To: "123"<sip:[email protected];transport=TCP>^M From: "123"<sip:[email protected];transport=TCP>;tag=e72f4e32^M Call-ID: vqETasJUaN4hD1h5scKyUA..^M CSeq: 1 PUBLISH^M Expires: 600^M Allow: INVITE, ACK, CANCEL, BYE, NOTIFY, REFER, MESSAGE, OPTIONS, INFO, SUBSCRIBE^M Content-Type: application/pidf+xml^M Supported: replaces, norefersub, extended-refer, timer, outbound, path, X-cisco-serviceuri^M User-Agent: Z 3.9.32144 r32121^M Event: presence^M Allow-Events: presence, kpml^M Content-Length: 271^M ^M <?xml version="1.0" encoding="UTF-8"?>^M <presence xmlns="urn:ietf:params:xml:ns:pidf"^M entity="sip:[email protected];transport=TCP">^M <tuple id="6505550782" >^M <status><basic>open</basic></status>^M <note>Online</note>^M </tuple>^M </presence>^M Is that helpful? Thanks. Jia Xuan 发件人: Chris Elford [mailto:[email protected]] 发送时间: 2016年2月26日 0:27 收件人: jiaxuan 抄送: [email protected] 主题: RE: [Project Clearwater] Test Clearwater on Openshift Hi, I’ve taken a look at your Sprout logs. In order to diagnose your problem, it would be useful to have some more information about what you are seeing. ・ What are the subscribers are you trying to call between? ・ Are you seeing all calls fail or just occasional ones? We’ve looked into the two issues that you mentioned below. It looks like Ralf is configured to send billing data to a CDF at example.com. The logs in your message show that it is failing to do so. You can configure a real CDF by changing the value of ‘cdf_identity’ in the deployment’s shared config. See http://clearwater.readthedocs.org/en/latest/Clearwater_Configuration_Options _Reference/index.html for details. We have looked into the tombstone issue that you mention below. We do not think that this will cause any problems. We have raised https://github.com/Metaswitch/sprout/issues/1334 to track working around it and cleaning up those entries. Yours, Chris From: Clearwater [mailto:[email protected]] On Behalf Of jiaxuan Sent: 25 February 2016 08:44 To: [email protected] Subject: [Project Clearwater] Test Clearwater on Openshift Hi list: I have installed clearwater, following “Manual Install Instrutions”. The difference is that it’s running on Openshift which is using Kurbernetes. Currently the Ellis model is running well, as I can make my X-Lite client registered in. The problem is that: The call function doesn’t work properly. I list what I have found: 1. Ralf: from /var/log/ralf/ralf_2016XXX 24-02-2016 13:00:34.447 UTC Verbose diameterstack.cpp:1413: Sending Diameter message of type 271 on transaction 0x7f17d0005100 24-02-2016 13:00:34.447 UTC Debug diameterstack.cpp:397: Routing out callback from freeDiameter 24-02-2016 13:00:34.447 UTC Error diameterstack.cpp:293: Routing error: 'No remaining suitable candidate to route the message to' for message with Command-Code 271, Destination-Host and Destination-Realm example.com 24-02-2016 13:00:34.447 UTC Debug freeDiameter: Iterating on rules of COMMAND: '(generic error format)'. 24-02-2016 13:00:34.447 UTC Debug freeDiameter: Calling callback registered when query was sent (0x437910, 0x7f17d0005100) 24-02-2016 13:00:34.447 UTC Verbose diameterstack.cpp:1093: Got Diameter response of type 271 - calling callback on transaction 0x7f17d0005100 24-02-2016 13:00:34.447 UTC Warning peer_message_sender.cpp:125: Failed to send ACR to (number 0) 24-02-2016 13:00:34.447 UTC Error peer_message_sender.cpp:145: Failed to connect to all CCFs, message not sent 24-02-2016 13:00:34.447 UTC Warning session_manager.cpp:319: Session for 79048MzlhMGNjMzc4YmEyY2NlMGEyZmU2ODNmYjZjM2E1ZGY received error from CDF 24-02-2016 13:00:34.470 UTC Verbose httpstack.cpp:286: Process request for URL /call-id/79048MzlhMGNjMzc4YmEyY2NlMGEyZmU2ODNmYjZjM2E1ZGY, args (null) 2. Sprout:from /var/log/sprout/sprout_current.txt . We can see there’re several ‘tombstone’ element in this json. 25-02-2016 08:05:00.076 UTC Debug avstore.cpp:72: Set AV for [email protected]\7b66730d304dcdcb {"digest":{"ha1":"3fb2ca6bb49c77b0907e080079d325a0","realm":"example.com","q op":"auth"},"branch":"z9hG4bKPjbALlVrXogjQKxNGPi1wmdGhDKXTFBr7h","tombstone" :true,"tombstone":true,"tombstone":true} I attached all the log files in this email. We appreciate your response. Thanks. Jia Xuan China Mobile Research Institute
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
