This means that your xCAT management node is not syncing the status
updates to the service nodes. What happens if you do a "nodeset node001
install" on the management node? Do both service nodes report back to
you that they have done the command correctly?
You should not be needing to manually sync anything from the management
node to the service nodes, this is done on deployment. If you are having
to do this, something is broken and will continue to stay broken unless
you correct the root cause. I would recommend redeploying both of your
service nodes (nodeset service install; rpower service reset) and seeing
if after a clean redeployment this corrects the issue.
Could it be a matter of DNS is supposed to be configured on the
SNs (I have servicenode.nameserver=1) and it's not configured despite the
fact that the site.nameservers shows the IP for the xCAT MN (I edited that
out of my earlier email).
Service node's DNS are deployed as caching nodes, so their DNS cache
will only contain nodes they have previously deployed. If they do not
have the DNS info they need when it is requested, they look to the
management node for this information and then cache it so that they
don't have to request it again (I believe it is networks.nameservers
that sets the IP they look to. Someone can correct me if this is wrong.)
[Thu Jul 21 13:40:58 2011] conserver (19859): ERROR: no consoles found in
configuration file
Are you using console servers? Is servicenode.conserver set to "1"?
Should site.nameservers have the SN IPs in there as well for DNS to work?
I want to say no, but I am not entirely sure what this setting actually
changes. I've put a bad IP in it and makehosts/makedns/makedhcp -n'd it and it
doesn't seem to be actually changing anything. Perhaps someone else can chime
in here on what that touches, as all of my compute node's /etc/resolv.conf
contains the entries from networks.nameservers, as does /etc/named.conf's
forwarders section on my service nodes.
What about site.dhcpinterfaces? Will adding the SNs in there cause makedhcp to
build working DHCP configs on the SNs?
site.dhcpinterfaces defines what interfaces are facing the cluster network and
DHCP should respond to. This is available in the man page "man site".
dhcpinterfaces: The network interfaces DHCP should listen on. If it is the
same
for all nodes, use simple comma-separated
list of NICs. To
specify different NICs for different nodes:
mnâeth1,eth2;serviceâbond0.
An example of my
dhcpinterfaces:"dhcpinterfaces","mn|eth1;service01|eth0;service02|eth0",,
On 7/21/2011 4:31 PM, Christian Caruthers wrote:
> A little more information re: the last email. I looked
> in /tftpboot/pxelinux.cfg/node001 and found that the xCAT MN still points
> to the kickstart install while both SNs have been updated to boot the node
> from HDD.
>
> Christian D. Caruthers
> Linux HPC Consultant
> STG Lab Services
> 757-656-9675
>
>
>
> From: Christian Caruthers/Richmond/IBM@IBMUS
>
> To: xCAT Users Mailing list<xcat-user@lists.sourceforge.net>
>
> Date: 07/21/2011 04:53 PM
>
> Subject: Re: [xcat-user] service nodes node working correctly
>
>
>
>
>
>
> Still more information. I rsync'ed the /etc/xcat/hostkeys and /etc/xcat/ca
> directories from the MN to the SNs and retried an install. After
> installation, while the node is rebooting, I did a nodeset node001 stat and
> saw something weird:
>
> nodeset node001 stat
> node001: install rhels6.1-x86_64-gpu
> node001: boot
> node001: boot
>
> So then I looked at the chain table...
>
> [root@mgt gpfs]# nodels node001 chain
> node001: chain.ondiscover: nodediscover
> node001: chain.chain: runcmd=bmcsetup,standby
> node001: chain.node: node001
> node001: chain.currstate: boot
> node001: chain.currchain: boot
> node001: chain.comments:
> node001: chain.disable:
> [root@mgt gpfs]# psh service nodels node001 chain
> sn02: node001: chain.ondiscover: nodediscover
> sn02: node001: chain.chain: runcmd=bmcsetup,standby
> sn02: node001: chain.node: node001
> sn02: node001: chain.currstate: boot
> sn02: node001: chain.currchain: boot
> sn02: node001: chain.comments:
> sn02: node001: chain.disable:
> sn01: node001: chain.ondiscover: nodediscover
> sn01: node001: chain.chain: runcmd=bmcsetup,standby
> sn01: node001: chain.node: node001sn01: node001: chain.currstate: boot
> sn01: node001: chain.currchain: boot
> sn01: node001: chain.comments:
> sn01: node001: chain.disable:
>
> Thoughts?
>
> Christian D. Caruthers
> Linux HPC Consultant
> STG Lab Services
> 757-656-9675
>
>
>
> From: Christian Caruthers/Richmond/IBM@IBMUS
>
>
> To: xCAT Users Mailing list<xcat-user@lists.sourceforge.net>
>
>
> Date: 07/21/2011 02:46 PM
>
>
> Subject: Re: [xcat-user] service nodes node working correctly
>
>
>
>
>
>
>
> Some more information: On the service nodes, when I run service xcatd
> restart, I see the following errors:
>
> service xcatd restart
> Restarting xCATd Starting conserver: [Thu Jul 21 13:40:58 2011] conserver
> (19859): conserver.com version 8.1.16
> [Thu Jul 21 13:40:58 2011] conserver (19859): started as `root' by `root'
> [Thu Jul 21 13:40:58 2011] conserver (19859): ERROR: no consoles found in
> configuration file
> [Thu Jul 21 13:40:58 2011] conserver (19859): terminated
> [FAILED]
> Error on command: service conserver start
>
> Shutting down vsftpd: [ OK ]
> Starting vsftpd for vsftpd: [ OK ]
> Generating public/private rsa1 key pair.
> Your identification has been saved
> in /tmp/mknb.19819.N2Jezn/etc/ssh_host_key.
> Your public key has been saved
> in /tmp/mknb.19819.N2Jezn/etc/ssh_host_key.pub.
> The key fingerprint is:
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> Generating public/private rsa key pair.
> Your identification has been saved
> in /tmp/mknb.19819.N2Jezn/etc/ssh_host_rsa_key.
> Your public key has been saved
> in /tmp/mknb.19819.N2Jezn/etc/ssh_host_rsa_key.pub.
> The key fingerprint is:
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> Generating public/private dsa key pair.
> Your identification has been saved
> in /tmp/mknb.19819.N2Jezn/etc/ssh_host_dsa_key.
> Your public key has been saved
> in /tmp/mknb.19819.N2Jezn/etc/ssh_host_dsa_key.pub.
> The key fingerprint is:
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> Creating nbfs.ppc64.gz in /tftpboot/xcat
> 34862 blocks
> Use of uninitialized value in scalar assignment at /usr/sbin/xcatd line
> 1093.
> Generating public/private rsa1 key pair.
> Your identification has been saved
> in /tmp/mknb.19819.C2EvPW/etc/ssh_host_key.
> Your public key has been saved
> in /tmp/mknb.19819.C2EvPW/etc/ssh_host_key.pub.
> The key fingerprint is:
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> Generating public/private rsa key pair.
> Your identification has been saved
> in /tmp/mknb.19819.C2EvPW/etc/ssh_host_rsa_key.
> Your public key has been saved
> in /tmp/mknb.19819.C2EvPW/etc/ssh_host_rsa_key.pub.
> The key fingerprint is:
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> Generating public/private dsa key pair.
> Your identification has been saved
> in /tmp/mknb.19819.C2EvPW/etc/ssh_host_dsa_key.
> Your public key has been saved
> in /tmp/mknb.19819.C2EvPW/etc/ssh_host_dsa_key.pub.
> The key fingerprint is:
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> Creating nbfs.x86.gz in /tftpboot/xcat
> 26690 blocks
> Generating public/private rsa1 key pair.
> Your identification has been saved
> in /tmp/mknb.19819.7Oef2g/etc/ssh_host_key.
> Your public key has been saved
> in /tmp/mknb.19819.7Oef2g/etc/ssh_host_key.pub.
> The key fingerprint is:
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> Generating public/private rsa key pair.
> Your identification has been saved
> in /tmp/mknb.19819.7Oef2g/etc/ssh_host_rsa_key.
> Your public key has been saved
> in /tmp/mknb.19819.7Oef2g/etc/ssh_host_rsa_key.pub.
> The key fingerprint is:
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> Generating public/private dsa key pair.
> Your identification has been saved
> in /tmp/mknb.19819.7Oef2g/etc/ssh_host_dsa_key.
> Your public key has been saved
> in /tmp/mknb.19819.7Oef2g/etc/ssh_host_dsa_key.pub.
> The key fingerprint is:
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> Creating nbfs.x86_64.gz in /tftpboot/xcat
> 40007 blocks
> Stopping ATFTP [ OK ]
> Starting ATFTP [ OK ]
> Stopping named: [ OK ]
> Starting named: [ OK ]
> Shutting down NFS mountd: [ OK ]
> Shutting down NFS daemon: [ OK ]
> Shutting down NFS quotas: [ OK ]
> Shutting down NFS services: [FAILED]
> Starting NFS services: [ OK ]
> Starting NFS quotas: [ OK ]
> Starting NFS daemon: [ OK ]
> Starting NFS mountd: [ OK ]
> Stopping httpd: [ OK ]
> Starting httpd: httpd: apr_sockaddr_info_get() failed for sn01.cluster.net
> httpd: Could not reliably determine the server's fully qualified domain
> name, using 127.0.0.1 for ServerName
> [ OK ]
> Adding NIC eth0
> Warning: No dynamic range specified for 172.30.0.0. If hardware discovery
> is being used, a dynamic range is required.
> Warning: No dynamic range specified for 172.29.0.0. If hardware discovery
> is being used, a dynamic range is required.
> Shutting down dhcpd: [ OK ]
> Starting dhcpd: [ OK ]
> Shutting down dhcpd: [ OK ]
> Starting dhcpd: [ OK ]
> [ OK ]
>
> Christian D. Caruthers
> Linux HPC Consultant
> STG Lab Services
> 757-656-9675
>
>
>
> From: Christian Caruthers/Richmond/IBM@IBMUS
>
>
> To: xCAT Users Mailing list<xcat-user@lists.sourceforge.net>
>
>
> Date: 07/21/2011 02:07 PM
>
>
> Subject: Re: [xcat-user] service nodes node working correctly
>
>
>
>
>
>
>
> Thanks for the help! This is the last thing I have to get working to
> complete this engagement!
>
> Ran xdsh service -K but no hostkeys were copied. The /etc/xcat/hostkeys
> directory exists on the MN, and the noderes.xcatmaster and
> noderes.servicenode are the IP of the MN. I verified the root password on
> both nodes by logging in through the console. The xdsh command came back
> with:
>
> /usr/bin/ssh setup is complete
> return code = 0
>
> Running xdsh service date and xdsh service tabdump site come back with the
> correct results.
>
> sn01: rpm -qa | grep xCAT
> xCAT-nbroot-oss-x86-2.0-snap200804021050
> xCAT-nbroot-oss-ppc64-2.0-snap200801291320
> perl-xCAT-2.5.2-snap201103041120
> xCAT-nbkernel-x86-2.6.18_164-8
> xCAT-nbroot-oss-x86_64-2.0-snap200801291344
> xCAT-nbroot-core-x86_64-2.5.1-snap201011121008
> xCAT-nbroot-core-x86-2.5.1-snap201011121008
> xCAT-nbroot-core-ppc64-2.5.1-snap201011121008
> xCAT-client-2.5.2-snap201102251840
> xCAT-nbkernel-x86_64-2.6.18_164-8
> xCAT-nbkernel-ppc64-2.6.18_92-4
> xCATsn-2.5.1-snap201011101325
> xCAT-server-2.5.2-snap201103041121
> xCATsn-2.5.1-snap201011101325
>
> sn02: rpm -qa | grep xCAT
> xCAT-nbroot-oss-x86-2.0-snap200804021050
> xCAT-nbroot-oss-ppc64-2.0-snap200801291320
> perl-xCAT-2.5.2-snap201103041120
> xCAT-nbkernel-x86-2.6.18_164-8
> xCAT-nbroot-oss-x86_64-2.0-snap200801291344
> xCAT-nbroot-core-x86_64-2.5.1-snap201011121008
> xCAT-nbroot-core-x86-2.5.1-snap201011121008
> xCAT-nbroot-core-ppc64-2.5.1-snap201011121008
> xCAT-client-2.5.2-snap201102251840
> xCAT-nbkernel-x86_64-2.6.18_164-8
> xCAT-nbkernel-ppc64-2.6.18_92-4
> xCATsn-2.5.1-snap201011101325
> xCAT-server-2.5.2-snap201103041121
> xCATsn-2.5.1-snap201011101325
>
> As for the DHCP, I gathered from an earlier email from Linda Mellor on the
> recent thread on SNs:
>
> "And the network possibilities with all of this can start to be
> mind-boggling, but we try to address the most common ones as best we can:
> - the entire cluster on one flat network. This means there will be
> multiple DHCP servers (and tftp servers), and xCAT needs to configure
> any DHCP server to respond correctly to a broadcast request on the
> network, so all dhcpd.leases files will need to be identical, with
> the "next-server" value set to the designated tftpserver for a given
> node."
>
> When I do nodeset node001 install, the tftpboot/pxelinux.cfg/node001 file
> shows sn02 as the host it's pulling its files from. Also
> in /install/autoinst on the MN& both SNs, the kickstart file shows
> url=http://sn02/... and all that appears to work. When I manually nodeset
> node001 boot and it boots up to the OS, I can see that DNS is working
> properly. Could it be a matter of DNS is supposed to be configured on the
> SNs (I have servicenode.nameserver=1) and it's not configured despite the
> fact that the site.nameservers shows the IP for the xCAT MN (I edited that
> out of my earlier email). Should site.nameservers have the SN IPs in there
> as well for DNS to work? What about site.dhcpinterfaces? Will adding the
> SNs in there cause makedhcp to build working DHCP configs on the SNs?
>
> Christian D. Caruthers
> Linux HPC Consultant
> STG Lab Services
> 757-656-9675
>
>
>
> From: Lissa Valletta/Poughkeepsie/IBM@IBMUS
>
>
> To: xCAT Users Mailing list<xcat-user@lists.sourceforge.net>
>
>
> Cc: xcat-user@lists.sourceforge.net
>
>
> Date: 07/21/2011 01:26 PM
>
>
> Subject: Re: [xcat-user] service nodes node working correctly
>
>
>
>
>
>
>
> So first I think we need to make sure your /etc/xcat/hostkeys are setup
> correctly on the Service Nodes. That should have been done during the
> install of the Service Nodes. Not sure what happened there. You can run
> xdsh<service> -K and that will correctly setup all the credentials and
> keys on the Service Nodes. Assuming you have the service as your
> servicenode group. You will be prompted for root's password.
>
> A couple of things to check after this. Can you ssh to the service node
> with no password prompt and can you access the database from the
> servicenode.
> xdsh sn01,sn02 date
> xdsh sn01,sn02 tabdump site
>
> One other thing rpm -qa | grep xCAT for each service node and return the
> output. Just checking that the xCATsn-* rpm is on the service nodes not
> the xCAT-* rpm which is for the Management Node. This has happened
> before.
>
> It looks like you are setup to have a dhcp server running on both service
> nodes. Isn't this a bad on a flat network. I thought we could only have
> one dhcp server. This is not my strong area.
>
> Having the install directory mounted is typical.
>
> Looking at your definition of node001. You do have servicenode name the
> same as xcatmaster. The servicenode is suppose to be the ip address as
> known by the management node. The xcatmaster is suppose to be the
> servicenode address as known by node0001. Is it correct to be the same?
> Also will the xcatmaster name be resolved correctly during install.
> servicenode=sn02
> xcatmaster=sn02
>
> Lissa K. Valletta
> 2-3/T12
> Poughkeepsie, NY 12601
> (tie 293) 433-3102
>
>
>
>
>
> From:
> Christian
> Caruthers/Richmond/IBM@IBMUS
> To:
> xcat-user@lists.sourceforge.net
> Date:
> 07/21/2011 10:50 AM
> Subject:
> [xcat-user]
> service nodes node working
> correctly
>
>
>
>
> I have a number of compute nodes, 2 service nodes and an xCAT MN (xCAT
> 2.5.2) on a flat network. I would like to have all available services
> running off the service nodes, but I am running into some problems.
>
> When a compute node PXE boots for stateful install (rhel 6.1) it gets a
> DHCP response from the xCAT MN rather than the SNs. Looking on the SNs, I
> see an empty dhcpd.leases file. Running makedhcp doesn't resolve this.
>
> After the DHCP response, the node pulls its boot image and installation
> from the correct SN, but it fails in updating its status and reinstalls
> after rebooting. If I run nodeset node boot from the MN, some of the
> postscripts don't appear to run correctly. For example, remoteshell doesn't
> run, and when I run it using updatenode from the MN, I get an error:
>
> <error>Unable to read private DSA key from /etc/xcat/hostkeys</error>
> <error>Unable to read private RSA key from /etc/xcat/hostkeys</error>
>
> Looking on the SNs, I don't see any /etc/xcat/hostkeys directory. What's
> supposed to set this up?
>
> Sharing the /install directory. Currently, my SNs are configured to
> NFS-mount the /install directory from the MN on boot. Is this correct or
> should they be syncing that directory? I may have missed it, but the wiki
> page was unclear on this to me.
>
> Finally, Looking on the node that was installed by the SN, I see syslog is
> configured to log to the SN, but I don't see that happening.
>
> nodels sn01 servicenode
> qservice01: servicenode.dhcpserver: 1
> qservice01: servicenode.tftpserver: 1
> qservice01: servicenode.node: sn01
> qservice01: servicenode.nameserver: 1
> qservice01: servicenode.nimserver: 1
> qservice01: servicenode.ftpserver: 1
> qservice01: servicenode.conserver: 1
> qservice01: servicenode.monserver: 1
> qservice01: servicenode.nfsserver: 1
> qservice01: servicenode.comments:
> qservice01: servicenode.ldapserver:
> qservice01: servicenode.ntpserver:
> qservice01: servicenode.ipforward:
> qservice01: servicenode.disable:
>
> tabdump site
> #key,value,comments,disable
> "xcatdport","3001",,
> "xcatiport","3002",,
> "tftpdir","/tftpboot",,
> "master","mn01",,
> "domain","cluster.net",,
> "installdir","/install",,
> "timezone","America/Chicago",,
> "forwarders","XXX",,
> "dhcpinterfaces","bond0",,
> "ntpservers","mn01",,
> "consoleondemand","yes",,
> "sharedtftp","0",,
> "nameservers","mn01",,
> "installloc","/install",,
>
> nodels node0001 noderes
> qgpu0001: noderes.primarynic: eth0
> qgpu0001: noderes.xcatmaster: sn02
> qgpu0001: noderes.installnic: eth0
> qgpu0001: noderes.netboot: pxe
> qgpu0001: noderes.servicenode: sn02
> qgpu0001: noderes.node: node0001
> qgpu0001: noderes.nfsserver: sn02
> qgpu0001: noderes.tftpserver:
> qgpu0001: noderes.comments:
> qgpu0001: noderes.nfsdir:
> qgpu0001: noderes.disable:
> qgpu0001: noderes.discoverynics:
> qgpu0001: noderes.nimserver:
> qgpu0001: noderes.cmdinterface:
> qgpu0001: noderes.next_osimage:
> qgpu0001: noderes.current_osimage:
> qgpu0001: noderes.monserver:
>
> lsdef sn02
>
> Object name: sn02
> arch=x86_64
> bmc=sn02-bmc
> bmcport=0
> currchain=boot
> currstate=boot
> groups=service,ipmi,bnt102-service,x3650m2,all
> initrd=xcat/rhels5.4/x86_64/initrd.img
> installnic=eth0
> interface=eth0
> ip=XXXXXX
> kcmdline=nofb utf8 ks=http://mn01/install/autoinst/qservice02
> ksdevice=eth0 console=tty0 console=ttyS0,115200 noipv6
> kernel=xcat/rhels5.4/x86_64/vmlinuz
> mac=E4:1F:13:44:F5:9C
> mgt=ipmi
> mtm=7945AC1
> netboot=pxe
> nfsserver=mn01
> os=rhels5.4
> postbootscripts=otherpkgs,setupntp,setupntp
>
> postscripts=syslog,remoteshell,syncfiles,nwu.service,servicenode,xcatserver,xcatclient
>
>
>
>
>
> primarynic=eth0
> profile=service
> provmethod=install
> serial= 06GA470
> serialport=0
> serialspeed=115200
> servicenode=mn01
> setupconserver=1
> setupdhcp=1
> setupftp=1
> setupnameserver=1
> setupnfs=1
> setupnim=1
> setuptftp=1
> status=booting
> statustime=07-20-2011 16:25:39
> switch=bnt102
> switchport=8
> tftpserver=mn01
> xcatmaster=mn01
>
> lsdef node0001
>
> Object name: node0001
> arch=x86_64
> bmc=node0001-bmc
> bmcport=0
> chain=runcmd=bmcsetup,standby
> currchain=boot
> currstate=boot
> groups=gpu,ipmi,dx360m3,gpubnt01,gpurack01,all,allgpu
> initrd=xcat/rhels6.1/x86_64/initrd.img
> installnic=eth0
> interface=eth0
> ip=XXXXXX
> kcmdline=nofb utf8 ks=http://sn02/install/autoinst/qgpu0001
> ksdevice=eth0 console=tty0 console=ttyS0,115200n8r noipv6
> kernel=xcat/rhels6.1/x86_64/vmlinuz
> mac=e4:1f:13:f0:80:9c
> mgt=ipmi
> mtm=6391AC1
> netboot=pxe
> nfsserver=sn02
> ondiscover=nodediscover
> os=rhels6.1
> postbootscripts=otherpkgs,setupntp,nwu.ipoib
> postscripts=syslog,remoteshell,syncfiles,nwu.ofed
> primarynic=eth0
> profile=gpu
> provmethod=install
> serial=06CGM96
> serialflow=hard
> serialport=0
> serialspeed=115200
> servicenode=sn02
> status=booted
> statustime=07-20-2011 22:23:36
> supportedarchs=x86,x86_64
> switch=gpubnt01
> switchinterface=eth0
> switchport=1
> switchvlan=1
> xcatmaster=sn02
>
> Christian D. Caruthers
> Linux HPC Consultant
> STG Lab Services
> 757-656-9675
>
>
> ------------------------------------------------------------------------------
>
>
>
>
>
> 5 Ways to Improve& Secure Unified Communications
> Unified Communications promises greater efficiencies for business. UC can
> improve internal communications as well as offer faster, more efficient
> ways
> to interact with customers and streamline customer service. Learn more!
> http://www.accelacomm.com/jaw/sfnl/114/51426253/
> _______________________________________________
> xCAT-user mailing list
> xCAT-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xcat-user
>
>
>
> ------------------------------------------------------------------------------
>
>
>
>
> 5 Ways to Improve& Secure Unified Communications
> Unified Communications promises greater efficiencies for business. UC can
> improve internal communications as well as offer faster, more efficient
> ways
> to interact with customers and streamline customer service. Learn more!
> http://www.accelacomm.com/jaw/sfnl/114/51426253/
> _______________________________________________
> xCAT-user mailing list
> xCAT-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xcat-user
>
>
>
>
> ------------------------------------------------------------------------------
>
>
>
> 5 Ways to Improve& Secure Unified Communications
> Unified Communications promises greater efficiencies for business. UC can
> improve internal communications as well as offer faster, more efficient
> ways
> to interact with customers and streamline customer service. Learn more!
> http://www.accelacomm.com/jaw/sfnl/114/51426253/
> _______________________________________________
> xCAT-user mailing list
> xCAT-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xcat-user
>
>
>
>
> ------------------------------------------------------------------------------
>
>
> 5 Ways to Improve& Secure Unified Communications
> Unified Communications promises greater efficiencies for business. UC can
> improve internal communications as well as offer faster, more efficient
> ways
> to interact with customers and streamline customer service. Learn more!
> http://www.accelacomm.com/jaw/sfnl/114/51426253/
> _______________________________________________
> xCAT-user mailing list
> xCAT-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xcat-user
>
>
>
>
> ------------------------------------------------------------------------------
>
> 5 Ways to Improve& Secure Unified Communications
> Unified Communications promises greater efficiencies for business. UC can
> improve internal communications as well as offer faster, more efficient
> ways
> to interact with customers and streamline customer service. Learn more!
> http://www.accelacomm.com/jaw/sfnl/114/51426253/
> _______________________________________________
> xCAT-user mailing list
> xCAT-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xcat-user
>
>
>
>
> ------------------------------------------------------------------------------
> 5 Ways to Improve& Secure Unified Communications
> Unified Communications promises greater efficiencies for business. UC can
> improve internal communications as well as offer faster, more efficient ways
> to interact with customers and streamline customer service. Learn more!
> http://www.accelacomm.com/jaw/sfnl/114/51426253/
> _______________________________________________
> xCAT-user mailing list
> xCAT-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xcat-user
>
------------------------------------------------------------------------------
5 Ways to Improve & Secure Unified Communications
Unified Communications promises greater efficiencies for business. UC can
improve internal communications as well as offer faster, more efficient ways
to interact with customers and streamline customer service. Learn more!
http://www.accelacomm.com/jaw/sfnl/114/51426253/
_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
