ERROR: Java process not running. Try restarting the SSVM.

[Cloud List]

Dear all,

I am running ACS 4.2.0 in our test environment and after restarting the KVM
host, I am not able to get SSVM agent up and running, despite destroying
and recreating it.

The problem is similar to what Mike Tutkowsi reported on below thread, same
symptoms but different cloud.out stack error message:

http://comments.gmane.org/gmane.comp.apache.cloudstack.devel/28827

I can SSH into SSVM but unable to start the Java agent. SSVM health check
result shows two error:
- Unable to connect NFS (wrong IP detected); and
- Unable to start Java.

Unable to connect NFS is because wrong IP is detected for the NFS ip. I can
manually mount the NFS server IP using correct IP.

Result:


root@s-121-VM:/var/log/cloud# /usr/local/cloud/systemvm/ssvm-check.sh

First DNS server is  8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=49 time=4.572 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=49 time=4.365 ms
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 4.365/4.468/4.572/0.104 ms
Good: Can ping DNS server

Good: DNS resolves download.cloud.com

ERROR: NFS is not currently mounted
Try manually mounting from inside the VM
NFS server is  169.254.0.127
PING 169.254.0.127 (169.254.0.127): 56 data bytes
64 bytes from 169.254.0.127: icmp_seq=0 ttl=64 time=0.106 ms
64 bytes from 169.254.0.127: icmp_seq=1 ttl=64 time=0.040 ms
--- 169.254.0.127 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.040/0.073/0.106/0.033 ms
Good: Can ping NFS server

Management server is 192.168.0.222. Checking connectivity.
Good: Can connect to management server port 8250

ERROR: Java process not running.  Try restarting the SSVM.


Repeated error shown on /var/log/cloud/cloud.out:


+ java -Djavax.net.ssl.trustStore=./certs/realhostip.keystore
-Dlog.home=/var/log/cloud/ -mx-24m -cp
./:./conf:activation-1.1.jar:antisamy-1.4.3.jar:aopalliance-1.0.jar:apache-log4j-extras-1.1.jar:aspectjrt-1.7.1.jar:aspectjweaver-1.7.1.jar:aws-java-sdk-1.3.22.jar:backport-util-concurrent-3.1.jar:batik-css-1.7.jar:batik-ext-1.7.jar:batik-util-1.7.jar:bcprov-jdk16-1.46.jar:bsh-core-2.0b4.jar:c3p0-0.9.1.1.jar:cglib-nodep-2.2.2.jar:cloud-agent-4.2.0.jar:cloud-api-4.2.0.jar:cloud-console-proxy-4.2.0.jar:cloud-core-4.2.0.jar:cloud-engine-api-4.2.0.jar:cloud-engine-components-api-4.2.0.jar:cloud-engine-schema-4.2.0.jar:cloud-framework-events-4.2.0.jar:cloud-framework-ipc-4.2.0.jar:cloud-framework-jobs-4.2.0.jar:cloud-framework-rest-4.2.0.jar:cloud-secondary-storage-4.2.0.jar:cloud-server-4.2.0.jar:cloud-utils-4.2.0.jar:commons-beanutils-core-1.7.0.jar:commons-codec-1.6.jar:commons-collections-3.2.1.jar:commons-configuration-1.8.jar:commons-daemon-1.0.10.jar:commons-dbcp-1.4.jar:commons-discovery-0.5.jar:commons-fileupload-1.2.jar:commons-httpclient-3.1.jar:commons-io-1.4.jar:commons-lang-2.6.jar:commons-logging-1.1.1.jar:commons-pool-1.6.jar:cxf-bundle-jaxrs-2.7.0.jar:dom4j-1.6.1.jar:ehcache-1.5.0.jar:ejb-api-3.0.jar:esapi-2.0.1.jar:geronimo-javamail_1.4_spec-1.7.1.jar:geronimo-servlet_3.0_spec-1.0.jar:gson-1.7.1.jar:guava-11.0.2.jar:httpclient-4.2.1.jar:httpcore-4.2.1.jar:jackson-annotations-2.1.1.jar:jackson-core-2.1.1.jar:jackson-core-asl-1.8.9.jar:jackson-databind-2.1.1.jar:jackson-jaxrs-json-provider-2.1.1.jar:jackson-mapper-asl-1.8.9.jar:jackson-module-jaxb-annotations-2.1.1.jar:jasypt-1.9.0.jar:java-ipv6-0.10.jar:javassist-3.12.1.GA.jar:javax.inject-1.jar:javax.persistence-2.0.0.jar:javax.ws.rs-api-2.0-m10.jar:jaxen-1.1-beta-8.jar:jdom-1.0.jar:jetty-continuation-8.1.7.v20120910.jar:jetty-http-8.1.7.v20120910.jar:jetty-io-8.1.7.v20120910.jar:jetty-security-8.1.7.v20120910.jar:jetty-util-8.1.7.v20120910.jar:jsch-0.1.42.jar:jsr107cache-1.0.jar:jsr305-1.3.9.jar:jstl-1.2.jar:log4j-1.2.16.jar:mail-1.4.jar:nekohtml-1.9.12.jar:quartz-2.1.6.jar:reflections-0.9.8.jar:slf4j-api-1.6.2.jar:spring-aop-3.1.2.RELEASE.jar:spring-asm-3.1.2.RELEASE.jar:spring-beans-3.1.2.RELEASE.jar:spring-context-3.1.2.RELEASE.jar:spring-core-3.1.2.RELEASE.jar:spring-expression-3.1.2.RELEASE.jar:spring-web-3.1.2.RELEASE.jar:stax2-api-3.1.1.jar:trilead-ssh2-build213-svnkit-1.3-patch.jar:woodstox-core-asl-4.1.4.jar:wsdl4j-1.6.2.jar:xalan-2.7.0.jar:xercesImpl-2.6.2.jar:xml-apis-1.0.b2.jar:xml-apis-ext-1.3.04.jar:xmlParserAPIs-2.6.2.jar:xmlschema-core-2.0.3.jar:xom-1.1.jar:xpp3_min-1.1.4c.jar:xstream-1.3.1.jar
com.cloud.agent.AgentShell template=domP type=secstorage host=192.168.0.222
port=8250 name=s-121-VM zone=1 pod=1 guid=s-121-VM
resource=org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource
instance=SecStorage sslcopy=true role=templateProcessor mtu=1500
eth2ip=192.168.5.170 

Help with Usage data - There is no data from Cloudmonkey of types: 4,5,10,21,22,23 or 24 (Network and VM Disk data)

[Cory Fuchs]

Hello Everyone,


I hope that someone can point me in the direction of how to resolve my issue 
and let me know what I am doing wrong.


Cloudstack version: 4.5.2


Running the following command from Cloudmonkey returns no results.


 list usagerecords startdate=2016-06-16 enddate=2016-06-17 type=4


Changing the type to any of the following returns no results: 4, 5, 10, 21, 22, 
or 23

Changing the type to any of the following does return results: 1, 2, 3, 6, 7, 
8, 9, 11, 12, 13, 14, or 25


This tells me the usage job is running and collecting data.  Just not on the 
Network or VM Disk data.


Looking at /var/log/cloudstack/usage/usage.log


Grepping for an accountId that I know has usage on.  I get lines like the 
following.


 2016-06-17 00:15:03,347 DEBUG [cloud.usage.UsageManagerImpl] 
(Usage-Job-1:null) creating networkHelperEntry...accountId: 7 in zone: 1; abr: 
142919240951; abs: 40911793471; curABS: 40911793471; curABR: 142919240951; ubs: 
0; ubr: 0

 2016-06-17 00:15:03,348 DEBUG [cloud.usage.UsageManagerImpl] 
(Usage-Job-1:null) getting current accounted bytes for... accountId: 7 in zone: 
1; abr: 39361808078; abs: 1006529247

  2016-06-17 00:15:03,348 DEBUG [cloud.usage.UsageManagerImpl] 
(Usage-Job-1:null) creating networkHelperEntry... accountId: 7 in zone: 1; abr: 
39361808078; abs: 1006529247; curABS: 1006529247; curABR: 39361808078; ubs: 0; 
ubr: 0



Grepping for account 7.  I get some of the following lines:


 2016-06-17 00:15:13,976 DEBUG [usage.parser.NetworkUsageParser] 
(Usage-Job-1:null) No usage record (0 bytes used) generated for account: 7

 2016-06-17 00:15:13,976 DEBUG [usage.parser.NetworkUsageParser] 
(Usage-Job-1:null) No usage record (0 bytes used) generated for account: 7

 2016-06-17 00:15:13,976 DEBUG [usage.parser.VmDiskUsageParser] 
(Usage-Job-1:null) Parsing all Vm Disk usage events for account: 7

 2016-06-17 00:15:14,074 DEBUG [usage.parser.VmDiskUsageParser] 
(Usage-Job-1:null) No vm disk usage record (0 bytes used) generated for 
account: 7

 2016-06-17 00:15:14,074 DEBUG [usage.parser.VmDiskUsageParser] 
(Usage-Job-1:null) No vm disk usage record (0 bytes used) generated for 
account: 7


Where can I look or what am I doing wrong?

I am mainly looking for network usage.


Thank you to anyone that can assist.


Cory Fuchs

Re: CloudStack Design: Ceph and local storage

[Frank Louwers]

Has anyone looked at StorPool? They seem to be doing an OnApp-storage like 
setup…

Regards,

Frank


> On 17 Jun 2016, at 16:24, Dustin Wright  
> wrote:
> 
> Thank you for the valuable feedback. I have been considering the same setup
> Jeroen proposed.
> 
> I think what most of us want is something like onapp integrated storage. At
> least that is what it sounds like to me.
> 
> It would be neat if ACS had a system vm baked in for running ceph, or some
> type of distributed storage. Either VM or object storage. IMO, Running big
> expensive enterprise SAN for your block storage doesn't feel very
> cloud-like.
> 
> Dustin
> 
> On Fri, Jun 17, 2016 at 9:59 AM, Stephan Seitz <
> s.se...@secretresearchfacility.com> wrote:
> 
>> Hi!
>> 
>> Independently from cloudstack, I'ld strongly recommend to not use ceph
>> and hypervisors on the very same machines. If you just want to build a
>> POC this is fine, but If you put load on it, you'll see unpredictible
>> behavior (at least on the ceph side) due to heavy I/O demands.
>> Ceph recommends at least 1 Core and 1 GB RAM as a rule of thumb for
>> each OSD.
>> BTW. I also won't run a ceph cluster with only two nodes. Your MON
>> should be able to form a quorum, so you'ld need at least three  nodes.
>> 
>> If you run a cluster with less than about 6 or 8 nodes, I'ld give
>> gluster a try. I've never tried it myself but I assume this should
>> be usable as "pre-setup" Storage at least with KVM Hosts.
>> 
>> cheers,
>> 
>> - Stephan
>> 
>> 
>> 
>> Am Freitag, den 17.06.2016, 13:36 +0200 schrieb Jeroen Keerrel:
>>> Good afternoon from Hamburg, Germany!
>>> 
>>> Short question:
>>> Is it feasible to use CloudStack with Ceph on local storage? As in
>>> “hyperconverged”?
>>> 
>>> Before ramping up the infrastructure, I’d like to be sure, before
>>> buying new hardware.
>>> 
>>> At the moment: 2 Hosts, each 2 6c XEON CPU, 24GB RAM and each have 6
>>> 300GB SAS drives.
>>> 
>>> According to CEPH, they advise bigger disks and separate storage
>>> “nodes”.
>>> CloudStack documentation says: Smaller, High RPM disks.
>>> 
>>> What would you advise? Buy separate “Storage Nodes” or  ramp up the
>>> current nodes?
>>> 
>>> Cheers!
>>> Jeroen
>>> 
>>> 
>>> 
>>> Jeroen Keerl
>>> Keerl IT Services GmbH
>>> Birkenstraße 1b . 21521 Aumühle
>>> +49 177 6320 317
>>> www.keerl-it.com
>>> i...@keerl-it.com
>>> Geschäftsführer. Jacobus J. Keerl
>>> Registergericht Lubeck. HRB-Nr. 14511
>>> Unsere Allgemeine Geschäftsbedingungen finden Sie hier.
>>> 
>>> 
>> 

Re: CloudStack Design: Ceph and local storage

[Dustin Wright]

Thank you for the valuable feedback. I have been considering the same setup
Jeroen proposed.

I think what most of us want is something like onapp integrated storage. At
least that is what it sounds like to me.

It would be neat if ACS had a system vm baked in for running ceph, or some
type of distributed storage. Either VM or object storage. IMO, Running big
expensive enterprise SAN for your block storage doesn't feel very
cloud-like.

Dustin

On Fri, Jun 17, 2016 at 9:59 AM, Stephan Seitz <
s.se...@secretresearchfacility.com> wrote:

> Hi!
>
> Independently from cloudstack, I'ld strongly recommend to not use ceph
> and hypervisors on the very same machines. If you just want to build a
> POC this is fine, but If you put load on it, you'll see unpredictible
> behavior (at least on the ceph side) due to heavy I/O demands.
> Ceph recommends at least 1 Core and 1 GB RAM as a rule of thumb for
> each OSD.
> BTW. I also won't run a ceph cluster with only two nodes. Your MON
> should be able to form a quorum, so you'ld need at least three  nodes.
>
> If you run a cluster with less than about 6 or 8 nodes, I'ld give
> gluster a try. I've never tried it myself but I assume this should
> be usable as "pre-setup" Storage at least with KVM Hosts.
>
> cheers,
>
> - Stephan
>
>
>
> Am Freitag, den 17.06.2016, 13:36 +0200 schrieb Jeroen Keerrel:
> > Good afternoon from Hamburg, Germany!
> >
> > Short question:
> > Is it feasible to use CloudStack with Ceph on local storage? As in
> > “hyperconverged”?
> >
> > Before ramping up the infrastructure, I’d like to be sure, before
> > buying new hardware.
> >
> > At the moment: 2 Hosts, each 2 6c XEON CPU, 24GB RAM and each have 6
> > 300GB SAS drives.
> >
> > According to CEPH, they advise bigger disks and separate storage
> > “nodes”.
> > CloudStack documentation says: Smaller, High RPM disks.
> >
> > What would you advise? Buy separate “Storage Nodes” or  ramp up the
> > current nodes?
> >
> > Cheers!
> > Jeroen
> >
> >
> >
> > Jeroen Keerl
> > Keerl IT Services GmbH
> > Birkenstraße 1b . 21521 Aumühle
> > +49 177 6320 317
> > www.keerl-it.com
> > i...@keerl-it.com
> > Geschäftsführer. Jacobus J. Keerl
> > Registergericht Lubeck. HRB-Nr. 14511
> > Unsere Allgemeine Geschäftsbedingungen finden Sie hier.
> >
> >
>

Re: CloudStack Design: Ceph and local storage

[Stephan Seitz]

Hi!

Independently from cloudstack, I'ld strongly recommend to not use ceph
and hypervisors on the very same machines. If you just want to build a
POC this is fine, but If you put load on it, you'll see unpredictible
behavior (at least on the ceph side) due to heavy I/O demands.
Ceph recommends at least 1 Core and 1 GB RAM as a rule of thumb for
each OSD.
BTW. I also won't run a ceph cluster with only two nodes. Your MON
should be able to form a quorum, so you'ld need at least three  nodes.

If you run a cluster with less than about 6 or 8 nodes, I'ld give
gluster a try. I've never tried it myself but I assume this should
be usable as "pre-setup" Storage at least with KVM Hosts.

cheers,

- Stephan



Am Freitag, den 17.06.2016, 13:36 +0200 schrieb Jeroen Keerrel:
> Good afternoon from Hamburg, Germany!
>  
> Short question:
> Is it feasible to use CloudStack with Ceph on local storage? As in
> “hyperconverged”?
>  
> Before ramping up the infrastructure, I’d like to be sure, before
> buying new hardware.
>  
> At the moment: 2 Hosts, each 2 6c XEON CPU, 24GB RAM and each have 6
> 300GB SAS drives.
>  
> According to CEPH, they advise bigger disks and separate storage
> “nodes”.
> CloudStack documentation says: Smaller, High RPM disks.
>  
> What would you advise? Buy separate “Storage Nodes” or  ramp up the
> current nodes?
>  
> Cheers!
> Jeroen
>  
> 
> 
> Jeroen Keerl
> Keerl IT Services GmbH
> Birkenstraße 1b . 21521 Aumühle
> +49 177 6320 317
> www.keerl-it.com
> i...@keerl-it.com
> Geschäftsführer. Jacobus J. Keerl
> Registergericht Lubeck. HRB-Nr. 14511
> Unsere Allgemeine Geschäftsbedingungen finden Sie hier.
> 
> 

CloudStack Design: Ceph and local storage

[Jeroen Keerl]

Good afternoon from Hamburg, Germany!

Short question:
Is it feasible to use CloudStack with Ceph on local storage? As in 
“hyperconverged”?

Before ramping up the infrastructure, I’d like to be sure, before buying new 
hardware.

At the moment: 2 Hosts, each 2 6c XEON CPU, 24GB RAM and each have 6 300GB SAS 
drives.

According to CEPH, they advise bigger disks and separate storage “nodes”.
CloudStack documentation says: Smaller, High RPM disks.

What would you advise? Buy separate “Storage Nodes” or  ramp up the current 
nodes?

Cheers!
Jeroen





Jeroen Keerl


Keerl IT Services GmbH
Birkenstraße 1b . 21521 Aumühle

+49 177 6320 317

www.keerl-it.com
i...@keerl-it.com

Geschäftsführer. Jacobus J. Keerl
Registergericht Lubeck. HRB-Nr. 14511

Unsere Allgemeine Geschäftsbedingungen finden Sie hier.

Re: SSVM Agent state connecting

[Makrand]

SSH into SSVM and just STOP and START (not reboot pr reload) the cloud
service. After some time it comes back to up state. This is what I've
observed in 4.3 CS version.


What version you're on?

--
Best,
Makrand


On Fri, Jun 17, 2016 at 11:58 AM, Timothy Lothering 
wrote:

> Hi Mohd,
>
> Try using the following URL to assist in troubleshooting:
>
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM,+templates,+Seco
> ndary+storage+troubleshooting
>
>
> -Original Message-
> From: Mohd Zainal Abidin Rabani [mailto:zai...@nocser.net]
> Sent: Thursday, 16 June 2016 10:02 AM
> To: users@cloudstack.apache.org
> Subject: SSVM Agent state connecting
>
> Hi,
>
>
>
> I having problem with SSVM. I can ssh SSVM and login. How to make agent
> state running from connecting state. Please advice.
>
>
>
> Thanks.
>
>
>

RE: SSVM Agent state connecting

[Timothy Lothering]

Hi Mohd,

Try using the following URL to assist in troubleshooting:

https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM,+templates,+Seco
ndary+storage+troubleshooting


-Original Message-
From: Mohd Zainal Abidin Rabani [mailto:zai...@nocser.net] 
Sent: Thursday, 16 June 2016 10:02 AM
To: users@cloudstack.apache.org
Subject: SSVM Agent state connecting

Hi,

 

I having problem with SSVM. I can ssh SSVM and login. How to make agent
state running from connecting state. Please advice.

 

Thanks.