Hi I have some problems with CS 4.4.1 - agent was killed with SIGSEGV. Please, help me to detect a source of problem. System Ubuntu Server 14.04 LTS, hypervisor KVM, Ceph RBD as primary storage, secondary storage - Ceph radosgw (S3).
There is records from apport.log: root@lw01p01-node03:~# cat /var/log/apport.log ERROR: apport (pid 43770) Thu Nov 27 09:21:49 2014: called for pid 30814, signal 11, core limit 0 ERROR: apport (pid 43770) Thu Nov 27 09:21:49 2014: executable: /usr/bin/jsvc (command line "jsvc.exec -cp /usr/share/java/commons-daemon.jar:/usr/share/cloudstack-agent/lib/activation-1.1.jar:/usr/share/cloudstack-agent/lib/antisamy-1.4.3.jar:/usr/share/cloudstack-agent/lib/aopalliance-1.0.jar:/usr/share/cloudstack-agent/lib/apache-log4j-extras-1.1.jar:/usr/share/cloudstack-agent/lib/aspectjweaver-1.7.0.jar:/usr/share/cloudstack-agent/lib/aws-java-sdk-1.3.22.jar:/usr/share/cloudstack-agent/lib/batik-css-1.7.jar:/usr/share/cloudstack-agent/lib/batik-ext-1.7.jar:/usr/share/cloudstack-agent/lib/batik-util-1.7.jar:/usr/share/cloudstack-agent/lib/bcprov-jdk16-1.46.jar:/usr/share/cloudstack-agent/lib/bsh-core-2.0b4.jar:/usr/share/cloudstack-agent/lib/cglib-nodep-2.2.2.jar:/usr/share/cloudstack-agent/lib/cloud-agent-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-api-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-core-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-engine-api-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-engine-components-api-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-engine-schema-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-cluster-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-config-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-db-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-events-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-ipc-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-jobs-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-managed-context-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-rest-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-security-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-plugin-hypervisor-kvm-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-plugin-network-ovs-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-server-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloudstack-agent.jar:/usr/share/cloudstack-agent/lib/cloud-utils-4.4.1.jar:/usr/share/cloudstack-agent/lib/commons-beanutils-core-1.7.0.jar:/usr/share/cloudstack-agent/lib/commons-codec-1.6.jar:/usr/share/cloudstack-agent/lib/commons-collections-3.2.jar:/usr/share/cloudstack-agent/lib/commons-configuration-1.8.jar:/usr/share/cloudstack-agent/lib/commons-daemon-1.0.10.jar:/usr/share/cloudstack-agent/lib/commons-dbcp-1.4.jar:/usr/share/cloudstack-agent/lib/commons-fileupload-1.2.jar:/usr/share/cloudstack-agent/lib/commons-httpclient-3.1.jar:/usr/share/cloudstack-agent/lib/commons-io-1.4.jar:/usr/share/cloudstack-agent/lib/commons-lang-2.6.jar:/usr/share/cloudstack-agent/lib/commons-logging-1.1.1.jar:/usr/share/cloudstack-agent/lib/commons-net-3.3.jar:/usr/share/cloudstack-agent/lib/commons-pool-1.6.jar:/usr/share/cloudstack-agent/lib/cxf-bundle-jaxrs-2.7.0.jar:/usr/share/cloudstack-agent/lib/dom4j-1.6.1.jar:/usr/share/cloudstack-agent/lib/ehcache-core-2.6.6.jar:/usr/share/cloudstack-agent/lib/ejb-api-3.0.jar:/usr/share/cloudstack-agent/lib/esapi-2.0.1.jar:/usr/share/cloudstack-agent/lib/geronimo-javamail_1.4_spec-1.7.1.jar:/usr/share/cloudstack-agent/lib/geronimo-servlet_3.0_spec-1.0.jar:/usr/share/cloudstack-agent/lib/gson-1.7.2.jar:/usr/share/cloudstack-agent/lib/guava-14.0-rc1.jar:/usr/share/cloudstack-agent/lib/httpclient-4.2.1.jar:/usr/share/cloudstack-agent/lib/httpcore-4.2.1.jar:/usr/share/cloudstack-agent/lib/jackson-annotations-2.1.1.jar:/usr/share/cloudstack-agent/lib/jackson-core-2.1.1.jar:/usr/share/cloudstack-agent/lib/jackson-core-asl-1.8.9.jar:/usr/share/cloudstack-agent/lib/jackson-databind-2.1.1.jar:/usr/share/cloudstack-agent/lib/jackson-jaxrs-json-provider-2.1.1.jar:/usr/share/cloudstack-agent/lib/jackson-mapper-asl-1.8.9.jar:/usr/share/cloudstack-agent/lib/jackson-module-jaxb-annotations-2.1.1.jar:/usr/share/cloudstack-agent/lib/jasypt-1.9.0.jar:/usr/share/cloudstack-agent/lib/java-ipv6-0.10.jar:/usr/share/cloudstack-agent/lib/javassist-3.12.1.GA.jar:/usr/share/cloudstack-agent/lib/javassist-3.18.1-GA.jar:/usr/share/cloudstack-agent/lib/javax.inject-1.jar:/usr/share/cloudstack-agent/lib/javax.persistence-2.0.0.jar:/usr/share/cloudstack-agent/lib/javax.ws.rs-api-2.0-m10.jar:/usr/share/cloudstack") ERROR: apport (pid 43770) Thu Nov 27 09:21:49 2014: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment ERROR: apport (pid 43770) Thu Nov 27 09:21:49 2014: apport: report /var/crash/_usr_bin_jsvc.0.crash already exists and unseen, doing nothing to avoid disk usage DoS Crash reports are to big to post in message, next is top lines of one. root@lw01p01-node03:~# head -100 /var/crash/_usr_bin_jsvc.0.crash ProblemType: Crash Architecture: amd64 Date: Wed Nov 19 14:44:28 2014 DistroRelease: Ubuntu 14.04 ExecutablePath: /usr/bin/jsvc ExecutableTimestamp: 1397734309 ProcCmdline: jsvc.exec -cp /usr/share/java/commons-daemon.jar:/usr/share/cloudstack-agent/lib/activation-1.1.jar:/usr/share/cloudstack-agent/lib/antisamy-1.4.3.jar:/usr/share/cloudstack-agent/lib/aopalliance-1.0.jar:/usr/share/cloudstack-agent/lib/apache-log4j-extras-1.1.jar:/usr/share/cloudstack-agent/lib/aspectjweaver-1.7.0.jar:/usr/share/cloudstack-agent/lib/aws-java-sdk-1.3.22.jar:/usr/share/cloudstack-agent/lib/batik-css-1.7.jar:/usr/share/cloudstack-agent/lib/batik-ext-1.7.jar:/usr/share/cloudstack-agent/lib/batik-util-1.7.jar:/usr/share/cloudstack-agent/lib/bcprov-jdk16-1.46.jar:/usr/share/cloudstack-agent/lib/bsh-core-2.0b4.jar:/usr/share/cloudstack-agent/lib/cglib-nodep-2.2.2.jar:/usr/share/cloudstack-agent/lib/cloud-agent-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-api-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-core-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-engine-api-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-engine-components-api-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-engine-schema-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-cluster-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-config-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-db-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-events-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-ipc-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-jobs-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-managed-context-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-rest-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-framework-security-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-plugin-hypervisor-kvm-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-plugin-network-ovs-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloud-server-4.4.1.jar:/usr/share/cloudstack-agent/lib/cloudstack-agent.jar:/usr/share/cloudstack-agent/lib/cloud-utils-4.4.1.jar:/usr/share/cloudstack-agent/lib/commons-beanutils-core-1.7.0.jar:/usr/share/cloudstack-agent/lib/commons-codec-1.6.jar:/usr/share/cloudstack-agent/lib/commons-collections-3.2.jar:/usr/share/cloudstack-agent/lib/commons-configuration-1.8.jar:/usr/share/cloudstack-agent/lib/commons-daemon-1.0.10.jar:/usr/share/cloudstack-agent/lib/commons-dbcp-1.4.jar:/usr/share/cloudstack-agent/lib/commons-fileupload-1.2.jar:/usr/share/cloudstack-agent/lib/commons-httpclient-3.1.jar:/usr/share/cloudstack-agent/lib/commons-io-1.4.jar:/usr/share/cloudstack-agent/lib/commons-lang-2.6.jar:/usr/share/cloudstack-agent/lib/commons-logging-1.1.1.jar:/usr/share/cloudstack-agent/lib/commons-net-3.3.jar:/usr/share/cloudstack-agent/lib/commons-pool-1.6.jar:/usr/share/cloudstack-agent/lib/cxf-bundle-jaxrs-2.7.0.jar:/usr/share/cloudstack-agent/lib/dom4j-1.6.1.jar:/usr/share/cloudstack-agent/lib/ehcache-core-2.6.6.jar:/usr/share/cloudstack-agent/lib/ejb-api-3.0.jar:/usr/share/cloudstack-agent/lib/esapi-2.0.1.jar:/usr/share/cloudstack-agent/lib/geronimo-javamail_1.4_spec-1.7.1.jar:/usr/share/cloudstack-agent/lib/geronimo-servlet_3.0_spec-1.0.jar:/usr/share/cloudstack-agent/lib/gson-1.7.2.jar:/usr/share/cloudstack-agent/lib/guava-14.0-rc1.jar:/usr/share/cloudstack-agent/lib/httpclient-4.2.1.jar:/usr/share/cloudstack-agent/lib/httpcore-4.2.1.jar:/usr/share/cloudstack-agent/lib/jackson-annotations-2.1.1.jar:/usr/share/cloudstack-agent/lib/jackson-core-2.1.1.jar:/usr/share/cloudstack-agent/lib/jackson-core-asl-1.8.9.jar:/usr/share/cloudstack-agent/lib/jackson-databind-2.1.1.jar:/usr/share/cloudstack-agent/lib/jackson-jaxrs-json-provider-2.1.1.jar:/usr/share/cloudstack-agent/lib/jackson-mapper-asl-1.8.9.jar:/usr/share/cloudstack-agent/lib/jackson-module-jaxb-annotations-2.1.1.jar:/usr/share/cloudstack-agent/lib/jasypt-1.9.0.jar:/usr/share/cloudstack-agent/lib/java-ipv6-0.10.jar:/usr/share/cloudstack-agent/lib/javassist-3.12.1.GA.jar:/usr/share/cloudstack-agent/lib/javassist-3.18.1-GA.jar:/usr/share/cloudstack-agent/lib/javax.inject-1.jar:/usr/share/cloudstack-agent/lib/javax.persistence-2.0.0.jar:/usr/share/cloudstack-agent/lib/javax.ws.rs-api-2.0-m10.jar:/usr/share/cloudstack ProcCwd: / ProcEnviron: TERM=xterm-256color PATH=(custom, no user) LANG=en_US.UTF-8 LD_LIBRARY_PATH=<set> ProcMaps: 00400000-0040b000 r-xp 00000000 fc:01 1068396 /usr/bin/jsvc 0060a000-0060b000 r--p 0000a000 fc:01 1068396 /usr/bin/jsvc 0060b000-0060c000 rw-p 0000b000 fc:01 1068396 /usr/bin/jsvc 0154c000-0156d000 rw-p 00000000 00:00 0 [heap] 0156d000-01e38000 rw-p 00000000 00:00 0 [heap] 407a00000-408f00000 rw-p 00000000 00:00 0 ... and also: ... 7fffa6ffe000-7fffa7000000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] ProcStatus: Name: jsvc State: D (disk sleep) Tgid: 27041 Ngid: 27041 Pid: 27041 PPid: 27038 TracerPid: 0 Uid: 0 0 0 0 Gid: 0 0 0 0 FDSize: 256 Groups: 0 VmPeak: 22568424 kB VmSize: 22444004 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 376728 kB VmRSS: 249872 kB VmData: 22250264 kB VmStk: 1012 kB VmExe: 44 kB VmLib: 37864 kB VmPTE: 1732 kB VmSwap: 30216 kB Threads: 61 SigQ: 4/514887 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000004 SigIgn: 0000000000001000 SigCgt: 20000001810048cf CapInh: 0000000000000000 CapPrm: 0000001fffffffff CapEff: 0000001fffffffff CapBnd: 0000001fffffffff Seccomp: 0 Cpus_allowed: ffff,ffffffff Cpus_allowed_list: 0-47 Mems_allowed: 00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 270184 nonvoluntary_ctxt_switches: 454 Signal: 11 Uname: Linux 3.14.1-031401-generic x86_64 UserGroups: _LogindSession: /user/1000.user/3.session CoreDump: base64 ... -- Michael Kolomiets
