Duncan Godwin created BROOKLYN-327:
--------------------------------------
Summary: Intermittent JBoss AS 6 Failure on Centos 6
Key: BROOKLYN-327
URL: https://issues.apache.org/jira/browse/BROOKLYN-327
Project: Brooklyn
Issue Type: Bug
Environment: Centos 6 on SoftLayer
Reporter: Duncan Godwin
JBoss AS 6 fails to reach SERVICE_UP on Centos 6 on SoftLayer intermittently (1
in 4 tests). The service does not start as java is not installed. The problem
seems to happen during the install java phase. The console file contains:
{code}
/home/users/amp/brooklyn-managed-processes/installs/JBoss6Server_6.0.0.Final/jboss-6.0.0.Final/bin/run.sh:
line 257: java: command not found
{code}
When SSH'd onto the machine yum doesn't work reporting the following for any
command:
{code}
[amp@br-bbrkxl0ipn JBoss6Server_bbrkxl0ipn]$ sudo yum install unzip
Loaded plugins: fastestmirror, security
Setting up Install Process
Loading mirror speeds from cached hostfile
[amp@br-bbrkxl0ipn JBoss6Server_bbrkxl0ipn]$ echo $?
132
{code}
Yum reports the following in /var/log/messages
{code}
Aug 9 11:28:47 (none) kernel: yum[1255] trap invalid opcode ip:7f6bd61f0d60
sp:7fff38c17f58 error:0 in libfreeblpriv3.so[7f6bd619e000+72000]
Aug 9 11:28:47 (none) abrtd: Directory 'ccpp-2016-08-09-11:28:47-1255'
creation detected
Aug 9 11:28:47 (none) abrt[1256]: Saved core dump of pid 1255
(/usr/bin/python) to /var/spool/abrt/ccpp-2016-08-09-11:28:47-1255 (16306176
bytes)
Aug 9 11:28:55 (none) abrtd: Generating core_backtrace
Aug 9 11:28:56 (none) abrtd: Duplicate: core backtrace
Aug 9 11:28:56 (none) abrtd: DUP_OF_DIR:
/var/spool/abrt/ccpp-2016-08-09-09:01:07-2000
Aug 9 11:28:56 (none) abrtd: Deleting problem directory
ccpp-2016-08-09-11:28:47-1255 (dup of ccpp-2016-08-09-09:01:07-2000)
Aug 9 11:28:56 (none) abrtd: Sending an email...
Aug 9 11:28:56 (none) abrtd: postdrop: warning: unable to look up
public/pickup: No such file or directory
Aug 9 11:28:56 (none) abrtd: Email was sent to: root@localhost
Aug 9 11:29:04 (none) kernel: yum[1523] trap invalid opcode ip:7fb660f46d60
sp:7ffe958b6288 error:0 in libfreeblpriv3.so[7fb660ef4000+72000]
Aug 9 11:29:04 (none) abrt[1524]: Not saving repeating crash in
'/usr/bin/python'
{code}
The OS on SoftLayer is "CentOS6.0-64 Minimal for VSI"
Jclouds Log:
{code}
[09/08/2016, 17:50:03] Aled Sage: jclouds using templateBuilder
PortableTemplateBuilder[ports=[22], locationId=sng01, imageId=CENTOS_6_64,
imageChooserFunction=org.apache.brooklyn.location.jclouds.BrooklynImageChooser$ImageChooserFromOrdering,
minRam=2000] for provisioning in JcloudsLocation[Softlayer
Singapore:software-development/softlayer:sng01@ton5afii2m] for
softlayer:sng01@JBoss6ServerImpl{id=bbrkxl0ipn}
[09/08/2016, 17:51:23] Aled Sage: jclouds found template
{image={id=CENTOS_6_64, providerId=CENTOS_6_64, os={family=centos, version=6.2,
description=CentOS 6.2-64 LAMP for Bare Metal, is64Bit=true},
description=CENTOS_6_64, status=AVAILABLE, loginUser=root},
hardware={id=cpu=1,memory=2048,disk=25,type=LOCAL,
providerId=cpu=1,memory=2048,disk=25,type=LOCAL, processors=[{cores=1.0,
speed=2.0}], ram=2048, volumes=[{id=0, type=LOCAL, size=25.0, device=0,
bootDevice=false, durable=true}], hypervisor=XenServer,
supportsImage=ALWAYS_TRUE}, location={scope=ZONE, id=sng01,
description=Singapore 1, parent=softlayer, iso3166Codes=[SG],
metadata={name=sng01}}, options={}} (image {id=CENTOS_6_64,
providerId=CENTOS_6_64, os={family=centos, version=6.2, description=CentOS
6.2-64 LAMP for Bare Metal, is64Bit=true}, description=CENTOS_6_64,
status=AVAILABLE, loginUser=root}) for provisioning in
JcloudsLocation[Softlayer
Singapore:software-development/softlayer:sng01@ton5afii2m] for
softlayer:sng01@JBoss6ServerImpl{id=bbrkxl0ipn}
{code}
The error appears to be related to this:
https://bugs.centos.org/view.php?id=11074
probably related - https://bugzilla.redhat.com/show_bug.cgi?id=1257332#c14
Java install stdin:
{code}
( { which apt-get && { echo apt-get exists, doing update && export
DEBIAN_FRONTEND=noninteractive && (( if test "$UID" -eq 0; then ( apt-get
update ); else sudo -E -n -S -- apt-get update; fi ) || true) && ( if test
"$UID" -eq 0; then ( apt-get install -y --allow-unauthenticated openjdk-7-jdk
); else sudo -E -n -S -- apt-get install -y --allow-unauthenticated
openjdk-7-jdk; fi ) ; } ; } || { which yum && { echo yum exists, doing update
&& (( if test "$UID" -eq 0; then ( yum check-update ); else sudo -E -n -S --
yum check-update; fi ) || true) && (( if test "$UID" -eq 0; then ( yum -y
install epel-release ); else sudo -E -n -S -- yum -y install epel-release; fi )
|| true) && (( if test "$UID" -eq 0; then ( yum upgrade -y ca-certificates
--disablerepo=epel ); else sudo -E -n -S -- yum upgrade -y ca-certificates
--disablerepo=epel; fi ) || true) && ( if test "$UID" -eq 0; then ( yum -y
--nogpgcheck install java-1.7.0-openjdk-devel ); else sudo -E -n -S -- yum -y
--nogpgcheck install java-1.7.0-openjdk-devel; fi ) ; } ; } || ( (( echo
"WARNING: no known/successful package manager to install {apt=openjdk-7-jdk,
yum=java-1.7.0-openjdk-devel}, may fail subsequently" | tee /dev/stderr ) ||
true) && { which zypper && { (( if test "$UID" -eq 0; then ( zypper
--non-interactive addrepo
http://download.opensuse.org/repositories/Java:/openjdk6:/Factory/SLE_11_SP3
java_sles_11 ); else sudo -E -n -S -- zypper --non-interactive addrepo
http://download.opensuse.org/repositories/Java:/openjdk6:/Factory/SLE_11_SP3
java_sles_11; fi ) || true) && (( if test "$UID" -eq 0; then ( zypper
--non-interactive addrepo
http://download.opensuse.org/repositories/Java:/openjdk6:/Factory/openSUSE_11.4
java_suse_11 ); else sudo -E -n -S -- zypper --non-interactive addrepo
http://download.opensuse.org/repositories/Java:/openjdk6:/Factory/openSUSE_11.4
java_suse_11; fi ) || true) && (( if test "$UID" -eq 0; then ( zypper
--non-interactive addrepo
http://download.opensuse.org/repositories/Java:/openjdk6:/Factory/openSUSE_12.3
java_suse_12 ); else sudo -E -n -S -- zypper --non-interactive addrepo
http://download.opensuse.org/repositories/Java:/openjdk6:/Factory/openSUSE_12.3
java_suse_12; fi ) || true) && (( if test "$UID" -eq 0; then ( zypper
--non-interactive addrepo
http://download.opensuse.org/repositories/Java:/openjdk6:/Factory/openSUSE_13.1
java_suse_13 ); else sudo -E -n -S -- zypper --non-interactive addrepo
http://download.opensuse.org/repositories/Java:/openjdk6:/Factory/openSUSE_13.1
java_suse_13; fi ) || true) && ( ( { which zypper && { echo zypper exists,
doing refresh && (( if test "$UID" -eq 0; then ( zypper --non-interactive
--no-gpg-checks refresh ); else sudo -E -n -S -- zypper --non-interactive
--no-gpg-checks refresh; fi ) || true) && ( if test "$UID" -eq 0; then ( zypper
--non-interactive --no-gpg-checks install java-1_7_0-openjdk-devel ); else sudo
-E -n -S -- zypper --non-interactive --no-gpg-checks install
java-1_7_0-openjdk-devel; fi ) ; } ; } || ( (( echo "WARNING: no
known/successful package manager to install {zypper=java-1_7_0-openjdk-devel},
may fail subsequently" | tee /dev/stderr ) || true) && exit 9 ) ) || ( { which
zypper && { echo zypper exists, doing refresh && (( if test "$UID" -eq 0; then
( zypper --non-interactive --no-gpg-checks refresh ); else sudo -E -n -S --
zypper --non-interactive --no-gpg-checks refresh; fi ) || true) && ( if test
"$UID" -eq 0; then ( zypper --non-interactive --no-gpg-checks install
java-1_7_0-ibm ); else sudo -E -n -S -- zypper --non-interactive
--no-gpg-checks install java-1_7_0-ibm; fi ) ; } ; } || ( (( echo "WARNING: no
known/successful package manager to install {zypper=java-1_7_0-ibm}, may fail
subsequently" | tee /dev/stderr ) || true) && exit 9 ) ) ) ; } ; } ) )
{code}
Java Install stdout
{code}
/usr/bin/yum
yum exists, doing update
Loaded plugins: fastestmirror, security
Determining fastest mirrors
initscripts.x86_64 9.03.53-1.el6.centos.1 updates
kernel.x86_64 2.6.32-642.3.1.el6 updates
kernel-firmware.noarch 2.6.32-642.3.1.el6 updates
libtiff.x86_64 3.9.4-18.el6_8 updates
mdadm.x86_64 3.3.4-1.el6_8.5 updates
nss-softokn.x86_64 3.14.3-23.3.el6_8 updates
nss-softokn-freebl.x86_64 3.14.3-23.3.el6_8 updates
tar.x86_64 2:1.23-15.el6_8 updates
tzdata.noarch 2016f-1.el6 updates
yum.noarch 3.2.29-75.el6.centos updates
Loaded plugins: fastestmirror, security
Setting up Install Process
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch 0:6-8 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
epel-release noarch 6-8 extras 14 k
Transaction Summary
================================================================================
Install 1 Package(s)
Total download size: 14 k
Installed size: 22 k
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : epel-release-6-8.noarch 1/1
Verifying : epel-release-6-8.noarch 1/1
Installed:
epel-release.noarch 0:6-8
Complete!
Loaded plugins: fastestmirror, security
Setting up Upgrade Process
Loading mirror speeds from cached hostfile
No Packages marked for Update
Loaded plugins: fastestmirror, security
Setting up Install Process
Loading mirror speeds from cached hostfile
WARNING: no known/successful package manager to install {apt=openjdk-7-jdk,
yum=java-1.7.0-openjdk-devel}, may fail subsequently
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
