Hi, On Fri, Aug 01, 2014 at 04:40:25PM +0000, Wendt Christian wrote: > Hello *, > > i did a lot of research but i´m not able to get the purpose whether our > oracle ressource fails on Wednesday. > > Oracle start will fail with the message: > > /usr/lib/ocf/resource.d/heartbeat/oracle start > INFO: orcSNBGW instance state is not OPEN (dbstat output: SP2-0640: Not > connected) > ERROR: oracle instance orcSNBGW not started: > > Showdbstat throws: > > /usr/lib/ocf/resource.d/heartbeat/oracle showdbstat > Full output: > SP2-0640: Not connected > Stripped output: > <OPEN> > So the first method of showdbstat monitoring the DB fails, but the second one > succeeds.
That seems to be a bug with the OCFMON monitoring user password expired. It was fixed. You can either update the oracle RA from upstream or reset the OCFMON password (OCFMON) and set its life to unlimited. Thanks, Dejan > It is not possible to start oracle within the pacemaker cluster anymore. > Everytime we start it, it´ll fail. I´ve attached the bash output while > starting oracle with the ocf script. > > Database and OS is fine. Nothing changed in the last days. > > Do have any ideas? > > Thank you in advance. > > Mit freundlichen Grüßen / Best regards > > Christian Wendt > > Bosch Software Innovations GmbH > Information Technology (INST/PRV3-IT) > Schöneberger Ufer 89-91 > 10785 Berlin > GERMANY > www.bosch-si.de > www.blog.bosch-si.com > > Tel. +49 30.72 61 12-308 > Fax +49 30.72.61.12-100 > [email protected] > > Registered office: Berlin, Register court: Amtsgericht Charlottenburg, HRB > 148411 B > Executives: Rainer Kallenbach; Thomas Cotic, Michael Hahn, Klaus Hüftle > Content-Description: bashx_oracle_ocf.txt > HOSTXXX:/opt/oracle/app/diag/rdbms/orcsnbgw/orcSNBGW/trace # bash -x > /usr/lib/ocf/resource.d/heartbeat/oracle start > + : /usr/lib/ocf/lib/heartbeat > + . /usr/lib/ocf/lib/heartbeat/ocf-shellfuncs > ++ unset LC_ALL > ++ export LC_ALL > ++ unset LANGUAGE > ++ export LANGUAGE > +++ basename /usr/lib/ocf/resource.d/heartbeat/oracle > ++ __SCRIPT_NAME=oracle > ++ '[' -z /usr/lib/ocf ']' > ++ '[' /usr/lib/ocf/lib/heartbeat = /usr/lib/ocf/resource.d/heartbeat ']' > ++ : /usr/lib/ocf/lib/heartbeat > ++ . /usr/lib/ocf/lib/heartbeat/ocf-binaries > +++ > PATH=/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:/sbin:/bin:/usr/sbin:/usr/bin > +++ > PATH=/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/ucb > +++ export PATH > +++ : gawk > +++ : '/usr/bin/grep -E' > +++ : -a > +++ : mail > +++ : /bin/ping > +++ : /bin/sh > +++ : /usr/bin/test > +++ : /usr/bin/test > +++ : basename > +++ : blockdev > +++ : cat > +++ : fsck > +++ : fuser > +++ : getent > +++ : grep > +++ : ifconfig > +++ : iptables > +++ : ip > +++ : mdadm > +++ : modprobe > +++ : mount > +++ : msgfmt > +++ : netstat > +++ : perl > +++ : python > +++ : raidstart > +++ : raidstop > +++ : route > +++ : umount > +++ : reboot > +++ : poweroff > +++ : wget > +++ : whoami > +++ : strings > +++ : scp > +++ : ssh > +++ : swig > +++ : gzip > +++ : tar > +++ : md5 > +++ : drbdadm > +++ : drbdsetup > ++ . /usr/lib/ocf/lib/heartbeat/ocf-returncodes > +++ OCF_SUCCESS=0 > +++ OCF_ERR_GENERIC=1 > +++ OCF_ERR_ARGS=2 > +++ OCF_ERR_UNIMPLEMENTED=3 > +++ OCF_ERR_PERM=4 > +++ OCF_ERR_INSTALLED=5 > +++ OCF_ERR_CONFIGURED=6 > +++ OCF_NOT_RUNNING=7 > +++ OCF_RUNNING_MASTER=8 > +++ OCF_FAILED_MASTER=9 > ++ . /usr/lib/ocf/lib/heartbeat/ocf-directories > +++ prefix=/usr > +++ exec_prefix=/usr > +++ : /etc/init.d > +++ : /etc/ha.d > +++ : /etc/ha.d/rc.d > +++ : /etc/ha.d/conf > +++ : /etc/ha.d/ha.cf > +++ : /var/lib/heartbeat > +++ : /var/run/resource-agents > +++ : /var/run/heartbeat/rsctmp > +++ : /var/lib/heartbeat/fifo > +++ : /usr/lib64/heartbeat > +++ : /usr/sbin > +++ : '%Y/%m/%d_%T ' > +++ : /dev/null > +++ : /etc/ha.d/resource.d > +++ : /usr/share/doc/heartbeat > +++ : oracle > +++ : /var/run/ > +++ : /var/lock/subsys/ > ++ . /usr/lib/ocf/lib/heartbeat/ocf-rarun > ++ : 0 > ++ __ocf_set_defaults start > ++ __OCF_ACTION=start > ++ unset LANG > ++ LC_ALL=C > ++ export LC_ALL > ++ '[' -z '' ']' > ++ : 0 > ++ '[' '!' -d /usr/lib/ocf ']' > ++ '[' -z '' ']' > ++ : oracle > ++ '[' -z '' ']' > ++ : We are being invoked as an init script. > ++ : Fill in some things with reasonable values. > ++ : default > ++ return 0 > ++ : > ++ ocf_is_true '' > ++ case "$1" in > ++ false > + . /usr/lib/ocf/lib/heartbeat/ora-common.sh > + PROCS_CLEANUP_TIME=30 > + MONUSR=OCFMON > + OCF_REQUIRED_PARAMS=sid > + OCF_REQUIRED_BINARIES=sqlplus > + ocf_rarun start > + mk_action_func > ++ echo oracle_start > ++ tr - _ > + ACTION_FUNC=oracle_start > + validate_args > + is_function oracle_start > ++ command -v oracle_start > + test zoracle_start = zoracle_start > + simple_actions > + case $__OCF_ACTION in > + check_required_params > + local v > + for v in '$OCF_REQUIRED_PARAMS' > + is_var_defined OCF_RESKEY_sid > +++ echo OCF_RESKEY_sid > ++ eval echo '$OCF_RESKEY_sid' > +++ echo orcSNBGW > + test z '!=' zorcSNBGW > + run_function oracle_getconfig > + is_function oracle_getconfig > ++ command -v oracle_getconfig > + test zoracle_getconfig = zoracle_getconfig > + oracle_getconfig > + ora_common_getconfig orcSNBGW /opt/oracle/app/product/11.2.0/dbhome_1 > oracle '' > + ORACLE_SID=orcSNBGW > + ORACLE_HOME=/opt/oracle/app/product/11.2.0/dbhome_1 > + ORACLE_OWNER=oracle > + TNS_ADMIN= > + '[' x = x/opt/oracle/app/product/11.2.0/dbhome_1 ']' > + '[' x = xoracle ']' > + '[' x = x ']' > + TNS_ADMIN=/opt/oracle/app/product/11.2.0/dbhome_1/network/admin > + LD_LIBRARY_PATH=/opt/oracle/app/product/11.2.0/dbhome_1/lib > + LIBPATH=/opt/oracle/app/product/11.2.0/dbhome_1/lib > + > PATH=/opt/oracle/app/product/11.2.0/dbhome_1/bin:/opt/oracle/app/product/11.2.0/dbhome_1/dbs:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/ucb > + export ORACLE_SID ORACLE_HOME ORACLE_OWNER TNS_ADMIN > + export LD_LIBRARY_PATH LIBPATH > ++ mktemp > + ORA_ENVF=/tmp/tmp.QJbuqB2dgm > + dumporaenv > + cat > + chmod 644 /tmp/tmp.QJbuqB2dgm > + TMPFILES=/tmp/tmp.QJbuqB2dgm > + trap rmtmpfiles EXIT > + clear_backupmode=false > + shutdown_method=immediate > + IPCRM=instance > + validate_env > + check_required_binaries > + local v > + for v in '$OCF_REQUIRED_BINARIES' > + have_binary sqlplus > + '[' '' = 1 ']' > ++ echo sqlplus > ++ sed -e 's/ -.*//' > + local bin=sqlplus > ++ which sqlplus > + test -x /opt/oracle/app/product/11.2.0/dbhome_1/bin/sqlplus > + is_function oracle_validate_all > ++ command -v oracle_validate_all > + test zoracle_validate_all = zoracle_validate_all > + local rc > + LSB_STATUS_STOPPED=3 > + oracle_validate_all > + case "${shutdown_method}" in > + case "${IPCRM}" in > + ora_common_validate_all > + '[' x = x/opt/oracle/app/product/11.2.0/dbhome_1 ']' > + '[' x = xoracle ']' > ++ id -u -n > + US=root > + '[' root '!=' root -a root '!=' oracle ']' > + return 0 > + rc=0 > + '[' 0 -ne 0 ']' > + ocf_is_probe > + '[' start = monitor -a 0 = 0 ']' > + shift 1 > + oracle_start > + local status output > + is_proc_running > + ps -ef > + grep -wiqs '[^ ]*[_]pmon_orcSNBGW' > ++ dbasql dbstart_mount > ++ runsql 'connect / as sysdba' dbstart_mount > ++ local 'conn_s=connect / as sysdba' > ++ shift 1 > ++ local func > ++ echo 'connect / as sysdba' > ++ execsql > ++ common_sql_opts > ++ cat > ++ '[' root = oracle ']' > ++ su - oracle -s /bin/sh -c '. /tmp/tmp.QJbuqB2dgm; sqlplus -S /nolog' > ++ common_sql_filter > ++ grep -v '^Connected' > ++ grep -v '^ENV MSG:' > ++ for func in '"$@"' > ++ dbstart_mount > ++ echo 'startup mount' > ++ grep -v 'Your password will expire in' > + output='ORACLE instance started. > Total System Global Area 4275781632 bytes > Fixed Size 2260088 bytes > Variable Size 889193352 bytes > Database Buffers 3372220416 bytes > Redo Buffers 12107776 bytes > Database mounted.' > + echo 'ORACLE instance started. > Total System Global Area 4275781632 bytes > Fixed Size 2260088 bytes > Variable Size 889193352 bytes > Database Buffers 3372220416 bytes > Redo Buffers 12107776 bytes > Database mounted.' > + grep ORA-01081 > ++ dbasql_one dbstat > ++ dbasql dbstat > ++ runsql 'connect / as sysdba' dbstat > ++ tail -1 > ++ local 'conn_s=connect / as sysdba' > ++ shift 1 > ++ local func > ++ echo 'connect / as sysdba' > ++ execsql > ++ common_sql_opts > ++ cat > ++ '[' root = oracle ']' > ++ su - oracle -s /bin/sh -c '. /tmp/tmp.QJbuqB2dgm; sqlplus -S /nolog' > ++ common_sql_filter > ++ grep -v '^Connected' > ++ grep -v 'Your password will expire in' > ++ grep -v '^ENV MSG:' > ++ for func in '"$@"' > ++ dbstat > ++ echo 'select status from v$instance;' > + status=MOUNTED > + case "$status" in > + is_clear_backupmode_set > + '[' xfalse = xtrue ']' > ++ dbasql dbopen > ++ runsql 'connect / as sysdba' dbopen > ++ local 'conn_s=connect / as sysdba' > ++ shift 1 > ++ local func > ++ echo 'connect / as sysdba' > ++ execsql > ++ common_sql_opts > ++ common_sql_filter > ++ cat > ++ '[' root = oracle ']' > ++ su - oracle -s /bin/sh -c '. /tmp/tmp.QJbuqB2dgm; sqlplus -S /nolog' > ++ grep -v '^Connected' > ++ grep -v 'Your password will expire in' > ++ grep -v '^ENV MSG:' > ++ for func in '"$@"' > ++ dbopen > ++ echo 'alter database open;' > + output= > + check_mon_user > + local output > + dbasql show_mon_user > + runsql 'connect / as sysdba' show_mon_user > + local 'conn_s=connect / as sysdba' > + shift 1 > + local func > + echo 'connect / as sysdba' > + common_sql_opts > + cat > + grep -w '^OCFMON' > + common_sql_filter > + grep -v 'Your password will expire in' > + grep -v '^Connected' > + for func in '"$@"' > + show_mon_user > + echo 'select USERNAME, ACCOUNT_STATUS from dba_users where > USERNAME='\''OCFMON'\'';' > + execsql > + '[' root = oracle ']' > + su - oracle -s /bin/sh -c '. /tmp/tmp.QJbuqB2dgm; sqlplus -S /nolog' > + grep -v '^ENV MSG:' > + return 0 > + is_proc_running > + ps -ef > + grep -wiqs '[^ ]*[_]pmon_orcSNBGW' > + instance_live > ++ monsql_one dbstat > ++ monsql dbstat > ++ runsql 'connect OCFMON/OCFMON' dbstat > ++ tail -1 > ++ local 'conn_s=connect OCFMON/OCFMON' > ++ shift 1 > ++ local func > ++ echo 'connect OCFMON/OCFMON' > ++ execsql > ++ common_sql_opts > ++ '[' root = oracle ']' > ++ cat > ++ su - oracle -s /bin/sh -c '. /tmp/tmp.QJbuqB2dgm; sqlplus -S /nolog' > ++ common_sql_filter > ++ for func in '"$@"' > ++ dbstat > ++ echo 'select status from v$instance;' > ++ grep -v '^ENV MSG:' > ++ grep -v 'Your password will expire in' > ++ grep -v '^Connected' > > + local 'status=SP2-0640: Not connected' > + '[' 'SP2-0640: Not connected' = OPEN ']' > + ocf_log info 'orcSNBGW instance state is not OPEN (dbstat output: SP2-0640: > Not connected)' > + '[' 2 -lt 2 ']' > + __OCF_PRIO=info > + shift > + __OCF_MSG='orcSNBGW instance state is not OPEN (dbstat output: SP2-0640: > Not connected)' > + case "${__OCF_PRIO}" in > + __OCF_PRIO=INFO > + '[' INFO = DEBUG ']' > + ha_log 'INFO: orcSNBGW instance state is not OPEN (dbstat output: SP2-0640: > Not connected)' > + local loglevel > + '[' none = '' ']' > + tty > + '[' x = x0 -a x = xdebug ']' > + '[' '' ']' > + echo 'INFO: orcSNBGW instance state is not OPEN (dbstat output: SP2-0640: > Not connected)' > INFO: orcSNBGW instance state is not OPEN (dbstat output: SP2-0640: Not > connected) > + return 0 > + return 1 > + ocf_log err 'oracle instance orcSNBGW not started: ' > + '[' 2 -lt 2 ']' > + __OCF_PRIO=err > + shift > + __OCF_MSG='oracle instance orcSNBGW not started: ' > + case "${__OCF_PRIO}" in > + __OCF_PRIO=ERROR > + '[' ERROR = DEBUG ']' > + ha_log 'ERROR: oracle instance orcSNBGW not started: ' > + local loglevel > + '[' none = '' ']' > + tty > + '[' x = x0 -a x = xdebug ']' > + '[' '' ']' > + echo 'ERROR: oracle instance orcSNBGW not started: ' > ERROR: oracle instance orcSNBGW not started: > + return 0 > + return 1 > + rmtmpfiles > + rm -f /tmp/tmp.QJbuqB2dgm > > > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
