[
https://issues.apache.org/jira/browse/AMBARI-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15135345#comment-15135345
]
Alejandro Fernandez commented on AMBARI-14950:
----------------------------------------------
Actually, looks like it's related to prepare-war and starting oozie server.
{code}
| 2 | DOWNGRADE | 1965 | Oozie | Preparing
Oozie Server on os-r6-tcdlus-dalm20tom21secdwngd-5 | COMPLETED |
os-r6-tcdlus-dalm20tom21secdwngd-5.novalocal |
| 2 | DOWNGRADE | 1966 | Oozie |
Restarting Oozie Server on 3 hosts | COMPLETED
| os-r6-tcdlus-dalm20tom21secdwngd-3.novalocal |
| 2 | DOWNGRADE | 1967 | Oozie |
Restarting Oozie Server on 3 hosts | COMPLETED
| os-r6-tcdlus-dalm20tom21secdwngd-4.novalocal |
| 2 | DOWNGRADE | 1968 | Oozie |
Restarting Oozie Server on 3 hosts | COMPLETED
| os-r6-tcdlus-dalm20tom21secdwngd-5.novalocal |
{code}
On host 4, it shows this for output-1967.txt
{code}
2016-02-03 04:04:52,973 - Execute['cd /var/tmp/oozie &&
/usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war '] {'not_if': ...,
'user': 'cstm-oozie'}
2016-02-03 04:04:53,031 - Skipping Execute['cd /var/tmp/oozie &&
/usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war '] due to not_if
2016-02-03 04:04:53,032 - File['/usr/hdp/current/oozie-server/.hashcode']
{'content': '046a880c90fcbbfea52bec80cb88dd8f', 'mode': 0644}
2016-02-03 04:04:53,129 -
File['/usr/hdp/current/oozie-server/.prepare_war_cmd'] {'content': 'cd
/var/tmp/oozie && /usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war
', 'mode': 0644}
...
######## Hortonworks #############
This is MOTD message, added for testing in qe infra
2016-02-03 04:04:54,089 -
call['/usr/hdp/current/oozie-server/bin/oozie-setup.sh prepare-war -d
/usr/hdp/current/oozie-server/libext'] {'logoutput': False, 'user':
'cstm-oozie', 'env': {'CATALINA_BASE':
'/usr/hdp/current/oozie-server/oozie-server', 'OOZIE_HOME':
'/usr/hdp/current/oozie-server'}, 'quiet': False}
2016-02-03 04:05:08,413 - call returned (0, '######## Hortonworks #############
This is MOTD message, added for testing in qe infra
setting OOZIE_CONFIG=${OOZIE_CONFIG:-/usr/hdp/current/oozie-client/conf}
setting
CATALINA_BASE=${CATALINA_BASE:-/usr/hdp/current/oozie-client/oozie-server}
setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}
setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
setting JAVA_HOME=/usr/jdk64/jdk1.7.0_67
setting JRE_HOME=${JAVA_HOME}
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx2048m -XX:MaxPermSize=256m"
setting OOZIE_LOG=/grid/0/log/oozie
setting CATALINA_PID=/var/run/oozie/oozie.pid
setting OOZIE_DATA=/grid/0/hadoop/oozie/data
setting OOZIE_HTTP_PORT=11000
setting OOZIE_ADMIN_PORT=11001
setting JAVA_LIBRARY_PATH=/usr/hdp/2.4.0.0-151/hadoop/lib/native/Linux-amd64-64
setting OOZIE_CLIENT_OPTS="${OOZIE_CLIENT_OPTS}
-Doozie.connection.retry.count=5 "
setting OOZIE_CONFIG=${OOZIE_CONFIG:-/usr/hdp/current/oozie-client/conf}
setting
CATALINA_BASE=${CATALINA_BASE:-/usr/hdp/current/oozie-client/oozie-server}
setting CATALINA_TMPDIR=${CATALINA_TMPDIR:-/var/tmp/oozie}
setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat
setting JAVA_HOME=/usr/jdk64/jdk1.7.0_67\n setting JRE_HOME=${JAVA_HOME}
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx2048m -XX:MaxPermSize=256m"
setting OOZIE_LOG=/grid/0/log/oozie
setting CATALINA_PID=/var/run/oozie/oozie.pid
setting OOZIE_DATA=/grid/0/hadoop/oozie/data
setting OOZIE_HTTP_PORT=11000
setting OOZIE_ADMIN_PORT=11001
setting JAVA_LIBRARY_PATH=/usr/hdp/2.4.0.0-151/hadoop/lib/native/Linux-amd64-64
setting OOZIE_CLIENT_OPTS="${OOZIE_CLIENT_OPTS}
-Doozie.connection.retry.count=5 "
INFO: Adding extension:
/usr/hdp/current/oozie-server/libext/falcon-oozie-el-extension-0.6.1.2.3.4.0-3485.jar
INFO: Adding extension:
/usr/hdp/current/oozie-server/libext/hadoop-lzo-0.6.0.2.3.4.0-3485.jar
INFO: Adding extension:
/usr/hdp/current/oozie-server/libext/hadoop-lzo-0.6.0.2.3.4.0-3485-sources.jar
INFO: Adding extension:
/usr/hdp/current/oozie-server/libext/hadoop-lzo-0.6.0.2.3.4.0-3485-test.jar
INFO: Adding extension:
/usr/hdp/current/oozie-server/libext/mysql-connector-java.jar
New Oozie WAR file with added \'ExtJS library, JARs\' at
/usr/hdp/current/oozie-server/oozie-server/webapps/oozie.war
INFO: Oozie is ready to be started')
2016-02-03 04:05:08,415 - Execute['cd /var/tmp/oozie &&
/usr/hdp/current/oozie-server/bin/oozie-start.sh'] {'environment':
{'OOZIE_CONFIG': '/usr/hdp/current/oozie-server/conf'}, 'not_if':
"ambari-sudo.sh su cstm-oozie -l -s /bin/bash -c 'ls /var/run/oozie/oozie.pid
>/dev/null 2>&1 && ps -p `cat /var/run/oozie/oozie.pid` >/dev/null 2>&1'",
'user': 'cstm-oozie'}
{code}
The timestamp of /usr/hdp/current/oozie-server/oozie-server/webapps/oozie.war
shows it is from Feb 3 04:05, which is right when the oozie servers were all
restarted during downgrade.
However, the /usr/hdp/current/oozie-server/oozie-server/webapps/oozie folder
was only present after I restarted Oozie Server manually.
> EU: Oozie service check failed after downgrade with multiple Oozie servers
> since rerunning prepare-war removes webapps/oozie folder
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: AMBARI-14950
> URL: https://issues.apache.org/jira/browse/AMBARI-14950
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.2.1
> Reporter: Alejandro Fernandez
> Assignee: Alejandro Fernandez
> Fix For: 2.2.2
>
>
> *Steps*
> # Setup HDP 2.3.4.0 cluster with Ambari 2.2.1 (secure, non-HA cluster)
> # Start Express Upgrade to HDP-2.4.0.0-149 and reach till Finalize step
> # Perform Downgrade back to original version and let it complete
> # Run service check for Oozie
> *Result*
> Service check failed with error:
> {code}
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of
> '/var/lib/ambari-agent/tmp/oozieSmoke2.sh redhat
> /usr/hdp/current/oozie-client /usr/hdp/current/oozie-client/conf
> /usr/hdp/current/oozie-client/bin
> http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie
> /usr/hdp/current/oozie-client/doc /usr/hdp/current/hadoop-client/conf
> /usr/hdp/current/hadoop-client/bin ambari-qa True
> /etc/security/keytabs/smokeuser.headless.keytab /usr/bin/kinit
> [email protected]' returned 1.
> /usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab
> [email protected]; source
> /usr/hdp/current/oozie-client/conf/oozie-env.sh ;
> /usr/hdp/current/oozie-client/bin/oozie -Doozie.auth.token.cache=false job
> -oozie http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie -config
> /usr/hdp/current/oozie-client/doc/examples/apps/map-reduce/job.properties -run
> Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server.
> No of retries = 1. Exception = Could not authenticate, Authentication failed,
> status: 404, message: Not Found
> Invalid sub-command: Missing argument for option: info
> use 'help [sub-command]' for help details
> Invalid sub-command: Missing argument for option: info
> use 'help [sub-command]' for help details
> workflow_status=
> {code}
> catalina.out prints below
> {code}
> Jan 30, 2016 10:43:13 PM org.apache.catalina.core.StandardContext
> resourcesStart
> SEVERE: Error starting static Resources
> java.lang.IllegalArgumentException: Document base
> /usr/hdp/current/oozie-client/oozie-server/webapps/oozie does not exist or is
> not a readable directory
> at
> org.apache.naming.resources.FileDirContext.setDocBase(FileDirContext.java:142)
> at
> org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4324)
> Jan 30, 2016 10:43:31 PM org.apache.catalina.loader.WebappClassLoader
> loadClass
> INFO: Illegal access: this web application instance has been stopped already.
> Could not load com.google.common.cache.RemovalCause. The eventual following
> stack trace is caused by an error thrown for debugging purposes as well as to
> attempt to terminate the thread which caused the illegal access, and has no
> functional impact.
> java.lang.IllegalStateException
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
> {code}
> Attached complete logs
> *Note - I did a restart of Oozie service and the issue is no longer seen*
> Apparently /usr/hdp/current/oozie-client/oozie-server/webapps/oozie got
> created as part of Oozie service restart
> {code}
> ls -lhrt /usr/hdp/2.3.4.0-3485/oozie/oozie-server/webapps/
> total 136M
> lrwxrwxrwx. 1 oozie hadoop 31 Jan 30 19:26 docs ->
> /usr/hdp/2.3.4.0-3485/oozie/doc
> lrwxrwxrwx. 1 oozie hadoop 42 Jan 30 19:26 ext-2.2 ->
> /usr/hdp/2.3.4.0-3485/oozie/libext/ext-2.2
> -rw-r-----. 1 oozie hadoop 136M Jan 30 22:42 oozie.war
> *drwxr-x---. 8 oozie hadoop 4.0K Feb 1 06:27 oozie*
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
