[
https://issues.apache.org/jira/browse/AMBARI-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Fernandez updated AMBARI-14950:
-----------------------------------------
Description:
*Steps*
# Setup HDP 2.3.4.0 cluster with Ambari 2.2.1 (secure, multiple oozie servers)
# Start Express Upgrade to HDP-2.4.0.0-149 and reach till Finalize step
# Perform Downgrade back to original version and let it complete
# Run service check for Oozie
*Result*
Service check failed with error:
{code}
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of
'/var/lib/ambari-agent/tmp/oozieSmoke2.sh redhat /usr/hdp/current/oozie-client
/usr/hdp/current/oozie-client/conf /usr/hdp/current/oozie-client/bin
http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie
/usr/hdp/current/oozie-client/doc /usr/hdp/current/hadoop-client/conf
/usr/hdp/current/hadoop-client/bin ambari-qa True
/etc/security/keytabs/smokeuser.headless.keytab /usr/bin/kinit
[email protected]' returned 1. /usr/bin/kinit
-kt /etc/security/keytabs/smokeuser.headless.keytab
[email protected]; source
/usr/hdp/current/oozie-client/conf/oozie-env.sh ;
/usr/hdp/current/oozie-client/bin/oozie -Doozie.auth.token.cache=false job
-oozie http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie -config
/usr/hdp/current/oozie-client/doc/examples/apps/map-reduce/job.properties -run
Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No
of retries = 1. Exception = Could not authenticate, Authentication failed,
status: 404, message: Not Found
Invalid sub-command: Missing argument for option: info
use 'help [sub-command]' for help details
Invalid sub-command: Missing argument for option: info
use 'help [sub-command]' for help details
workflow_status=
{code}
catalina.out prints below
{code}
Jan 30, 2016 10:43:13 PM org.apache.catalina.core.StandardContext resourcesStart
SEVERE: Error starting static Resources
java.lang.IllegalArgumentException: Document base
/usr/hdp/current/oozie-client/oozie-server/webapps/oozie does not exist or is
not a readable directory
at
org.apache.naming.resources.FileDirContext.setDocBase(FileDirContext.java:142)
at
org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4324)
Jan 30, 2016 10:43:31 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.
Could not load com.google.common.cache.RemovalCause. The eventual following
stack trace is caused by an error thrown for debugging purposes as well as to
attempt to terminate the thread which caused the illegal access, and has no
functional impact.
java.lang.IllegalStateException
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
{code}
Attached complete logs
*Note - I did a restart of Oozie service and the issue is no longer seen*
Apparently /usr/hdp/current/oozie-client/oozie-server/webapps/oozie got created
as part of Oozie service restart
{code}
ls -lhrt /usr/hdp/2.3.4.0-3485/oozie/oozie-server/webapps/
total 136M
lrwxrwxrwx. 1 oozie hadoop 31 Jan 30 19:26 docs ->
/usr/hdp/2.3.4.0-3485/oozie/doc
lrwxrwxrwx. 1 oozie hadoop 42 Jan 30 19:26 ext-2.2 ->
/usr/hdp/2.3.4.0-3485/oozie/libext/ext-2.2
-rw-r-----. 1 oozie hadoop 136M Jan 30 22:42 oozie.war
*drwxr-x---. 8 oozie hadoop 4.0K Feb 1 06:27 oozie*
{code}
*Running the prepare-war command multiple times causes it to remove the
/usr/hdp/current/oozie-server/oozie-server/webapps/oozie folder.*
Further, there's a bug that the /usr/hdp/current/oozie-server/.prepare_war_cmd
marker file is still generated even if the prepare war command was not ran!
was:
*Steps*
# Setup HDP 2.3.4.0 cluster with Ambari 2.2.1 (secure, non-HA cluster)
# Start Express Upgrade to HDP-2.4.0.0-149 and reach till Finalize step
# Perform Downgrade back to original version and let it complete
# Run service check for Oozie
*Result*
Service check failed with error:
{code}
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of
'/var/lib/ambari-agent/tmp/oozieSmoke2.sh redhat /usr/hdp/current/oozie-client
/usr/hdp/current/oozie-client/conf /usr/hdp/current/oozie-client/bin
http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie
/usr/hdp/current/oozie-client/doc /usr/hdp/current/hadoop-client/conf
/usr/hdp/current/hadoop-client/bin ambari-qa True
/etc/security/keytabs/smokeuser.headless.keytab /usr/bin/kinit
[email protected]' returned 1. /usr/bin/kinit
-kt /etc/security/keytabs/smokeuser.headless.keytab
[email protected]; source
/usr/hdp/current/oozie-client/conf/oozie-env.sh ;
/usr/hdp/current/oozie-client/bin/oozie -Doozie.auth.token.cache=false job
-oozie http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie -config
/usr/hdp/current/oozie-client/doc/examples/apps/map-reduce/job.properties -run
Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No
of retries = 1. Exception = Could not authenticate, Authentication failed,
status: 404, message: Not Found
Invalid sub-command: Missing argument for option: info
use 'help [sub-command]' for help details
Invalid sub-command: Missing argument for option: info
use 'help [sub-command]' for help details
workflow_status=
{code}
catalina.out prints below
{code}
Jan 30, 2016 10:43:13 PM org.apache.catalina.core.StandardContext resourcesStart
SEVERE: Error starting static Resources
java.lang.IllegalArgumentException: Document base
/usr/hdp/current/oozie-client/oozie-server/webapps/oozie does not exist or is
not a readable directory
at
org.apache.naming.resources.FileDirContext.setDocBase(FileDirContext.java:142)
at
org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4324)
Jan 30, 2016 10:43:31 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.
Could not load com.google.common.cache.RemovalCause. The eventual following
stack trace is caused by an error thrown for debugging purposes as well as to
attempt to terminate the thread which caused the illegal access, and has no
functional impact.
java.lang.IllegalStateException
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
{code}
Attached complete logs
*Note - I did a restart of Oozie service and the issue is no longer seen*
Apparently /usr/hdp/current/oozie-client/oozie-server/webapps/oozie got created
as part of Oozie service restart
{code}
ls -lhrt /usr/hdp/2.3.4.0-3485/oozie/oozie-server/webapps/
total 136M
lrwxrwxrwx. 1 oozie hadoop 31 Jan 30 19:26 docs ->
/usr/hdp/2.3.4.0-3485/oozie/doc
lrwxrwxrwx. 1 oozie hadoop 42 Jan 30 19:26 ext-2.2 ->
/usr/hdp/2.3.4.0-3485/oozie/libext/ext-2.2
-rw-r-----. 1 oozie hadoop 136M Jan 30 22:42 oozie.war
*drwxr-x---. 8 oozie hadoop 4.0K Feb 1 06:27 oozie*
{code}
*Running the prepare-war command multiple times causes it to remove the
/usr/hdp/current/oozie-server/oozie-server/webapps/oozie folder.*
Further, there's a bug that the /usr/hdp/current/oozie-server/.prepare_war_cmd
marker file is still generated even if the prepare war command was not ran!
> EU: Oozie service check failed after downgrade with multiple Oozie servers
> since rerunning prepare-war removes webapps/oozie folder
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: AMBARI-14950
> URL: https://issues.apache.org/jira/browse/AMBARI-14950
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.2.1
> Reporter: Alejandro Fernandez
> Assignee: Alejandro Fernandez
> Fix For: 2.2.2
>
> Attachments: AMBARI-14950.trunk.patch
>
>
> *Steps*
> # Setup HDP 2.3.4.0 cluster with Ambari 2.2.1 (secure, multiple oozie servers)
> # Start Express Upgrade to HDP-2.4.0.0-149 and reach till Finalize step
> # Perform Downgrade back to original version and let it complete
> # Run service check for Oozie
> *Result*
> Service check failed with error:
> {code}
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of
> '/var/lib/ambari-agent/tmp/oozieSmoke2.sh redhat
> /usr/hdp/current/oozie-client /usr/hdp/current/oozie-client/conf
> /usr/hdp/current/oozie-client/bin
> http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie
> /usr/hdp/current/oozie-client/doc /usr/hdp/current/hadoop-client/conf
> /usr/hdp/current/hadoop-client/bin ambari-qa True
> /etc/security/keytabs/smokeuser.headless.keytab /usr/bin/kinit
> [email protected]' returned 1.
> /usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab
> [email protected]; source
> /usr/hdp/current/oozie-client/conf/oozie-env.sh ;
> /usr/hdp/current/oozie-client/bin/oozie -Doozie.auth.token.cache=false job
> -oozie http://os-r6-yyeqtu-ambari-se-eu-7-re-3.novalocal:11000/oozie -config
> /usr/hdp/current/oozie-client/doc/examples/apps/map-reduce/job.properties -run
> Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server.
> No of retries = 1. Exception = Could not authenticate, Authentication failed,
> status: 404, message: Not Found
> Invalid sub-command: Missing argument for option: info
> use 'help [sub-command]' for help details
> Invalid sub-command: Missing argument for option: info
> use 'help [sub-command]' for help details
> workflow_status=
> {code}
> catalina.out prints below
> {code}
> Jan 30, 2016 10:43:13 PM org.apache.catalina.core.StandardContext
> resourcesStart
> SEVERE: Error starting static Resources
> java.lang.IllegalArgumentException: Document base
> /usr/hdp/current/oozie-client/oozie-server/webapps/oozie does not exist or is
> not a readable directory
> at
> org.apache.naming.resources.FileDirContext.setDocBase(FileDirContext.java:142)
> at
> org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4324)
> Jan 30, 2016 10:43:31 PM org.apache.catalina.loader.WebappClassLoader
> loadClass
> INFO: Illegal access: this web application instance has been stopped already.
> Could not load com.google.common.cache.RemovalCause. The eventual following
> stack trace is caused by an error thrown for debugging purposes as well as to
> attempt to terminate the thread which caused the illegal access, and has no
> functional impact.
> java.lang.IllegalStateException
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
> {code}
> Attached complete logs
> *Note - I did a restart of Oozie service and the issue is no longer seen*
> Apparently /usr/hdp/current/oozie-client/oozie-server/webapps/oozie got
> created as part of Oozie service restart
> {code}
> ls -lhrt /usr/hdp/2.3.4.0-3485/oozie/oozie-server/webapps/
> total 136M
> lrwxrwxrwx. 1 oozie hadoop 31 Jan 30 19:26 docs ->
> /usr/hdp/2.3.4.0-3485/oozie/doc
> lrwxrwxrwx. 1 oozie hadoop 42 Jan 30 19:26 ext-2.2 ->
> /usr/hdp/2.3.4.0-3485/oozie/libext/ext-2.2
> -rw-r-----. 1 oozie hadoop 136M Jan 30 22:42 oozie.war
> *drwxr-x---. 8 oozie hadoop 4.0K Feb 1 06:27 oozie*
> {code}
> *Running the prepare-war command multiple times causes it to remove the
> /usr/hdp/current/oozie-server/oozie-server/webapps/oozie folder.*
> Further, there's a bug that the
> /usr/hdp/current/oozie-server/.prepare_war_cmd marker file is still generated
> even if the prepare war command was not ran!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
