[
https://issues.apache.org/jira/browse/AMBARI-14779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Fernandez updated AMBARI-14779:
-----------------------------------------
Description:
*Steps*
# Install HDP 2.1 cluster with ambari 2.2.1 (HFDS, YARN, ZK, Hive)
# Kerberize the cluster, this will remove ATS
# Bootstrap the cluster with versions
# Install HDP 2.3 and perform EU
# Run Hive service check after EU is complete
*Result:*
Failed with below error
{code}
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt
/etc/security/keytabs/smokeuser.headless.keytab [email protected]; env
JAVA_HOME=/usr/jdk64/jdk1.7.0_67 /var/lib/ambari-agent/tmp/hcatSmoke.sh
hcatsmokeid16acce51_date582116 prepare true' returned 12. WARNING: Use "yarn
jar" to launch YARN applications.
16/01/21 16:59:51 WARN conf.HiveConf: HiveConf of name
hive.optimize.mapjoin.mapreduce does not exist
16/01/21 16:59:51 WARN conf.HiveConf: HiveConf of name hive.heapsize does not
exist
16/01/21 16:59:51 WARN conf.HiveConf: HiveConf of name
hive.server2.enable.impersonation does not exist
16/01/21 16:59:51 WARN conf.HiveConf: HiveConf of name
hive.auto.convert.sortmerge.join.noconditionaltask does not exist
ivysettings.xml file not found in HIVE_HOME or
HIVE_CONF_DIR,file:/grid/0/hdp/2.3.4.0-3485/hadoop/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-sources.jar!/ivysettings.xml
will be used
FAILED: Hive Internal Error:
org.apache.hadoop.service.ServiceStateException(java.io.IOException:
/ats/active does not exist)
org.apache.hadoop.service.ServiceStateException: java.io.IOException:
/ats/active does not exist
at
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
at org.apache.hadoop.hive.ql.hooks.ATSHook.<init>(ATSHook.java:74)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:374)
at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:60)
at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1351)
at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1335)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1395)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1237)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1091)
at org.apache.hive.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)
at org.apache.hive.hcatalog.cli.HCatCli.processCmd(HCatCli.java:291)
at org.apache.hive.hcatalog.cli.HCatCli.processLine(HCatCli.java:245)
at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:183)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: /ats/active does not exist
at
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceStart(TimelineClientImpl.java:371)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
... 23 more
{code}
In HDP 2.1, YARN ATS v1.0 did not support kerberos, so kerberizing the cluster
removed that component. This means that EU from HDP 2.1 to 2.3 cannot persist
the ATS v1.0 configs and instead adds the ATS v1.5 configs that default to
stack HDP 2.3. For that reason, Hive Service Check fails.
The fix is to persist the ATS v1.0 configs even if ATS is not present. The
service checks will still fail, but the user can skip then during EU. After EU
is done, they will have to re-add ATS and setup keytabs.
was:
*Steps*
# Install HDP 2.1 cluster with ambari 2.2.1 (HFDS, YARN, ZK, Hive)
# Kerberize the cluster, this will remove ATS
# Bootstrap the cluster with versions
# Install HDP 2.3 and perform EU
# Run Hive service check after EU is complete
*Result:*
Failed with below error
{code}
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt
/etc/security/keytabs/smokeuser.headless.keytab [email protected]; env
JAVA_HOME=/usr/jdk64/jdk1.7.0_67 /var/lib/ambari-agent/tmp/hcatSmoke.sh
hcatsmokeid16acce51_date582116 prepare true' returned 12. WARNING: Use "yarn
jar" to launch YARN applications.
16/01/21 16:59:51 WARN conf.HiveConf: HiveConf of name
hive.optimize.mapjoin.mapreduce does not exist
16/01/21 16:59:51 WARN conf.HiveConf: HiveConf of name hive.heapsize does not
exist
16/01/21 16:59:51 WARN conf.HiveConf: HiveConf of name
hive.server2.enable.impersonation does not exist
16/01/21 16:59:51 WARN conf.HiveConf: HiveConf of name
hive.auto.convert.sortmerge.join.noconditionaltask does not exist
ivysettings.xml file not found in HIVE_HOME or
HIVE_CONF_DIR,file:/grid/0/hdp/2.3.4.0-3485/hadoop/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-sources.jar!/ivysettings.xml
will be used
FAILED: Hive Internal Error:
org.apache.hadoop.service.ServiceStateException(java.io.IOException:
/ats/active does not exist)
org.apache.hadoop.service.ServiceStateException: java.io.IOException:
/ats/active does not exist
at
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
at org.apache.hadoop.hive.ql.hooks.ATSHook.<init>(ATSHook.java:74)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:374)
at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:60)
at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1351)
at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1335)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1395)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1237)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1091)
at org.apache.hive.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)
at org.apache.hive.hcatalog.cli.HCatCli.processCmd(HCatCli.java:291)
at org.apache.hive.hcatalog.cli.HCatCli.processLine(HCatCli.java:245)
at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:183)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: /ats/active does not exist
at
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceStart(TimelineClientImpl.java:371)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
... 23 more
{code}
In HDP 2.1, YARN ATS v1.0 did not support kerberos, so kerberizing the cluster
removed that component. This means that EU from HDP 2.1 to 2.3 cannot persist
the ATS v1.0 configs and instead adds the ATS v1.5 configs that default to
stack HDP 2.3. For that reason, Hive Service Check fails.
The fix is to persist the ATS v1.0 configs even if ATS is not present. The
service checks will still fail, but the user can then add ATS after EU is done.
> Hive service check failed after EU from HDP 2.1 to 2.3 since ATS is missing
> in Kerberized cluster and configs are wrong
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: AMBARI-14779
> URL: https://issues.apache.org/jira/browse/AMBARI-14779
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
> Affects Versions: 2.2.0
> Reporter: Alejandro Fernandez
> Assignee: Alejandro Fernandez
> Fix For: 2.2.1
>
> Attachments: AMBARI-14779.trunk.patch
>
>
> *Steps*
> # Install HDP 2.1 cluster with ambari 2.2.1 (HFDS, YARN, ZK, Hive)
> # Kerberize the cluster, this will remove ATS
> # Bootstrap the cluster with versions
> # Install HDP 2.3 and perform EU
> # Run Hive service check after EU is complete
> *Result:*
> Failed with below error
> {code}
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of '/usr/bin/kinit -kt
> /etc/security/keytabs/smokeuser.headless.keytab [email protected]; env
> JAVA_HOME=/usr/jdk64/jdk1.7.0_67 /var/lib/ambari-agent/tmp/hcatSmoke.sh
> hcatsmokeid16acce51_date582116 prepare true' returned 12. WARNING: Use "yarn
> jar" to launch YARN applications.
> 16/01/21 16:59:51 WARN conf.HiveConf: HiveConf of name
> hive.optimize.mapjoin.mapreduce does not exist
> 16/01/21 16:59:51 WARN conf.HiveConf: HiveConf of name hive.heapsize does not
> exist
> 16/01/21 16:59:51 WARN conf.HiveConf: HiveConf of name
> hive.server2.enable.impersonation does not exist
> 16/01/21 16:59:51 WARN conf.HiveConf: HiveConf of name
> hive.auto.convert.sortmerge.join.noconditionaltask does not exist
> ivysettings.xml file not found in HIVE_HOME or
> HIVE_CONF_DIR,file:/grid/0/hdp/2.3.4.0-3485/hadoop/lib/hadoop-lzo-0.6.0.2.3.4.0-3485-sources.jar!/ivysettings.xml
> will be used
> FAILED: Hive Internal Error:
> org.apache.hadoop.service.ServiceStateException(java.io.IOException:
> /ats/active does not exist)
> org.apache.hadoop.service.ServiceStateException: java.io.IOException:
> /ats/active does not exist
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
> at org.apache.hadoop.hive.ql.hooks.ATSHook.<init>(ATSHook.java:74)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at java.lang.Class.newInstance(Class.java:374)
> at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:60)
> at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1351)
> at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1335)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1395)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1101)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1091)
> at org.apache.hive.hcatalog.cli.HCatDriver.run(HCatDriver.java:43)
> at org.apache.hive.hcatalog.cli.HCatCli.processCmd(HCatCli.java:291)
> at org.apache.hive.hcatalog.cli.HCatCli.processLine(HCatCli.java:245)
> at org.apache.hive.hcatalog.cli.HCatCli.main(HCatCli.java:183)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.io.IOException: /ats/active does not exist
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceStart(TimelineClientImpl.java:371)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> ... 23 more
> {code}
> In HDP 2.1, YARN ATS v1.0 did not support kerberos, so kerberizing the
> cluster removed that component. This means that EU from HDP 2.1 to 2.3 cannot
> persist the ATS v1.0 configs and instead adds the ATS v1.5 configs that
> default to stack HDP 2.3. For that reason, Hive Service Check fails.
> The fix is to persist the ATS v1.0 configs even if ATS is not present. The
> service checks will still fail, but the user can skip then during EU. After
> EU is done, they will have to re-add ATS and setup keytabs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
