[
https://issues.apache.org/jira/browse/AMBARI-24587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Wagle updated AMBARI-24587:
-------------------------------------
Description:
Livy2 restart fails from Ambari due to Ambari could not fetch some Hadoop
configs?
StdErr:
{code}
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/livy2_server.py",
line 148, in <module>
LivyServer().execute()
File
"/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py",
line 351, in execute
method(env)
File
"/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/livy2_server.py",
line 62, in start
self.configure(env)
File
"/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/livy2_server.py",
line 52, in configure
setup_livy(env, 'server', upgrade_type=upgrade_type, action = 'config')
File
"/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/setup_livy2.py",
line 53, in setup_livy
params.HdfsResource(None, action="execute")
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line
166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py",
line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py",
line 124, in run_action
provider_action()
File
"/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py",
line 681, in action_execute
self.get_hdfs_resource_executor().action_execute(self)
File
"/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py",
line 164, in action_execute
logoutput=logoutput,
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line
166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py",
line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py",
line 124, in run_action
provider_action()
File
"/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line
263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
102, in checked_call
tries=tries, try_sleep=try_sleep,
timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
314, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'hadoop
--config /usr/hdp/3.0.1.0-175/hadoop/conf jar
/var/lib/ambari-agent/lib/fast-hdfs-resource.jar
/var/lib/ambari-agent/tmp/hdfs_resources_1535895647.58.json' returned 1.
Initializing filesystem uri: hdfs://mycluster
Creating: Resource [source=null,
target=wasb://spark2l-at30wu-livy2-121...@humbtesting5wua2.blob.core.windows.net/user/livy/recovery-69e01f25-3b7b-4af4-a787-37664ab45f0c,
type=directory, action=create, owner=livy, group=null, mode=700,
recursiveChown=false, recursiveChmod=false, changePermissionforParents=false,
manageIfExists=true] in hdfs://mycluster
Exception occurred, Reason: Wrong FS:
wasb://spark2l-at30wu-livy2-121...@humbtesting5wua2.blob.core.windows.net/user/livy/recovery-69e01f25-3b7b-4af4-a787-37664ab45f0c,
expected: hdfs://mycluster
java.lang.IllegalArgumentException: Wrong FS:
wasb://spark2l-at30wu-livy2-121...@humbtesting5wua2.blob.core.windows.net/user/livy/recovery-69e01f25-3b7b-4af4-a787-37664ab45f0c,
expected: hdfs://mycluster
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:240)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1583)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1595)
at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1768)
at
org.apache.ambari.fast_hdfs_resource.Resource.checkResourceParameters(Resource.java:193)
at org.apache.ambari.fast_hdfs_resource.Runner.main(Runner.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
{code}
Live cluster:
<https://spark2l-at30wu-
livy2-121012.southeastasia.cloudapp.azure.com/#/main/services/SPARK2/summary>
Please note that the a system test sets the
property | from | to
---|---|---
`livy.server.recovery.state-store` | `zookeeper` | `filesystem`
`livy.server.recovery.state-store.url` |
`zk1-b24996.zu2zfpuge4su5ceknrmkpsq3ra.ix.internal.cloudapp.net:2181,zk2-b24996.zu2zfpuge4su5ceknrmkpsq3ra.ix.internal.cloudapp.net:2181,zk5-b24996.zu2zfpuge4su5ceknrmkpsq3ra.ix.internal.cloudapp.net:2181`
| `wasb://spark2l-at30wu-
livy2-121...@humbtesting5wua2.blob.core.windows.net/user/livy/recovery-
69e01f25-3b7b-4af4-a787-37664ab45f0c`
and then the restart fails.
was:
Livy2 restart fails from Ambari due to Ambari could not fetch some Hadoop
configs?
StdErr:
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/livy2_server.py",
line 148, in <module>
LivyServer().execute()
File
"/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py",
line 351, in execute
method(env)
File
"/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/livy2_server.py",
line 62, in start
self.configure(env)
File
"/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/livy2_server.py",
line 52, in configure
setup_livy(env, 'server', upgrade_type=upgrade_type, action = 'config')
File
"/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/setup_livy2.py",
line 53, in setup_livy
params.HdfsResource(None, action="execute")
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line
166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py",
line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py",
line 124, in run_action
provider_action()
File
"/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py",
line 681, in action_execute
self.get_hdfs_resource_executor().action_execute(self)
File
"/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py",
line 164, in action_execute
logoutput=logoutput,
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line
166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py",
line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py",
line 124, in run_action
provider_action()
File
"/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line
263, in action_run
returns=self.resource.returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
102, in checked_call
tries=tries, try_sleep=try_sleep,
timeout_kill_strategy=timeout_kill_strategy, returns=returns)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line
314, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'hadoop
--config /usr/hdp/3.0.1.0-175/hadoop/conf jar
/var/lib/ambari-agent/lib/fast-hdfs-resource.jar
/var/lib/ambari-agent/tmp/hdfs_resources_1535895647.58.json' returned 1.
Initializing filesystem uri: hdfs://mycluster
Creating: Resource [source=null,
target=wasb://spark2l-at30wu-livy2-121...@humbtesting5wua2.blob.core.windows.net/user/livy/recovery-69e01f25-3b7b-4af4-a787-37664ab45f0c,
type=directory, action=create, owner=livy, group=null, mode=700,
recursiveChown=false, recursiveChmod=false, changePermissionforParents=false,
manageIfExists=true] in hdfs://mycluster
Exception occurred, Reason: Wrong FS:
wasb://spark2l-at30wu-livy2-121...@humbtesting5wua2.blob.core.windows.net/user/livy/recovery-69e01f25-3b7b-4af4-a787-37664ab45f0c,
expected: hdfs://mycluster
java.lang.IllegalArgumentException: Wrong FS:
wasb://spark2l-at30wu-livy2-121...@humbtesting5wua2.blob.core.windows.net/user/livy/recovery-69e01f25-3b7b-4af4-a787-37664ab45f0c,
expected: hdfs://mycluster
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:240)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1583)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1595)
at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1768)
at
org.apache.ambari.fast_hdfs_resource.Resource.checkResourceParameters(Resource.java:193)
at org.apache.ambari.fast_hdfs_resource.Runner.main(Runner.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Live cluster:
<https://spark2l-at30wu-
livy2-121012.southeastasia.cloudapp.azure.com/#/main/services/SPARK2/summary>
Please note that the a system test sets the
property | from | to
---|---|---
`livy.server.recovery.state-store` | `zookeeper` | `filesystem`
`livy.server.recovery.state-store.url` |
`zk1-b24996.zu2zfpuge4su5ceknrmkpsq3ra.ix.internal.cloudapp.net:2181,zk2-b24996.zu2zfpuge4su5ceknrmkpsq3ra.ix.internal.cloudapp.net:2181,zk5-b24996.zu2zfpuge4su5ceknrmkpsq3ra.ix.internal.cloudapp.net:2181`
| `wasb://spark2l-at30wu-
livy2-121...@humbtesting5wua2.blob.core.windows.net/user/livy/recovery-
69e01f25-3b7b-4af4-a787-37664ab45f0c`
and then the restart fails.
> HDI Livy2 fails to restart
> --------------------------
>
> Key: AMBARI-24587
> URL: https://issues.apache.org/jira/browse/AMBARI-24587
> Project: Ambari
> Issue Type: Bug
> Reporter: Andrew Onischuk
> Assignee: Andrew Onischuk
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.7.2
>
> Attachments: AMBARI-24587.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Livy2 restart fails from Ambari due to Ambari could not fetch some Hadoop
> configs?
> StdErr:
> {code}
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/livy2_server.py",
> line 148, in <module>
> LivyServer().execute()
> File
> "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py",
> line 351, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/livy2_server.py",
> line 62, in start
> self.configure(env)
> File
> "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/livy2_server.py",
> line 52, in configure
> setup_livy(env, 'server', upgrade_type=upgrade_type, action =
> 'config')
> File
> "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/SPARK2/package/scripts/setup_livy2.py",
> line 53, in setup_livy
> params.HdfsResource(None, action="execute")
> File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line
> 166, in __init__
> self.env.run()
> File
> "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line
> 160, in run
> self.run_action(resource, action)
> File
> "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line
> 124, in run_action
> provider_action()
> File
> "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py",
> line 681, in action_execute
> self.get_hdfs_resource_executor().action_execute(self)
> File
> "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py",
> line 164, in action_execute
> logoutput=logoutput,
> File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line
> 166, in __init__
> self.env.run()
> File
> "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line
> 160, in run
> self.run_action(resource, action)
> File
> "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line
> 124, in run_action
> provider_action()
> File
> "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py",
> line 263, in action_run
> returns=self.resource.returns)
> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py",
> line 72, in inner
> result = function(command, **kwargs)
> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py",
> line 102, in checked_call
> tries=tries, try_sleep=try_sleep,
> timeout_kill_strategy=timeout_kill_strategy, returns=returns)
> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py",
> line 150, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py",
> line 314, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of 'hadoop
> --config /usr/hdp/3.0.1.0-175/hadoop/conf jar
> /var/lib/ambari-agent/lib/fast-hdfs-resource.jar
> /var/lib/ambari-agent/tmp/hdfs_resources_1535895647.58.json' returned 1.
> Initializing filesystem uri: hdfs://mycluster
> Creating: Resource [source=null,
> target=wasb://spark2l-at30wu-livy2-121...@humbtesting5wua2.blob.core.windows.net/user/livy/recovery-69e01f25-3b7b-4af4-a787-37664ab45f0c,
> type=directory, action=create, owner=livy, group=null, mode=700,
> recursiveChown=false, recursiveChmod=false, changePermissionforParents=false,
> manageIfExists=true] in hdfs://mycluster
> Exception occurred, Reason: Wrong FS:
> wasb://spark2l-at30wu-livy2-121...@humbtesting5wua2.blob.core.windows.net/user/livy/recovery-69e01f25-3b7b-4af4-a787-37664ab45f0c,
> expected: hdfs://mycluster
> java.lang.IllegalArgumentException: Wrong FS:
> wasb://spark2l-at30wu-livy2-121...@humbtesting5wua2.blob.core.windows.net/user/livy/recovery-69e01f25-3b7b-4af4-a787-37664ab45f0c,
> expected: hdfs://mycluster
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:781)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:240)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1583)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1580)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1595)
> at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1768)
> at
> org.apache.ambari.fast_hdfs_resource.Resource.checkResourceParameters(Resource.java:193)
> at org.apache.ambari.fast_hdfs_resource.Runner.main(Runner.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
>
> {code}
> Live cluster:
> <https://spark2l-at30wu-
> livy2-121012.southeastasia.cloudapp.azure.com/#/main/services/SPARK2/summary>
> Please note that the a system test sets the
> property | from | to
> ---|---|---
> `livy.server.recovery.state-store` | `zookeeper` | `filesystem`
> `livy.server.recovery.state-store.url` |
> `zk1-b24996.zu2zfpuge4su5ceknrmkpsq3ra.ix.internal.cloudapp.net:2181,zk2-b24996.zu2zfpuge4su5ceknrmkpsq3ra.ix.internal.cloudapp.net:2181,zk5-b24996.zu2zfpuge4su5ceknrmkpsq3ra.ix.internal.cloudapp.net:2181`
> | `wasb://spark2l-at30wu-
> livy2-121...@humbtesting5wua2.blob.core.windows.net/user/livy/recovery-
> 69e01f25-3b7b-4af4-a787-37664ab45f0c`
>
> and then the restart fails.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
