[
https://issues.apache.org/jira/browse/HDFS-14870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huhaiyang updated HDFS-14870:
-----------------------------
Description:
Client Settings agent refers to ObserverReadProxyProvider class, read the RPC
request will be forwarded to the ONN, but involves the need to modify the file,
the request will be processed twice, cause the request time is too long;
When ONN received from the client to read a file, will call FSNamesystem
getBlockLocations method to get the list of file information, and the current
atime decided the current file needs to be updated
(dfs.namenode.accesstime.precision, HDFS file access time is accurate, The
default value is 1 hour）. The ONN node does not currently support write
operations. This logic throws an alternate exception, and the client retries
the RPC request and forwards it to ANN for execution. As a result, the current
file getBlockLocations method will be called twice, lengthening the entire RPC
request.
test case:
{{hadoop fs -cat /becnhmarks/NNBench/output/part-00000}}
{{}}{{}}
{{19}}{{/}}{{09}}{{/}}{{24}} {{14}}{{:}}{{26}}{{:}}{{21}} {{WARN
ha.ObserverReadProxyProvider: Invocation returned exception on
[xxx}}\{{:}}\{{8020}}{{]; }}{{1}} {{failure(s) so far}}
{{org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
Operation category WRITE is not supported in state observer. Visit
https:}}{{//s.apache.org/sbnn-error}}
{{ }}{{at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:}}{{98}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:}}{{2001}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:}}{{1459}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:}}{{2014}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:}}{{740}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:}}{{439}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$}}{{2}}{{.callBlockingMethod(ClientNamenodeProtocolProtos.java)}}
{{ }}{{at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:}}{{531}}{{)}}
{{ }}{{at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:}}{{1070}}{{)}}
{{ }}{{at
org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:}}{{928}}{{)}}
{{ }}{{at
org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:}}{{863}}{{)}}
{{ }}{{at java.security.AccessController.doPrivileged(Native Method)}}
{{ }}{{at javax.security.auth.Subject.doAs(Subject.java:}}{{422}}{{)}}
{{ }}{{at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:}}{{1891}}{{)}}
{{ }}{{at
org.apache.hadoop.ipc.Server$Handler.run(Server.java:}}{{2814}}{{)}}
{{ }}{{at org.apache.hadoop.ipc.Client.call(Client.java:}}{{1527}}{{)}}
{{ }}{{at org.apache.hadoop.ipc.Client.call(Client.java:}}{{1461}}{{)}}
{{ }}{{at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:}}{{234}}{{)}}
{{ }}{{at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)}}
{{ }}{{at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:}}{{263}}{{)}}
{{ }}{{at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
{{ }}{{at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:}}{{62}}{{)}}
{{ }}{{at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:}}{{43}}{{)}}
{{ }}{{at java.lang.reflect.Method.invoke(Method.java:}}{{498}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider$ObserverReadInvocationHandler.invoke(ObserverReadProxyProvider.java:}}{{402}}{{)}}
{{ }}{{at com.sun.proxy.$Proxy5.getBlockLocations(Unknown Source)}}
{{ }}{{at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
{{ }}{{at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:}}{{62}}{{)}}
{{ }}{{at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:}}{{43}}{{)}}
{{ }}{{at java.lang.reflect.Method.invoke(Method.java:}}{{498}}{{)}}
{{ }}{{at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:}}{{253}}{{)}}
{{ }}{{at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:}}{{101}}{{)}}
{{ }}{{at com.sun.proxy.$Proxy5.getBlockLocations(Unknown Source)}}
{{ }}{{at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:}}{{1230}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:}}{{1217}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:}}{{1205}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:}}{{307}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:}}{{273}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:}}{{265}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:}}{{1530}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DistributedFileSystem$}}{{3}}{{.doCall(DistributedFileSystem.java:}}{{359}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DistributedFileSystem$}}{{3}}{{.doCall(DistributedFileSystem.java:}}{{354}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:}}{{81}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:}}{{367}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.FileSystem.open(FileSystem.java:}}{{837}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.shell.Display$Cat.getInputStream(Display.java:}}{{114}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:}}{{102}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.shell.Command.processPaths(Command.java:}}{{317}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:}}{{289}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.shell.Command.processArgument(Command.java:}}{{271}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.shell.Command.processArguments(Command.java:}}{{255}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:}}{{201}}{{)}}
{{ }}{{at org.apache.hadoop.fs.shell.Command.run(Command.java:}}{{165}}{{)}}
{{ }}{{at org.apache.hadoop.fs.FsShell.run(FsShell.java:}}{{287}}{{)}}
{{ }}{{at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:}}{{70}}{{)}}
{{ }}{{at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:}}{{84}}{{)}}
{{ }}{{at org.apache.hadoop.fs.FsShell.main(FsShell.java:}}{{340}}{{)}}
{{19}}{{/}}{{09}}{{/}}{{24}} {{14}}{{:}}{{26}}{{:}}{{21}} {{WARN
ha.ObserverReadProxyProvider: }}{{1}} {{observers have failed }}{{for}} {{read
request getBlockLocations; also found }}{{1}} {{standby, }}{{1}} {{active, and
}}{{0}} {{unreachable. Falling back to active.}}
{{l:numOfExceptions }}{{0}}
{{l:successfulFileOps }}{{20000000}}
{{l:totalTimeAL1 }}{{652276407}}
{{l:totalTimeAL2 }}{{0}}
{{l:totalTimeTPmS }}{{65546010}}
{{max:mapEndTimeTPmS }}{{1569299234488}}
{{min:mapStartTimeTPmS }}{{1569297954255}}
\{{}}
\{{}}
{{ONN }}{{hdfs-audit.log}}
{{2019}}{-}{{09}}{-}{{24}} {{14}}{{:}}{{26}}{{:}}{{21}}{{,}}{{159}} {{INFO
FSNamesystem.audit: allowed=}}{{true}} {{ugi=hadoop(}}{{null}}{{) ip=/xxx
}}{{cmd=getfileinfo src=/becnhmarks/NNBench/output/part-}}{{00000}}
{{dst=}}{{null}} {{perm=}}{{null}} {{proto=rpc}}
{{2019}}{-}{{09}}{-}{{24}} {{14}}{{:}}{{26}}{{:}}{{21}}{{,}}{{196}} {{INFO
FSNamesystem.audit: allowed=}}{{true}} {{ugi=hadoop(}}{{null}}{{)
ip=/xxx}} {{cmd=open src=/becnhmarks/NNBench/output/part-}}{{00000}}
{{dst=}}{{null}} {{perm=}}{{null}} {{proto=rpc}}
{{ANN hdfs-audit.log}}
{{2019}}{-}{{09}}{-}{{24}} {{14}}{{:}}{{26}}{{:}}{{21}}{{,}}{{205}} {{INFO
FSNamesystem.audit: allowed=}}{{true}} {{ugi=hadoop(}}{{null}}{{)
ip=/xxx}} {{cmd=open src=/becnhmarks/NNBench/output/part-}}{{00000}}
{{dst=}}{{null}} {{perm=}}{{null}} {{proto=rpc}}
{{hadoop fs -cat /becnhmarks/NNBench/output/part-}}{{00000}}
{{}}{{}}
{{l:numOfExceptions }}{{0}}
{{l:successfulFileOps }}{{20000000}}
{{l:totalTimeAL1 }}{{652276407}}
{{l:totalTimeAL2 }}{{0}}
{{l:totalTimeTPmS }}{{65546010}}
{{max:mapEndTimeTPmS }}{{1569299234488}}
{{min:mapStartTimeTPmS }}{{1569297954255}}
{{ONN hdfs-audit.log}}
{{2019}}{-}{{09}}{-}{{24}} {{14}}{{:}}{{26}}{{:}}{{42}}{{,}}{{312}} {{INFO
FSNamesystem.audit: allowed=}}{{true}} {{ugi=hadoop(}}{{null}}{{) ip=/xxx
}}{{cmd=getfileinfo src=/becnhmarks/NNBench/output/part-}}{{00000}}
{{dst=}}{{null}} {{perm=}}{{null}} {{proto=rpc}}
{{2019}}{-}{{09}}{-}{{24}} {{14}}{{:}}{{26}}{{:}}{{42}}{{,}}{{638}} {{INFO
FSNamesystem.audit: allowed=}}{{true}} {{ugi=hadoop(}}{{null}}{{)
ip=/xxx}} {{cmd=open src=/becnhmarks/NNBench/output/part-}}{{00000}}
{{dst=}}{{null}} {{perm=}}{{null}} {{proto=rpc}}
{{ANN hdfs-audit.log No RPC request information}}
was:
Client Settings agent refers to ObserverReadProxyProvider class, read the RPC
request will be forwarded to the ONN, but involves the need to modify the file,
the request will be processed twice, cause the request time is too long;
When ONN received from the client to read a file, will call FSNamesystem
getBlockLocations method to get the list of file information, and the current
atime decided the current file needs to be updated
(dfs.namenode.accesstime.precision, HDFS file access time is accurate, The
default value is 1 hour）. The ONN node does not currently support write
operations. This logic throws an alternate exception, and the client retries
the RPC request and forwards it to ANN for execution. As a result, the current
file getBlockLocations method will be called twice, lengthening the entire RPC
request.
test case:
{{hadoop fs -cat /becnhmarks/NNBench/output/part-}}{{00000}}
{{}}{{}}
{{19}}{{/}}{{09}}{{/}}{{24}} {{14}}{{:}}{{26}}{{:}}{{21}} {{WARN
ha.ObserverReadProxyProvider: Invocation returned exception on
[xxx}}{{:}}{{8020}}{{]; }}{{1}} {{failure(s) so far}}
{{org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
Operation category WRITE is not supported in state observer. Visit
https:}}{{//s.apache.org/sbnn-error}}
{{ }}{{at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:}}{{98}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:}}{{2001}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:}}{{1459}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:}}{{2014}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:}}{{740}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:}}{{439}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$}}{{2}}{{.callBlockingMethod(ClientNamenodeProtocolProtos.java)}}
{{ }}{{at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:}}{{531}}{{)}}
{{ }}{{at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:}}{{1070}}{{)}}
{{ }}{{at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:}}{{928}}{{)}}
{{ }}{{at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:}}{{863}}{{)}}
{{ }}{{at java.security.AccessController.doPrivileged(Native Method)}}
{{ }}{{at javax.security.auth.Subject.doAs(Subject.java:}}{{422}}{{)}}
{{ }}{{at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:}}{{1891}}{{)}}
{{ }}{{at
org.apache.hadoop.ipc.Server$Handler.run(Server.java:}}{{2814}}{{)}}
{{ }}{{at org.apache.hadoop.ipc.Client.call(Client.java:}}{{1527}}{{)}}
{{ }}{{at org.apache.hadoop.ipc.Client.call(Client.java:}}{{1461}}{{)}}
{{ }}{{at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:}}{{234}}{{)}}
{{ }}{{at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)}}
{{ }}{{at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:}}{{263}}{{)}}
{{ }}{{at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
{{ }}{{at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:}}{{62}}{{)}}
{{ }}{{at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:}}{{43}}{{)}}
{{ }}{{at java.lang.reflect.Method.invoke(Method.java:}}{{498}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider$ObserverReadInvocationHandler.invoke(ObserverReadProxyProvider.java:}}{{402}}{{)}}
{{ }}{{at com.sun.proxy.$Proxy5.getBlockLocations(Unknown Source)}}
{{ }}{{at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
{{ }}{{at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:}}{{62}}{{)}}
{{ }}{{at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:}}{{43}}{{)}}
{{ }}{{at java.lang.reflect.Method.invoke(Method.java:}}{{498}}{{)}}
{{ }}{{at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:}}{{253}}{{)}}
{{ }}{{at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:}}{{101}}{{)}}
{{ }}{{at com.sun.proxy.$Proxy5.getBlockLocations(Unknown Source)}}
{{ }}{{at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:}}{{1230}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:}}{{1217}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:}}{{1205}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:}}{{307}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:}}{{273}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:}}{{265}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:}}{{1530}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DistributedFileSystem$}}{{3}}{{.doCall(DistributedFileSystem.java:}}{{359}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DistributedFileSystem$}}{{3}}{{.doCall(DistributedFileSystem.java:}}{{354}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:}}{{81}}{{)}}
{{ }}{{at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:}}{{367}}{{)}}
{{ }}{{at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:}}{{837}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.shell.Display$Cat.getInputStream(Display.java:}}{{114}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:}}{{102}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.shell.Command.processPaths(Command.java:}}{{317}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:}}{{289}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.shell.Command.processArgument(Command.java:}}{{271}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.shell.Command.processArguments(Command.java:}}{{255}}{{)}}
{{ }}{{at
org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:}}{{201}}{{)}}
{{ }}{{at org.apache.hadoop.fs.shell.Command.run(Command.java:}}{{165}}{{)}}
{{ }}{{at org.apache.hadoop.fs.FsShell.run(FsShell.java:}}{{287}}{{)}}
{{ }}{{at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:}}{{70}}{{)}}
{{ }}{{at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:}}{{84}}{{)}}
{{ }}{{at org.apache.hadoop.fs.FsShell.main(FsShell.java:}}{{340}}{{)}}
{{19}}{{/}}{{09}}{{/}}{{24}} {{14}}{{:}}{{26}}{{:}}{{21}} {{WARN
ha.ObserverReadProxyProvider: }}{{1}} {{observers have failed }}{{for}} {{read
request getBlockLocations; also found }}{{1}} {{standby, }}{{1}} {{active, and
}}{{0}} {{unreachable. Falling back to active.}}
{{l:numOfExceptions }}{{0}}
{{l:successfulFileOps }}{{20000000}}
{{l:totalTimeAL1 }}{{652276407}}
{{l:totalTimeAL2 }}{{0}}
{{l:totalTimeTPmS }}{{65546010}}
{{max:mapEndTimeTPmS }}{{1569299234488}}
{{min:mapStartTimeTPmS }}{{1569297954255}}
{{}}
{{}}
{{ONN }}{{hdfs-audit.log}}
{{2019}}{{-}}{{09}}{{-}}{{24}} {{14}}{{:}}{{26}}{{:}}{{21}}{{,}}{{159}} {{INFO
FSNamesystem.audit: allowed=}}{{true}} {{ugi=hadoop(}}{{null}}{{) ip=/xxx
}}{{cmd=getfileinfo src=/becnhmarks/NNBench/output/part-}}{{00000}}
{{dst=}}{{null}} {{perm=}}{{null}} {{proto=rpc}}
{{2019}}{{-}}{{09}}{{-}}{{24}} {{14}}{{:}}{{26}}{{:}}{{21}}{{,}}{{196}} {{INFO
FSNamesystem.audit: allowed=}}{{true}} {{ugi=hadoop(}}{{null}}{{)
ip=/xxx}} {{cmd=open src=/becnhmarks/NNBench/output/part-}}{{00000}}
{{dst=}}{{null}} {{perm=}}{{null}} {{proto=rpc}}
{{ANN hdfs-audit.log}}
{{2019}}{{-}}{{09}}{{-}}{{24}} {{14}}{{:}}{{26}}{{:}}{{21}}{{,}}{{205}} {{INFO
FSNamesystem.audit: allowed=}}{{true}} {{ugi=hadoop(}}{{null}}{{)
ip=/xxx}} {{cmd=open src=/becnhmarks/NNBench/output/part-}}{{00000}}
{{dst=}}{{null}} {{perm=}}{{null}} {{proto=rpc}}
{{hadoop fs -cat /becnhmarks/NNBench/output/part-}}{{00000}}
{{}}{{}}
{{l:numOfExceptions }}{{0}}
{{l:successfulFileOps }}{{20000000}}
{{l:totalTimeAL1 }}{{652276407}}
{{l:totalTimeAL2 }}{{0}}
{{l:totalTimeTPmS }}{{65546010}}
{{max:mapEndTimeTPmS }}{{1569299234488}}
{{min:mapStartTimeTPmS }}{{1569297954255}}
{{ONN hdfs-audit.log}}
{{2019}}{{-}}{{09}}{{-}}{{24}} {{14}}{{:}}{{26}}{{:}}{{42}}{{,}}{{312}} {{INFO
FSNamesystem.audit: allowed=}}{{true}} {{ugi=hadoop(}}{{null}}{{)
ip=/}}{{10.83}}{{.}}{{78.53}} {{cmd=getfileinfo
src=/becnhmarks/NNBench/output/part-}}{{00000}} {{dst=}}{{null}}
{{perm=}}{{null}} {{proto=rpc}}
{{2019}}{{-}}{{09}}{{-}}{{24}} {{14}}{{:}}{{26}}{{:}}{{42}}{{,}}{{638}} {{INFO
FSNamesystem.audit: allowed=}}{{true}} {{ugi=hadoop(}}{{null}}{{)
ip=/}}{{10.83}}{{.}}{{78.53}} {{cmd=open
src=/becnhmarks/NNBench/output/part-}}{{00000}} {{dst=}}{{null}}
{{perm=}}{{null}} {{proto=rpc}}
{{ANN hdfs-audit.log No RPC request information}}
> [SBN read] Client Settings to ObserverReadProxyProvider read a file request,
> due to the file atime time needs to be modified, lead to request to be
> processed twice
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-14870
> URL: https://issues.apache.org/jira/browse/HDFS-14870
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
> Affects Versions: 2.10.0, 3.3.0, 3.2.1
> Reporter: huhaiyang
> Priority: Major
>
> Client Settings agent refers to ObserverReadProxyProvider class, read the RPC
> request will be forwarded to the ONN, but involves the need to modify the
> file, the request will be processed twice, cause the request time is too long;
> When ONN received from the client to read a file, will call FSNamesystem
> getBlockLocations method to get the list of file information, and the current
> atime decided the current file needs to be updated
> (dfs.namenode.accesstime.precision, HDFS file access time is accurate, The
> default value is 1 hour）. The ONN node does not currently support write
> operations. This logic throws an alternate exception, and the client retries
> the RPC request and forwards it to ANN for execution. As a result, the
> current file getBlockLocations method will be called twice, lengthening the
> entire RPC request.
>
> test case:
> {{hadoop fs -cat /becnhmarks/NNBench/output/part-00000}}
> {{}}{{}}
> {{19}}{{/}}{{09}}{{/}}{{24}} {{14}}{{:}}{{26}}{{:}}{{21}} {{WARN
> ha.ObserverReadProxyProvider: Invocation returned exception on
> [xxx}}\{{:}}\{{8020}}{{]; }}{{1}} {{failure(s) so far}}
>
> {{org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category WRITE is not supported in state observer. Visit
> https:}}{{//s.apache.org/sbnn-error}}
> {{ }}{{at
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:}}{{98}}{{)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:}}{{2001}}{{)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:}}{{1459}}{{)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:}}{{2014}}{{)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:}}{{740}}{{)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:}}{{439}}{{)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$}}{{2}}{{.callBlockingMethod(ClientNamenodeProtocolProtos.java)}}
> {{ }}{{at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:}}{{531}}{{)}}
> {{ }}{{at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:}}{{1070}}{{)}}
> {{ }}{{at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:}}{{928}}{{)}}
> {{ }}{{at
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:}}{{863}}{{)}}
> {{ }}{{at java.security.AccessController.doPrivileged(Native Method)}}
> {{ }}{{at javax.security.auth.Subject.doAs(Subject.java:}}{{422}}{{)}}
> {{ }}{{at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:}}{{1891}}{{)}}
> {{ }}{{at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:}}{{2814}}{{)}}
>
> {{ }}{{at org.apache.hadoop.ipc.Client.call(Client.java:}}{{1527}}{{)}}
> {{ }}{{at org.apache.hadoop.ipc.Client.call(Client.java:}}{{1461}}{{)}}
> {{ }}{{at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:}}{{234}}{{)}}
> {{ }}{{at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:}}{{263}}{{)}}
> {{ }}{{at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
> {{ }}{{at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:}}{{62}}{{)}}
> {{ }}{{at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:}}{{43}}{{)}}
> {{ }}{{at java.lang.reflect.Method.invoke(Method.java:}}{{498}}{{)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider$ObserverReadInvocationHandler.invoke(ObserverReadProxyProvider.java:}}{{402}}{{)}}
> {{ }}{{at com.sun.proxy.$Proxy5.getBlockLocations(Unknown Source)}}
> {{ }}{{at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
> {{ }}{{at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:}}{{62}}{{)}}
> {{ }}{{at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:}}{{43}}{{)}}
> {{ }}{{at java.lang.reflect.Method.invoke(Method.java:}}{{498}}{{)}}
> {{ }}{{at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:}}{{253}}{{)}}
> {{ }}{{at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:}}{{101}}{{)}}
> {{ }}{{at com.sun.proxy.$Proxy5.getBlockLocations(Unknown Source)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:}}{{1230}}{{)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:}}{{1217}}{{)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:}}{{1205}}{{)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:}}{{307}}{{)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:}}{{273}}{{)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:}}{{265}}{{)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:}}{{1530}}{{)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.DistributedFileSystem$}}{{3}}{{.doCall(DistributedFileSystem.java:}}{{359}}{{)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.DistributedFileSystem$}}{{3}}{{.doCall(DistributedFileSystem.java:}}{{354}}{{)}}
> {{ }}{{at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:}}{{81}}{{)}}
> {{ }}{{at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:}}{{367}}{{)}}
> {{ }}{{at
> org.apache.hadoop.fs.FileSystem.open(FileSystem.java:}}{{837}}{{)}}
> {{ }}{{at
> org.apache.hadoop.fs.shell.Display$Cat.getInputStream(Display.java:}}{{114}}{{)}}
> {{ }}{{at
> org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:}}{{102}}{{)}}
> {{ }}{{at
> org.apache.hadoop.fs.shell.Command.processPaths(Command.java:}}{{317}}{{)}}
> {{ }}{{at
> org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:}}{{289}}{{)}}
> {{ }}{{at
> org.apache.hadoop.fs.shell.Command.processArgument(Command.java:}}{{271}}{{)}}
> {{ }}{{at
> org.apache.hadoop.fs.shell.Command.processArguments(Command.java:}}{{255}}{{)}}
> {{ }}{{at
> org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:}}{{201}}{{)}}
> {{ }}{{at
> org.apache.hadoop.fs.shell.Command.run(Command.java:}}{{165}}{{)}}
> {{ }}{{at org.apache.hadoop.fs.FsShell.run(FsShell.java:}}{{287}}{{)}}
> {{ }}{{at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:}}{{70}}{{)}}
> {{ }}{{at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:}}{{84}}{{)}}
> {{ }}{{at org.apache.hadoop.fs.FsShell.main(FsShell.java:}}{{340}}{{)}}
> {{19}}{{/}}{{09}}{{/}}{{24}} {{14}}{{:}}{{26}}{{:}}{{21}} {{WARN
> ha.ObserverReadProxyProvider: }}{{1}} {{observers have failed }}{{for}}
> {{read request getBlockLocations; also found }}{{1}} {{standby, }}{{1}}
> {{active, and }}{{0}} {{unreachable. Falling back to active.}}
> {{l:numOfExceptions }}{{0}}
> {{l:successfulFileOps }}{{20000000}}
> {{l:totalTimeAL1 }}{{652276407}}
> {{l:totalTimeAL2 }}{{0}}
> {{l:totalTimeTPmS }}{{65546010}}
> {{max:mapEndTimeTPmS }}{{1569299234488}}
> {{min:mapStartTimeTPmS }}{{1569297954255}}
> \{{}}
> \{{}}
> {{ONN }}{{hdfs-audit.log}}
> {{2019}}{-}{{09}}{-}{{24}} {{14}}{{:}}{{26}}{{:}}{{21}}{{,}}{{159}} {{INFO
> FSNamesystem.audit: allowed=}}{{true}} {{ugi=hadoop(}}{{null}}{{)
> ip=/xxx }}{{cmd=getfileinfo src=/becnhmarks/NNBench/output/part-}}{{00000}}
> {{dst=}}{{null}} {{perm=}}{{null}} {{proto=rpc}}
> {{2019}}{-}{{09}}{-}{{24}} {{14}}{{:}}{{26}}{{:}}{{21}}{{,}}{{196}} {{INFO
> FSNamesystem.audit: allowed=}}{{true}} {{ugi=hadoop(}}{{null}}{{)
> ip=/xxx}} {{cmd=open src=/becnhmarks/NNBench/output/part-}}{{00000}}
> {{dst=}}{{null}} {{perm=}}{{null}} {{proto=rpc}}
>
> {{ANN hdfs-audit.log}}
> {{2019}}{-}{{09}}{-}{{24}} {{14}}{{:}}{{26}}{{:}}{{21}}{{,}}{{205}} {{INFO
> FSNamesystem.audit: allowed=}}{{true}} {{ugi=hadoop(}}{{null}}{{)
> ip=/xxx}} {{cmd=open src=/becnhmarks/NNBench/output/part-}}{{00000}}
> {{dst=}}{{null}} {{perm=}}{{null}} {{proto=rpc}}
>
>
> {{hadoop fs -cat /becnhmarks/NNBench/output/part-}}{{00000}}
> {{}}{{}}
> {{l:numOfExceptions }}{{0}}
> {{l:successfulFileOps }}{{20000000}}
> {{l:totalTimeAL1 }}{{652276407}}
> {{l:totalTimeAL2 }}{{0}}
> {{l:totalTimeTPmS }}{{65546010}}
> {{max:mapEndTimeTPmS }}{{1569299234488}}
> {{min:mapStartTimeTPmS }}{{1569297954255}}
>
> {{ONN hdfs-audit.log}}
> {{2019}}{-}{{09}}{-}{{24}} {{14}}{{:}}{{26}}{{:}}{{42}}{{,}}{{312}} {{INFO
> FSNamesystem.audit: allowed=}}{{true}} {{ugi=hadoop(}}{{null}}{{)
> ip=/xxx }}{{cmd=getfileinfo src=/becnhmarks/NNBench/output/part-}}{{00000}}
> {{dst=}}{{null}} {{perm=}}{{null}} {{proto=rpc}}
> {{2019}}{-}{{09}}{-}{{24}} {{14}}{{:}}{{26}}{{:}}{{42}}{{,}}{{638}} {{INFO
> FSNamesystem.audit: allowed=}}{{true}} {{ugi=hadoop(}}{{null}}{{)
> ip=/xxx}} {{cmd=open src=/becnhmarks/NNBench/output/part-}}{{00000}}
> {{dst=}}{{null}} {{perm=}}{{null}} {{proto=rpc}}
>
> {{ANN hdfs-audit.log No RPC request information}}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
