ConfX created HDFS-17132:
----------------------------
Summary: Test
org.apache.hadoop.hdfs.server.namenode.ha.TestObserverReadProxyProvider#testObserverToActive
is flaky
Key: HDFS-17132
URL: https://issues.apache.org/jira/browse/HDFS-17132
Project: Hadoop HDFS
Issue Type: Bug
Reporter: ConfX
Attachments: reproduce.sh
h2. What happened
I turned on {{dfs.client.failover.random.order=true}} and then the test
{{{}org.apache.hadoop.hdfs.server.namenode.ha.TestObserverReadProxyProvider#testObserverToActive{}}}becomes
flaky.
h2. Where's the problem
In {{{}AbstractNNFailoverProxyProvider{}}},
{noformat}
// Randomize the list to prevent all clients pointing to the same one
boolean randomized = getRandomOrder(conf, uri);
if (randomized) {
Collections.shuffle(proxies);
}{noformat}
If the random order is open, the proxy would be rearranged such that the tests
fails with non-deterministically.
h2. How to reproduce
# set {{{}dfs.client.failover.random.order{}}}={{{}true{}}}
# run
{{org.apache.hadoop.hdfs.server.namenode.ha.TestObserverReadProxyProvider#testObserverToActive}}
you should observe
{noformat}
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
No writes!
at
org.apache.hadoop.hdfs.server.namenode.ha.TestObserverReadProxyProvider$NameNodeAnswer$ClientProtocolAnswer.answer(TestObserverReadProxyProvider.java:398)
at
org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:39)
at
org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:96)
at
org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
at
org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:35)
at
org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:61)
at
org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:49)
at
org.mockito.internal.creation.bytebuddy.MockMethodInterceptor$DispatcherDefaultingToRealMethod.interceptAbstract(MockMethodInterceptor.java:126)
at
org.apache.hadoop.hdfs.protocol.ClientProtocol$MockitoMock$1652553047.reportBadBlocks(Unknown
Source)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider$ObserverReadInvocationHandler.invoke(ObserverReadProxyProvider.java:519)
at com.sun.proxy.$Proxy32.reportBadBlocks(Unknown Source)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestObserverReadProxyProvider.doWrite(TestObserverReadProxyProvider.java:341)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestObserverReadProxyProvider.doWrite(TestObserverReadProxyProvider.java:332)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestObserverReadProxyProvider.testObserverToActive(TestObserverReadProxyProvider.java:224){noformat}
For an easy reproduction, run the reproduce.sh in the attachment.
We are happy to provide a patch if this issue is confirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
