Zita Dombi created HDDS-15996:
---------------------------------
Summary: Flaky TestClientRetryTimeout throws NPE
Key: HDDS-15996
URL: https://issues.apache.org/jira/browse/HDDS-15996
Project: Apache Ozone
Issue Type: Sub-task
Reporter: Zita Dombi
Assignee: Zita Dombi
Fixing
[this|https://github.com/apache/ozone/actions/runs/30094128362/job/89490329328?pr=10570]
flaky CI failure:
{code:java}
[ERROR]
org.apache.hadoop.ozone.client.rpc.TestClientRetryTimeout.testWriteWithLeaderFailureFailsFast
-- Time elapsed: 0.055 s <<< ERROR!java.lang.NullPointerException: Cannot
invoke "org.apache.hadoop.hdds.scm.XceiverClientRatis.getPipeline()" because
"ratisClient" is null at
org.apache.hadoop.ozone.client.rpc.TestClientRetryTimeout.testWriteWithLeaderFailureFailsFast(TestClientRetryTimeout.java:381)
at java.base/java.lang.reflect.Method.invoke(Method.java:565) at
java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:511) at
java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1450)
at
java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2019)
at
java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)
[ERROR] org.apache.hadoop.ozone.client.rpc.TestClientRetryTimeout -- Time
elapsed: 95.60 s <<< FAILURE!java.lang.AssertionError: Found 2 leaked objects,
check logs at
org.apache.hadoop.hdds.utils.db.CodecBuffer.assertNoLeaks(CodecBuffer.java:238)
at org.apache.hadoop.hdds.utils.db.CodecTestUtil.gc(CodecTestUtil.java:54)
at
org.apache.hadoop.ozone.MiniOzoneClusterImpl.shutdown(MiniOzoneClusterImpl.java:381)
at
org.apache.hadoop.ozone.client.rpc.TestClientRetryTimeout.shutdown(TestClientRetryTimeout.java:197)
at java.base/java.lang.reflect.Method.invoke(Method.java:565) at
java.base/java.util.ArrayList.forEach(ArrayList.java:1604) at
java.base/java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1118)
at
java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:511) at
java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1450)
at
java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2019)
at
java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)
{code}
The problem is that all three write tests grab the pipeline via a [hard-coded
first stream
entry|https://github.com/apache/ozone/blob/2473ec54d33604929d3ff7983b40555f167ff481/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestClientRetryTimeout.java#L372-L381]
on line 375:
{code:java}
// Get the pipeline and find the leader
KeyOutputStream keyOutputStream =
assertInstanceOf(KeyOutputStream.class, key.getOutputStream());
OutputStream stream = keyOutputStream.getStreamEntries().get(0)
.getOutputStream();
RatisBlockOutputStream blockOutputStream =
assertInstanceOf(RatisBlockOutputStream.class, stream);
XceiverClientRatis ratisClient =
(XceiverClientRatis) blockOutputStream.getXceiverClient();
Pipeline pipeline = ratisClient.getPipeline(); {code}
With the help of Claude:
{quote}This assumes entry 0 still holds a live XceiverClient after the setup
write/flush. But this is a shared, `@Ordered` PER_CLASS suite — the earlier
tests kill/restart datanodes, and `waitForPipelineTobeReady(THREE)` only
guarantees one healthy factor-3 pipeline exists, not that the specific pipeline
this key lands on is ready.
The shutdown "Found 2 leaked objects" error is a cascade: the NPE aborts before
key.close(), so CodecBuffer.assertNoLeaks() flags the abandoned stream's
buffers at teardown.
{quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
