Ethan Rose created HDDS-14686:
---------------------------------
Summary: HSync tests in TestOzoneShellHA depend on state from
other tests
Key: HDDS-14686
URL: https://issues.apache.org/jira/browse/HDDS-14686
Project: Apache Ozone
Issue Type: Sub-task
Reporter: Ethan Rose
Assignee: Ashish Kumar
In HDDS-14580, a new test {{testPrepareCommandNoOp}} was added to
{{{}TestOzoneShellHA{}}}. JUnit runs these tests in the following order:
# {{testPrepareCommandNoOp}}
# (other tests)
# {{testAdminCmdListOpenFilesWithDeletedKeys}}
# (other tests)
# {{testAdminCmdListOpenFiles}}
When this happens, the two open file tests start failing. In particular,
{{testAdminCmdListOpenFilesWithDeletedKeys}} fails with the following error:
{code:java}
2026-02-20 13:42:56,595 [ForkJoinPool-1-worker-9] INFO shell.TestOzoneShellHA
(TestOzoneShellHA.java:execute(235)) - Executing OzoneShell command with args
[om, lof, --service-id, om-service-test1, --show-deleted, -p,
/volume-list-del/buck1]
org.opentest4j.AssertionFailedError: key should be hsync'ed and not deleted ==>
Expected :true
Actual :false
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:214)
at
org.apache.hadoop.ozone.shell.TestOzoneShellHA.testAdminCmdListOpenFilesWithDeletedKeys(TestOzoneShellHA.java:748)
at java.lang.reflect.Method.invoke(Method.java:498)
at
java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:289)
{code}
When the test was passing previously, the assert read the following output:
{code:java}
5 total open files. Showing 5 open files (limit 100) under path prefix:
/volume-list-del/buck1
Client ID Creation time Hsync'ed Deleted
Open File Path
116099515204632578 2026-02-19T21:53:32.721Z No No
/volume-list-del/buck1/-9223372036854772991/key0
116099515213545475 2026-02-19T21:53:32.902Z No No
/volume-list-del/buck1/-9223372036854772991/key1
116099515214462980 2026-02-19T21:53:32.916Z No No
/volume-list-del/buck1/-9223372036854772991/key2
116099515214856197 2026-02-19T21:53:32.922Z No No
/volume-list-del/buck1/-9223372036854772991/key3
116099515217346566 2026-02-19T21:53:32.960Z Yes No
/volume-list-del/buck1/-9223372036854772991/key4
Reached the end of the list.
{code}
When the test failed, the assert read the following output:
{code:java}
6 total open files. Showing 5 open files (limit 100) under path prefix:
/volume-list-del/buck1
Client ID Creation time Hsync'ed Deleted
Open File Path
116099528644428109 2026-02-19T21:56:48.999Z No No
/volume-list-del/buck1/-9223372036854716927/key0
116099528644690254 2026-02-19T21:56:49.099Z No No
/volume-list-del/buck1/-9223372036854716927/key1
116099528644952399 2026-02-19T21:56:49.105Z No No
/volume-list-del/buck1/-9223372036854716927/key2
116099528645214544 2026-02-19T21:56:49.111Z No No
/volume-list-del/buck1/-9223372036854716927/key3
116099528645411153 2026-02-19T21:56:49.115Z No No
/volume-list-del/buck1/-9223372036854716927/key4
Reached the end of the list.
{code}
That 6th key showing up has somehow been overwritten within the test in the
failure case, because adding {{--show-overwritten}} to the command when the
test is failing shows
{code:java}
6 total open files. Showing 6 open files (limit 100) under path prefix:
/
Client ID Creation time Hsync'ed Deleted
Overwritten Open File Path
116099583048352077 2026-02-19T22:10:37.365Z No No
No /volume-list-del/buck1/-9223372036854716927/key0
116099583048810830 2026-02-19T22:10:37.481Z No No
No /volume-list-del/buck1/-9223372036854716927/key1
116099583049400655 2026-02-19T22:10:37.487Z No No
No /volume-list-del/buck1/-9223372036854716927/key2
116099583050121552 2026-02-19T22:10:37.492Z No No
No /volume-list-del/buck1/-9223372036854716927/key3
116099582361469038 2026-02-19T22:10:37.498Z Yes No
Yes /volume-list-del/buck1/-9223372036854716927/key4
116099583050514769 2026-02-19T22:10:37.498Z No No
No /volume-list-del/buck1/-9223372036854716927/key4
Reached the end of the list.
{code}
Based on the volume name we can see that this is not state introduced by the
prepare test. If only those three tests are run in the same order without the
other tests in the class, they all pass. This also seems to indicate that the
issue is not with the prepare test, but with assumptions that the open key
tests are making about state from previous tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
