[
https://issues.apache.org/jira/browse/HDFS-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17598027#comment-17598027
]
ASF GitHub Bot commented on HDFS-16740:
---------------------------------------
snmvaughan opened a new pull request, #4835:
URL: https://github.com/apache/hadoop/pull/4835
### Description of PR
Mini clusters used during HDFS unit tests are reporting test failures that
do not appear to be directly related to submitted changes. The failures are
the result of either interactions between tests run in parallel, or tests which
share common disk space for tests. In all cases, the tests can be run
individually serially without any errors. Addressing this issue will simplify
future submissions by eliminating the confusion introduced by these unrelated
test failures.
We can apply lessons recently from TestRollingUpgrade, which was recently
patched to unblock a recent submission. The fixes involved changing the HDFS
configuration to use temporary disk space for each individual tests, and using
try-with-resources to ensure that clusters were shutdown cleanly.
Add a constructor to the mini cluster constructors and builders that takes a
TemporaryFolder. This encourages the use of managed temporary folders that are
unique to the test and automatically cleaned up. The field is named "baseDir"
to communicate that this folder acts as the base directory for test data.
Using the new constructor encourages the consistent use of the managed
temporary folder.
Allow the same baseDir to be provided multiple times, simplifying the coding
of cluster restarts. This simplifies these tests as the use of the same
TemporaryFolder will always work (even for restarts)
This change also reduces the number of tests that make assumptions about the
mini clusters using encapsulation. These changes avoid configurations that
directly use HDFS_MINIDFS_BASEDIR.
### How was this patch tested?
Both Maven Surefire and an IDE were used to demonstrate that the tests
operate as expected.
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Mini cluster test flakiness
> ---------------------------
>
> Key: HDFS-16740
> URL: https://issues.apache.org/jira/browse/HDFS-16740
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, test
> Affects Versions: 3.4.0, 3.3.9
> Reporter: Steve Vaughan
> Assignee: Steve Vaughan
> Priority: Major
>
> Mini clusters used during HDFS unit tests are reporting test failures that do
> not appear to be directly related to submitted changes. The failures are the
> result of either interactions between tests run in parallel, or tests which
> share common disk space for tests. In all cases, the tests can be run
> individually serially without any errors. Addressing this issue will
> simplify future submissions by eliminating the confusion introduced by these
> unrelated test failures.
> We can apply lessons recently from TestRollingUpgrade, which was recently
> patched to unblock a recent submission. The fixes involved changing the HDFS
> configuration to use temporary disk space for each individual tests, and
> using try-with-resources to ensure that clusters were shutdown cleanly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
