[
https://issues.apache.org/jira/browse/HDDS-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039134#comment-17039134
]
Attila Doroszlai edited comment on HDDS-2964 at 2/18/20 5:14 PM:
-----------------------------------------------------------------
Good point [~shashikant]. Indeed, there are some tests that have been broken
while integration tests were disabled. Some of these were fixed instead of
disabling it.
[~elek] and me disabled tests incrementally, on first failure, /n the
[integration-test-cleanup|https://github.com/apache/hadoop-ozone/commits/integration-test-cleanup]
branch. I think anything that was [disabled
after|https://github.com/apache/hadoop-ozone/commits/integration-test-cleanup?after=49f74406346689ca6b3b15a4082c9fba51312bc2+99]
the [first
commit|https://github.com/apache/hadoop-ozone/commit/453496d43d3dc532351fe47ba9be10bc9c498ee2]
can be considered intermittent. Here's a list of such tests from commit infos:
{code}
$ git log --stat
453496d43d3dc532351fe47ba9be10bc9c498ee2..origin/integration-test-cleanup |
grep -F '.java' | grep -v TestRatisPipelineProvider | awk -F '/' '{ print $NF
}' | cut -f1 -d' ' | sort
Test2WayCommitInRatis.java
TestBlockDeletion.java
TestCloseContainerHandlingByClient.java
TestCommitWatcher.java
TestContainerReplicationEndToEnd.java
TestContainerStateMachine.java
TestContainerStateManagerIntegration.java
TestDeleteWithSlowFollower.java
TestFailureHandlingByClient.java
TestGetCommittedBlockLengthAndPutKey.java
TestMiniOzoneCluster.java
TestNodeFailure.java
TestOMRatisSnapshots.java
JTestOzoneAtRestEncryption.java
TestOzoneContainer.java
TestOzoneManagerHA.java
TestOzoneRpcClientWithRatis.java
TestQueryNode.java
TestRatisPipelineCreateAndDestroy.java
TestSCMPipelineMetrics.java
TestStorageContainerManager.java
TestWatchForCommit.java
{code}
For example TestCloseContainerHandlingByClient#testMultiBlockWrites3 failed
only once, unrelated to testBlockWriteViaRatis method being broken consistently.
Later I continued on a branch in my fork, disabled these I think:
{code}
TestSCMContainerPlacementPolicyMetrics.java
TestCSMMetrics.java
TestSecureContainerServer.java
TestMultiBlockWritesWithDnFailures.java
{code}
was (Author: adoroszlai):
Good point [~shashikant]. Indeed, there are some tests that have been broken
while integration tests were disabled. Some of these were fixed instead of
disabling it.
[~elek] and me disabled tests incrementally, on first failure, on the
[integration-test-cleanup|https://github.com/apache/hadoop-ozone/commits/integration-test-cleanup)
branch. I think anything that was [disabled
after|https://github.com/apache/hadoop-ozone/commits/integration-test-cleanup?after=49f74406346689ca6b3b15a4082c9fba51312bc2+99]
the [first
commit|https://github.com/apache/hadoop-ozone/commit/453496d43d3dc532351fe47ba9be10bc9c498ee2]
can be considered intermittent. Here's a list of such tests from commit infos:
{code}
$ git log --stat
453496d43d3dc532351fe47ba9be10bc9c498ee2..origin/integration-test-cleanup |
grep -F '.java' | grep -v TestRatisPipelineProvider | awk -F '/' '{ print $NF
}' | cut -f1 -d' ' | sort
Test2WayCommitInRatis.java
TestBlockDeletion.java
TestCloseContainerHandlingByClient.java
TestCommitWatcher.java
TestContainerReplicationEndToEnd.java
TestContainerStateMachine.java
TestContainerStateManagerIntegration.java
TestDeleteWithSlowFollower.java
TestFailureHandlingByClient.java
TestGetCommittedBlockLengthAndPutKey.java
TestMiniOzoneCluster.java
TestNodeFailure.java
TestOMRatisSnapshots.java
TestOzoneAtRestEncryption.java
TestOzoneContainer.java
TestOzoneManagerHA.java
TestOzoneRpcClientWithRatis.java
TestQueryNode.java
TestRatisPipelineCreateAndDestroy.java
TestSCMPipelineMetrics.java
TestStorageContainerManager.java
TestWatchForCommit.java
{code}
For example TestCloseContainerHandlingByClient#testMultiBlockWrites3 failed
only once, unrelated to testBlockWriteViaRatis method being broken consistently.
Later I continued on a branch in my fork, disabled these I think:
{code}
TestSCMContainerPlacementPolicyMetrics.java
TestCSMMetrics.java
TestSecureContainerServer.java
TestMultiBlockWritesWithDnFailures.java
{code}
> Fix @Ignore-d integration tests
> -------------------------------
>
> Key: HDDS-2964
> URL: https://issues.apache.org/jira/browse/HDDS-2964
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: test
> Reporter: Marton Elek
> Priority: Major
>
> We marked all the intermittent unit tests with @Ignore to get reliable
> feedback from CI builds.
> Before HDDS-2833 we had 21 @Ignore annotations, HDDS-2833 introduced 34 new
> one.
> We need to review all of these tests and either fix, or delete or convert
> them to real unit tests.
> The current list of ignore tests:
> {code:java}
> hadoop-hdds/server-scm
> org/apache/hadoop/hdds/scm/node/TestContainerPlacement.java: @Ignore
> hadoop-hdds/server-scm
> org/apache/hadoop/hdds/scm/node/TestDeadNodeHandler.java: @Ignore("Tracked
> by HDDS-2508.")
> hadoop-hdds/server-scm
> org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java: @Ignore
> hadoop-hdds/server-scm
> org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java: @Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/hdds/scm/container/TestContainerStateManagerIntegration.java:
> @Ignore("TODO:HDDS-1159")
> hadoop-ozone/integration-test
> org/apache/hadoop/hdds/scm/pipeline/TestNodeFailure.java: @Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/hdds/scm/pipeline/TestNodeFailure.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineCreateAndDestroy.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/hdds/scm/safemode/TestSCMSafeModeWithPipelineRules.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/Test2WayCommitInRatis.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestBlockOutputStreamWithFailures.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestCloseContainerHandlingByClient.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestCloseContainerHandlingByClient.java:
> @Ignore // test needs to be fixed after close container is handled for
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestCommitWatcher.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestContainerReplicationEndToEnd.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestContainerStateMachineFailures.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestContainerStateMachine.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestDeleteWithSlowFollower.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestFailureHandlingByClient.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestMultiBlockWritesWithDnFailures.java:@IgnoreM
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestOzoneAtRestEncryption.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestOzoneClientRetriesOnException.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java: @Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java:
> @Ignore("Debug Jenkins Timeout")
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientForAclAuditLog.java:@Ignore("Fix
> this after adding audit support for HA Acl code. This will be " +
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientWithRatis.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestSecureOzoneRpcClient.java:
> @Ignore("Needs to be moved out of this class as client setup is static")
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozo.e/container/common/statemachine/commandhandler/TestBlockDeletion.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestCloseContainerByPipeline.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/container/common/transport/server/ratis/TestCSMMetrics.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainer.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainerRatis.java:@Ignore("Disabling
> Ratis tests for pipeline work.")
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/container/ozoneimpl/TestOzoneContainerWithTLS.java:@Ignore("TODO:HDDS-1157")
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/container/ozoneimpl/TestRatisManager.java:@Ignore("Disabling
> Ratis tests for pipeline work.")
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/container/server/TestContainerServer.java:@Ignore("Takes
> too long to run this test. Ignoring for time being.")
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/container/server/TestSecureContainerServer.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/container/TestContainerReplication.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/freon/TestFreonWithDatanodeFastRestart.java:
> @Ignore("TODO:HDDS-1160")
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/freon/TestFreonWithDatanodeRestart.java: @Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/freon/TestFreonWithPipelineDestroy.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/om/TestOmMetrics.java:
> @Ignore("Test failing because of table cache. Revisit later.")
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/om/TestOMRatisSnapshots.java:@Ignore
> hadoop-ozone/integration-test org/apache/hadoop/ozone/om/TestOmSQLCli.java:
> @Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/om/TestOzoneManagerHA.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/om/TestOzoneManagerRestart.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/om/TestScmSafeMode.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/scm/node/TestQueryNode.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/scm/pipeline/TestSCMPipelineMetrics.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/scm/TestGetCommittedBlockLengthAndPutKey.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/scm/TestSCMContainerPlacementPolicyMetrics.java:@Ignore("HDDS-2961")
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/TestMiniOzoneCluster.java:@Ignore
> hadoop-ozone/integration-test
> org/apache/hadoop/ozone/TestStorageContainerManager.java:@Ignore
> hadoop-ozone/ozone-manager
> org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBufferWithOMResponse.java:@Ignore("HDDS-2648")
> hadoop-ozone/ozone-manager
> org/apache/hadoop/ozone/om/TestBucketManagerImpl.java:@Ignore("Bucket Manager
> does not use cache, Disable it for now.") {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
