[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16168254#comment-16168254 ] Steve Loughran commented on HADOOP-14553: - Closing this as fixed for 3.0, with HADOOP-14870 covering the backport & testing associated with that. About to submit a patch there, but I don't think it's ready yet > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch, HADOOP-14553-011.patch, > HADOOP-14553-012.patch, HADOOP-14553-014.patch, HADOOP-14553-015.patch, > HADOOP-14553-016.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16168178#comment-16168178 ] Hudson commented on HADOOP-14553: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12884 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12884/]) HADOOP-14553. Add (parallelized) integration tests to hadoop-azure (stevel: rev 2d2d97fa7d4224369b3c13bc4a45e8cc9e29afb1) * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java * (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/metrics/TestAzureFileSystemInstrumentation.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/ITestNativeAzureFileSystemConcurrencyLive.java * (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestOutOfBandAzureBlobOperationsLive.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestBlobOperationDescriptor.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AzureBlobStorageTestAccount.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/ITestBlobTypeSpeedDifference.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/ITestBlobDataValidation.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/ITestAzureFileSystemErrorConditions.java * (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemContractLive.java * (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemContractEmulator.java * (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractOpen.java * (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemClientLogging.java * (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAtomicRenameDirList.java * (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestFileSystemOperationExceptionMessage.java * (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemConcurrencyLive.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/ITestAzureNativeContractMkdir.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/ITestContainerChecks.java * (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestBlobDataValidation.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemUploadLogic.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/ITestNativeAzureFileSystemAppend.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockWasbAuthorizerImpl.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/ITestAzureNativeContractDelete.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AbstractWasbTestBase.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/ITestOutOfBandAzureBlobOperationsLive.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/ITestBlockBlobInputStream.java * (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/TestAzureNativeContractRename.java * (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestAzureFileSystemErrorConditions.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemMocked.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/ITestAzureNativeContractRename.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemConcurrency.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestBlobMetadata.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AbstractWasbTestWithTimeout.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/ITestNativeAzureFileSystemContractPageBlobLive.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/ITestReadAndSeekPageBlobAfterWrite.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/ITestAzureNativeContractGetFileStatus.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestWasbFsck.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/integration/ITestAzureHugeFiles.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/contract/ITestAzur
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16168155#comment-16168155 ] Steve Loughran commented on HADOOP-14553: - OK, committed to trunk & branch-3, looking @ branch-2 now > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch, HADOOP-14553-011.patch, > HADOOP-14553-012.patch, HADOOP-14553-014.patch, HADOOP-14553-015.patch, > HADOOP-14553-016.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16167002#comment-16167002 ] Aaron Fabbri commented on HADOOP-14553: --- Thanks [~ste...@apache.org]. I re-ran all the hadoop-azure tests and they passed. I'm +1 on v16 patch. > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch, HADOOP-14553-011.patch, > HADOOP-14553-012.patch, HADOOP-14553-014.patch, HADOOP-14553-015.patch, > HADOOP-14553-016.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166282#comment-16166282 ] Steve Loughran commented on HADOOP-14553: - Aaron; reverted the httpclient change...try now > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch, HADOOP-14553-011.patch, > HADOOP-14553-012.patch, HADOOP-14553-014.patch, HADOOP-14553-015.patch, > HADOOP-14553-016.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166150#comment-16166150 ] Steve Loughran commented on HADOOP-14553: - I'm going to be running some wasb "classic" tests soon, then this. One thing I will say: not seen these locally. Notable they are all the mock ones. try skipping Test* by doing a mvn verify -Dtest=anything & see what happens there, if its failing its potentially related. HADOOP-14857 is complaining about a regression too: filed HADOOP-14868 > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch, HADOOP-14553-011.patch, > HADOOP-14553-012.patch, HADOOP-14553-014.patch, HADOOP-14553-015.patch, > HADOOP-14553-016.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165663#comment-16165663 ] Aaron Fabbri commented on HADOOP-14553: --- Just noticed your comment above which answers one of my questions (CleanupTestContainers must be explicitly specified to run). I applied the v16 patch and ran all tests in hadoop-azure and hit a bunch of failures.. Not sure if I've done something stupid but this is what it looks like: {noformat} Failed tests: TestNativeAzureFileSystemConcurrency.testNoTempBlobsVisible:100->Assert.assertEquals:144->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 expected: but was: TestNativeAzureFileSystemContractMocked>FileSystemContractBaseTest.testListStatusRootDir:866->FileSystemContractBaseTest.assertListStatusFinds:896 Path wasb://mockcontai...@mockaccount.blob.core.windows.net/FileSystemContractBaseTest not found in directory wasb://mockcontai...@mockaccount.blob.core.windows.net/:FileStatus{path=wasb://mockAccount.blob.core.windows.net/mockContainer/FileSystemContractBaseTest; isDirectory=false; length=2048; replication=1; blocksize=536870912; modification_time=1505351566081; access_time=0; owner=fabbri; group=supergroup; permission=rw-r--r--; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false} TestNativeAzureFileSystemContractMocked>FileSystemContractBaseTest.testListStatus:330 expected:<1> but was:<2> TestNativeAzureFileSystemContractMocked>FileSystemContractBaseTest.testLSRootDir:856->FileSystemContractBaseTest.assertListFilesFinds:881 Path wasb://mockcontai...@mockaccount.blob.core.windows.net/FileSystemContractBaseTest not found in directory wasb://mockcontai...@mockaccount.blob.core.windows.net/:LocatedFileStatus{path=wasb://mockAccount.blob.core.windows.net/mockContainer/FileSystemContractBaseTest; isDirectory=false; length=2048; replication=1; blocksize=536870912; modification_time=1505351566450; access_time=0; owner=fabbri; group=supergroup; permission=rw-r--r--; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false} TestNativeAzureFileSystemContractMocked>FileSystemContractBaseTest.testDeleteRecursively:452 File doesn't exist TestNativeAzureFileSystemFileNameCheck.testWasbFsck:120->Assert.assertTrue:52->Assert.assertTrue:41->Assert.fail:86 null TestNativeAzureFileSystemMocked>NativeAzureFileSystemBaseTest.testDeepFileCreation:228->NativeAzureFileSystemBaseTest.testDeepFileCreationBase:218->NativeAzureFileSystemBaseTest.assertPathDoesNotExist:87->Assert.fail:88 deleted file: unexpectedly found deep/file/creation/test as FileStatus{path=wasb://mockcontai...@mockaccount.blob.core.windows.net/user/fabbri/deep/file/creation/test; isDirectory=false; length=0; replication=1; blocksize=536870912; modification_time=1505351569092; access_time=0; owner=fabbri; group=supergroup; permission=rw-r--r--; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false} TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testListStatus:312 expected:<1> but was:<2> TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testListStatusFilterWithSomeMatches:369 null TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusSomeMatchesInDirectories:426 expected:<2> but was:<4> TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusWithMultipleWildCardMatches:451 expected:<4> but was:<8> TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testDeleteRecursively:777 File doesn't exist TestOutOfBandAzureBlobOperations.testImplicitFolderDeleted:94->Assert.assertFalse:74->Assert.assertFalse:64->Assert.assertTrue:41->Assert.fail:86 null TestOutOfBandAzureBlobOperations.testImplicitFolderListed:80->Assert.assertEquals:144->Assert.assertEquals:115 expected: but was: TestWasbFsck.testDelete:130->Assert.assertEquals:542->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 expected:<0> but was:<1> TestNativeAzureFileSystemMocked>NativeAzureFileSystemBaseTest.testListDirectory:358->Assert.assertEquals:542->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88 expected:<1> but was:<2> TestNativeAzureFileSystemMocked>NativeAzureFileSystemBaseTest.testRedoRenameFolder:780->Assert.assertTrue:52->Assert.assertTrue:41->Assert.fail:86 null TestNativeAzureFileSystemMocked>NativeAzureFileSystemBaseTest.testStoreDeleteFolder:137->NativeAzureFileSystemBaseTest.assertPathDoesNotExist:87->Assert.fail:88 inner file: unexpectedly found wasb://mockcontai...@mockaccount.blob.core.windows.net/user/fabbri/fork-2/testStoreDeleteFolder/innerFile as FileStatus{path=wasb://mockcontai...@mockaccount.blob.core.windows.net/user/fabbri/fork-2/testStoreDeleteFolder/innerFile; isDirectory=
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165549#comment-16165549 ] Aaron Fabbri commented on HADOOP-14553: --- Coming back to test and review this again [~ste...@apache.org]. Have to remember my WASB setup. ;-) Quick question: Why is the container cleanup step a unit test? {noformat} mvn test -Dtest=CleanupTestContainers {noformat} Does that not run automatically in the unit test phase? I assume the other non-ITest extensions of {{AbstractWasbTestBase}} are all mocked tests? > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch, HADOOP-14553-011.patch, > HADOOP-14553-012.patch, HADOOP-14553-014.patch, HADOOP-14553-015.patch, > HADOOP-14553-016.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161518#comment-16161518 ] Hadoop QA commented on HADOOP-14553: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 103 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 6s{color} | {color:green} root generated 0 new + 1281 unchanged - 2 fixed = 1281 total (was 1283) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 19s{color} | {color:orange} root: The patch generated 123 new + 211 unchanged - 177 fixed = 334 total (was 388) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 44s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 0s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 10s{color} | {color:green} hadoop-azure in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 34s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 98m 56s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.net.TestDNS | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | HADOOP-14553 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12886433/HADOOP-14553-016.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux 8e7587899bd2 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 5ab21df | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13254/artifact/patchprocess/diff-checkstyle-root.txt | | whitespace | https://builds.apache.org/job/PreCommit-HAD
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16161401#comment-16161401 ] John Zhuge commented on HADOOP-14553: - Wow, 16 patches. Thanks [~steve_l] for the hard work! We probably all balked at the size of the patches. Promise I will play with it. BTW, shouldn't you name "testing.md" instead of "testing_azure.md", following the example of AWS and Common? > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch, HADOOP-14553-011.patch, > HADOOP-14553-012.patch, HADOOP-14553-014.patch, HADOOP-14553-015.patch, > HADOOP-14553-016.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16146129#comment-16146129 ] Steve Loughran commented on HADOOP-14553: - bq. {{mvn test}} must run all the tests. This behavior should not change. I think this should be the standard command to run tests for a project. Also, people will not be aware of the change we are making, so please add new arguments to mvn but do not change existing behavior. Maven has two test phases in its lifecycle: the unit tests & the integration tests.This patch keeps all the unit tests in the test phase, editing the POM so that {{mvn test}} and hence jenkins & yetus will run them *always*; they're now really "unit" tests. In contrast: The integration tests are meant to be that: the system integration tests. It kicks off after the packaging is done and the unit tests are passed, and lets you do more stuff [related to testing with/against real systems.|http://maven.apache.org/surefire/maven-failsafe-plugin/]. I think we'll probably be reviewing all the object stores' tests and split them off the same way, as and when someone (else) sits down to make test runs finish in single digits of minutes. The good news: # run {{mvn verify}} and everything is run, unit and integration. That happens today on any module, even those without any integration # make the test review process "declare that you executed mvn verify" and the process ensures all tests get run # yetus and jenkins will run the "mvn test" suite on every patch which touches hadoop-common. This reduces the risk of other changes breaking things. # You can still run individual tests in your IDE, which is generally the fastest way to get a feature/test working. # Even though the IT test phase is halted if the unit tests fail, you can bypass all the unit tests just by making up an nonexistent test {{mvn verify -Dtest=moo -Dit.test=ITestSomething}} This is the evolving best practise in the project related to unit & system testing; expect more of, especially in modules I'm involved in. bq. 2) The Results summary is no longer consolidated. For example, when you run mvn -T 1C -Dparallel-tests clean verify, at the end it looks like only a few tests were run, but if you scroll up in the console output you see that there were several runs and reports. Lets summarize the results in a single line at the end. This is because there are flour test runs: the sequential and parallel unit test runs, the parallel integration tests and the final set of IT tests which don't parallelise. Each test run produces its own list of results in the console output, and its own set of XML docs. target/surefire-reports collect the surefire ones (unit) and target/failsafe-reports for the failsafe ones. It's all the same XML format developed for back in 2000 and this JUnit stuff was new. The way to coalesce the results is to merge the XML docs, usually with XSLT, to produce HTML site doc, or whatever jenkins does. Now if you want console output, that's probably doable too...I think there's some in Stack overflow for an task to do the conversion, albeit after everything. {code} # the unit tests mvn surefire-report:report-only # integration tests mvn surefire-report:failsafe-report-only {code} These add some new html files in target/site with report summaries If you want both reports, the azure markdown and the images needed to make the test reports look good: {code} mvn site:site surefire-report:failsafe-report-only surefire-report:report {code} bq. 3) The tests currently marked scale are not all scale tests. Several are functional tests, like those in ITestBlockBlobInputStream.java. It is important for these tests to be run prior to each check-in. Unless you have added new scale tests (I did not look at the history of every scale test), all of the tests need to be run before check-in when we run mvn test. Maybe the way to view them is "scaleable" tests: tests which can run at a small scale & configured to run at larger scales if the caller wants it, but will always run. I'll see what I can do there. > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch, HADOOP-14553-011.patch, > HADOOP-14553-012.patch, HADOOP-14553-014.patch, H
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16146119#comment-16146119 ] Hadoop QA commented on HADOOP-14553: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 103 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 47s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 6s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 17s{color} | {color:red} hadoop-common in trunk failed. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 45s{color} | {color:green} root generated 0 new + 1282 unchanged - 2 fixed = 1282 total (was 1284) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 12s{color} | {color:orange} root: The patch generated 123 new + 212 unchanged - 177 fixed = 335 total (was 389) {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 17s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 58s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 56s{color} | {color:green} hadoop-azure in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 6s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}101m 9s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ipc.TestIPC | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HADOOP-14553 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12884320/HADOOP-14553-015.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux 6cf8fd876c0a 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 63fc1b0 | | Default Java | 1.8.0_144 | | mvnsite | https://builds.apache.org/job/PreCommit-HADOOP-Build/13123/artifact/patchprocess/branch-mvnsite-hadoop-common-project_hadoop-common.txt | | findbugs | v3.1.0-RC1 | | checkstyle
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131213#comment-16131213 ] Thomas Marquardt commented on HADOOP-14553: --- It is great to see the work that has been done here! We can now run the tests in half the time or less! This is a *huge* improvement to the engineering process! Thanks for enabling the tests to be run in parallel. Here are my comments on the change: 1) *mvn test* must run all the tests. This behavior should not change. I think this should be the standard command to run tests for a project. Also, people will not be aware of the change we are making, so please add new arguments to mvn but do not change existing behavior. 2) The Results summary is no longer consolidated. For example, when you run *mvn -T 1C -Dparallel-tests clean verify*, at the end it looks like only a few tests were run, but if you scroll up in the console output you see that there were several runs and reports. Lets summarize the results in a single line at the end. 3) The tests currently marked scale are not *all* scale tests. Several are functional tests, like those in ITestBlockBlobInputStream.java. It is important for these tests to be run prior to each check-in. Unless you have added new scale tests (I did not look at the history of every scale test), all of the tests need to be run before check-in when we run *mvn test*. 4) It took me 12 minutes to run *mvn -T 1C clean verify -Dscale*. Two things: i) the Results summary looks almost identical to the output from *mvn -T 1C clean verify* except the latter had 156 skipped tests in the 2nd set of results. ii) it looks like it is running the same tests that are run when you don't include -Dscale: *mvn -T 1C clean verify -Dscale* Tests run: 214, Failures: 0, Errors: 0, Skipped: 35 Tests run: 519, Failures: 0, Errors: 0, Skipped: 120 *mvn -T 1C clean verify* Tests run: 214, Failures: 0, Errors: 0, Skipped: 35 Tests run: 519, Failures: 0, Errors: 0, Skipped: 156 4) Before this patch, the test results summary read *Tests run 775, Failures: 0, Errors: 0, Skipped: 155*. It appears that some tests were removed, as the total Tests run is no longer 775. Here are the results I had for different commands: *mvn test* Tests run: 214, Failures: 0, Errors: 0, Skipped: 35 Total time: 01:23 min *mvn -T 1C clean verify* Tests run: 214, Failures: 0, Errors: 0, Skipped: 35 Tests run: 519, Failures: 0, Errors: 0, Skipped: 156 Total time: 21:36 min *mvn -T 1C -Dparallel-tests clean verify* Tests run: 213, Failures: 0, Errors: 0, Skipped: 35 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 Tests run: 323, Failures: 0, Errors: 0, Skipped: 98 Tests run: 150, Failures: 0, Errors: 0, Skipped: 58 Total time: 07:12 min *mvn -T 1C clean verify -Dscale* Tests run: 214, Failures: 0, Errors: 0, Skipped: 35 Tests run: 519, Failures: 0, Errors: 0, Skipped: 120 Total time: 12:14 min I realize this is a big patch and we want to commit it, as it is expensive to refresh. As long as we can fix 1) and determine why the total tests run has been reduced from the former 775 tests, we can commit this and continue working on it. Thanks! > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch, HADOOP-14553-011.patch, > HADOOP-14553-012.patch, HADOOP-14553-014.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with differ
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129340#comment-16129340 ] Hadoop QA commented on HADOOP-14553: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 101 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 31s{color} | {color:green} root generated 0 new + 1315 unchanged - 2 fixed = 1315 total (was 1317) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 15s{color} | {color:orange} root: The patch generated 123 new + 212 unchanged - 178 fixed = 335 total (was 390) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 18s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 10s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 17s{color} | {color:green} hadoop-azure in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 50s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 98m 52s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ipc.TestRPC | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HADOOP-14553 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12882179/HADOOP-14553-014.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux f00f3e70038e 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 1455306 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13049/artifact/patchprocess/diff-checkstyle-root.txt | | whitespace | https://builds.apache.org/job/PreCommit-HAD
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129029#comment-16129029 ] Hadoop QA commented on HADOOP-14553: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 101 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 10s{color} | {color:green} root generated 0 new + 1315 unchanged - 2 fixed = 1315 total (was 1317) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 9s{color} | {color:orange} root: The patch generated 154 new + 212 unchanged - 178 fixed = 366 total (was 390) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 10s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 12s{color} | {color:green} hadoop-azure in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 34s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 92m 16s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ipc.TestRPC | | | hadoop.security.TestKDiag | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HADOOP-14553 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12882143/HADOOP-14553-012.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux 772dfdde591a 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 588c190 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13046/artifact/patchprocess/diff-checkstyle-root.txt | | whitespace | https://bu
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128986#comment-16128986 ] Hadoop QA commented on HADOOP-14553: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 101 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 52s{color} | {color:green} root generated 0 new + 1315 unchanged - 2 fixed = 1315 total (was 1317) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 11s{color} | {color:orange} root: The patch generated 154 new + 212 unchanged - 178 fixed = 366 total (was 390) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 7s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 56s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 49s{color} | {color:green} hadoop-azure in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 33s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 91m 33s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ipc.TestRPC | | | hadoop.security.TestKDiag | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HADOOP-14553 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12882142/HADOOP-14553-011.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux e31ba0dac4ad 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 588c190 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13045/artifact/patchprocess/diff-checkstyle-root.txt | | whitespace | https://bu
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128856#comment-16128856 ] Steve Loughran commented on HADOOP-14553: - Test time overall; 13:07, with half the time coming from the not-yet-parallelised test suites Results : Tests run: 319, Failures: 0, Errors: 0, Skipped: 98 [INFO] [INFO] --- maven-failsafe-plugin:2.17:integration-test (sequential-integration-tests) @ hadoop-azure --- [INFO] Failsafe report directory: /Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-azure/target/failsafe-reports {code} Running org.apache.hadoop.fs.azure.ITestBlobDataValidation Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.982 sec - in org.apache.hadoop.fs.azure.ITestBlobDataValidation Running org.apache.hadoop.fs.azure.ITestBlockBlobInputStream Tests run: 21, Failures: 0, Errors: 0, Skipped: 21, Time elapsed: 0.989 sec - in org.apache.hadoop.fs.azure.ITestBlockBlobInputStream Running org.apache.hadoop.fs.azure.ITestFileSystemOperationsExceptionHandlingMultiThreaded Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.407 sec - in org.apache.hadoop.fs.azure.ITestFileSystemOperationsExceptionHandlingMultiThreaded Running org.apache.hadoop.fs.azure.ITestFileSystemOperationsWithThreads Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 228.507 sec - in org.apache.hadoop.fs.azure.ITestFileSystemOperationsWithThreads Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAuthorizationWithOwner Tests run: 27, Failures: 0, Errors: 0, Skipped: 27, Time elapsed: 3.862 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAuthorizationWithOwner Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemConcurrencyLive Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.435 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemConcurrencyLive Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemLive Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 274.746 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemLive Running org.apache.hadoop.fs.azure.ITestOutOfBandAzureBlobOperationsLive Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.224 sec - in org.apache.hadoop.fs.azure.ITestOutOfBandAzureBlobOperationsLive Running org.apache.hadoop.fs.azure.ITestWasbRemoteCallHelper Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 1.32 sec - in org.apache.hadoop.fs.azure.ITestWasbRemoteCallHelper {code} > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch, HADOOP-14553-011.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc)
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124613#comment-16124613 ] Steve Loughran commented on HADOOP-14553: - OK, I'll apply it & see what it does on macos too. 8:30 is a good number. It will improve your life significantly > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124090#comment-16124090 ] Esfandiar Manii commented on HADOOP-14553: -- Looked at Allen's work. The fix is only required for your patch. Its building for windows without your change and the only reason is the embedded JS code :) > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123802#comment-16123802 ] Esfandiar Manii commented on HADOOP-14553: -- around 8 mins and 30 seconds :) > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123785#comment-16123785 ] Steve Loughran commented on HADOOP-14553: - Talk to [~aw] about that, he's trying to get a patch which works for windows, linux, macos, windows; patch on HADOOP-14696. How long did the parallel test run take BTW? > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123728#comment-16123728 ] Esfandiar Manii commented on HADOOP-14553: -- Ok, found the solution, instead of direct placeholders we can use project.getProperty and that will ensure backslashes are interpreted correctly. Please update your JavaScript to the following and everything works smoothly. {code:javascript} {code} > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123593#comment-16123593 ] Esfandiar Manii commented on HADOOP-14553: -- All the three have issues since the paths are including one backslash for file separator and JavaScript interprets them as escape sequence characters. So whichever code is taking the JavaScript string between the target tags in the pom.xml and replacing those ${test.build.data}, ... variables with their values, should make sure that test.build.data has two back slashes instead of one. > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123070#comment-16123070 ] Steve Loughran commented on HADOOP-14553: - HADOOP-14696 is the JIRA we can work on for paths. Which dir did it miscreate in azure? hadoop.tmp.dir, I suspect, as that's the one which different bits of code think lives on local and remote filesystems > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122497#comment-16122497 ] Esfandiar Manii commented on HADOOP-14553: -- Actually "${test.build.data.replace('\\', '')}", "${test.build.dir.replace('\\', '')}", "${hadoop.tmp.dir.replace('\\', '')}"]; is not working well. It seems it created wrong directories under hadoop-azure. Is there any easy way to print two back slashes through pom.xml configurations? Currently the issue is you are doing var baseDirs = [ "${test.build.data}", "${test.build.dir}", "${hadoop.tmp.dir}"] in windows if test.build.data is f:\a\b in JS you will have baseDirs = [ 'f'] since \a \b are escape sequence chars and the path is invalid in Windows. > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch, HADOOP-14553-010.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122404#comment-16122404 ] Esfandiar Manii commented on HADOOP-14553: -- All the tests passed in windows after I applied a fix to the pom.xml file {code:java} var baseDirs = [ "${test.build.data.replace('\\', '')}", "${test.build.dir.replace('\\', '')}", "${hadoop.tmp.dir.replace('\\', '')}"]; for (var i in baseDirs) { for (var j = 1; j <= ${testsThreadCount}; ++j) { var mkdir = project.createTask("mkdir"); mkdir.setDir(new java.io.File(baseDirs[i], j)); mkdir.perform(); } } {code} {code:java} $ mvn -T 1C -Dparallel-tests clean verify [INFO] Scanning for projects... [INFO] [INFO] Using the MultiThreadedBuilder implementation with a thread count of 8 [INFO] [INFO] [INFO] Building Apache Hadoop Azure support 3.0.0-beta1-SNAPSHOT [INFO] [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-azure --- [INFO] Deleting F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\target [INFO] Deleting F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure (includes = [dependency-reduced-pom.xml], excludes = []) [INFO] [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-azure --- [INFO] Executing tasks main: [mkdir] Created dir: F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\target\test-dir [INFO] Executed tasks [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-azure --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hadoop-azure --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-azure --- [INFO] Compiling 62 source files to F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\target\classes [WARNING] /F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java: Some input files use or override a deprecated API. [WARNING] /F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java: Recompile with -Xlint:deprecation for details. [WARNING] /F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java: F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\src\main\java\org\apache\hadoop\fs\azure\security\TokenUtils.java uses unchecked or unsafe operations. [WARNING] /F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- maven-dependency-plugin:2.10:list (deplist) @ hadoop-azure --- [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hadoop-azure --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 5 resources [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hadoop-azure --- [INFO] Compiling 69 source files to F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\target\test-classes [WARNING] /F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java: F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\src\test\java\org\apache\hadoop\fs\azure\NativeAzureFileSystemBaseTest.java uses or overrides a deprecated API. [WARNING] /F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java: Recompile with -Xlint:deprecation for details. [INFO] [INFO] --- maven-antrun-plugin:1.7:run (create-parallel-tests-dirs) @ hadoop-azure --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-azure --- [INFO] Surefire report directory: F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\target\surefire-reports --- T E S T S --- --- T E S T S --- Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.458 sec - in org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater Running org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.339 sec - in org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAve
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120251#comment-16120251 ] Hadoop QA commented on HADOOP-14553: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 101 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 24s{color} | {color:green} root generated 0 new + 1372 unchanged - 2 fixed = 1372 total (was 1374) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 5s{color} | {color:orange} root: The patch generated 154 new + 212 unchanged - 178 fixed = 366 total (was 390) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 8s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 47s{color} | {color:green} hadoop-azure in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 87m 45s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.security.TestKDiag | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HADOOP-14553 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881033/HADOOP-14553-010.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux b9d3e1161573 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 63cfcb9 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/12992/artifact/patchprocess/diff-checkstyle-root.txt | | whitespace | https://builds.apache.org/job/PreComm
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120068#comment-16120068 ] Steve Loughran commented on HADOOP-14553: - Test results, without scale tests, 13 min. Still slow, but better than before {code} --- T E S T S --- Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus Running org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDistCp Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 6.017 sec - in org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 12.752 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDistCp Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.579 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen Running org.apache.hadoop.fs.azure.ITestAzureFileSystemErrorConditions Tests run: 7, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 23.806 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend Running org.apache.hadoop.fs.azure.ITestBlobTypeSpeedDifference Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.035 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete Running org.apache.hadoop.fs.azure.ITestContainerChecks Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.99 sec - in org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo Running org.apache.hadoop.fs.azure.ITestFileSystemOperationExceptionHandling Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.834 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename Running org.apache.hadoop.fs.azure.ITestFileSystemOperationExceptionMessage Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.655 sec - in org.apache.hadoop.fs.azure.ITestFileSystemOperationExceptionMessage Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAppend Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.711 sec - in org.apache.hadoop.fs.azure.ITestContainerChecks Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAtomicRenameDirList Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.189 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAtomicRenameDirList Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.323 sec - in org.apache.hadoop.fs.azure.ITestBlobTypeSpeedDifference Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemClientLogging Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractEmulator Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 0.75 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractEmulator Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.137 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemClientLogging Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.386 sec - in org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractPageBlobLive Running org.apache.hadoop.fs.azure.ITestNativeAzureFSAuthorizationCaching Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.955 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate Running org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 2.299 sec - in org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite Running org.apache.hadoop.fs.azure.ITestWasbUriAndConfiguration Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.526 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir Running org.apache.hadoop.fs.azure.metrics.ITestAzureFileSystemInstrumentation Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.83 sec
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119960#comment-16119960 ] Steve Loughran commented on HADOOP-14553: - The next iteration of this will include a test class which must be explicitly invoked on the command line via a -Dtest=, which will list and delete all {{wasbtests-}} containers of an account. This can be used to clean up container leakage: {code} 2017-08-09 15:08:54,335 INFO [JUnit-testDeleteContainers]: azure.AbstractWasbTestBase (CleanupTestContainers.java:testDeleteContainers(79)) - Container wasbtests-stevel-1501782115769 URI http://contender.blob.core.windows.net/wasbtests-stevel-1501782115769 2017-08-09 15:08:54,390 INFO [JUnit-testDeleteContainers]: azure.AbstractWasbTestBase (CleanupTestContainers.java:testDeleteContainers(79)) - Container wasbtests-stevel-1501782117324 URI http://contender.blob.core.windows.net/wasbtests-stevel-1501782117324 2017-08-09 15:08:54,444 INFO [JUnit-testDeleteContainers]: azure.AbstractWasbTestBase (CleanupTestContainers.java:testDeleteContainers(79)) - Container wasbtests-stevel-1501782149411 URI http://contender.blob.core.windows.net/wasbtests-stevel-1501782149411 2017-08-09 15:08:54,497 INFO [JUnit-testDeleteContainers]: azure.AbstractWasbTestBase (CleanupTestContainers.java:testDeleteContainers(86)) - Deleted 2436 test containers {code} > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. > *UPDATE August 4, 2017*: Adding a list of requirements to clarify the > acceptance criteria for this JIRA: > # Parallelize test execution > # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii) > IntegrationTests - requires Azure Storage account, executed by engineers > prior to check-in, and if needed, iii) ScaleTests – long running performance > and scalability tests. > # Define configuration profiles to run tests with different settings. Allows > an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and > false. Need to review settings to see what else would benefit. > # Maven commands to run b) and c). Turns out it is not easy to do with > Maven, so we might have to run it multiple times to run with different > configuration settings. > # Document how to add and run tests and the process for contributing to > Apache Hadoop. Steve shared an example at > https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md > > # UnitTests should run in under 2 minutes and IntegrationTests should run in > under 15 minutes, even on slower network connections. (These are rough goals) > # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large > persistent content used repeatedly to expedite test execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114197#comment-16114197 ] Steve Loughran commented on HADOOP-14553: - most of the checkstyles are just the existing style issues (many spurious) being considered as new when the files get renamed. I'll address the new ones, but not the existing, as that'd take the patch even deeper into the tests. I'm trying to parallelise and add some uniformity of base classes & setup, not rework the test cases themselves. > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113308#comment-16113308 ] Hadoop QA commented on HADOOP-14553: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 99 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 42s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m 33s{color} | {color:green} root generated 0 new + 1416 unchanged - 2 fixed = 1416 total (was 1418) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 28s{color} | {color:orange} root: The patch generated 150 new + 203 unchanged - 177 fixed = 353 total (was 380) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 40s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 19s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 59s{color} | {color:green} hadoop-azure in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 49s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}109m 58s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HADOOP-14553 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12880255/HADOOP-14553-009.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux 0a6e579197b7 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 293c74a | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/12944/artifact/patchprocess/diff-checkstyle-root.txt | | whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/12944/artifact/patchprocess/whitespace-eol.txt | | Te
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112972#comment-16112972 ] Steve Loughran commented on HADOOP-14553: - Note that as this lifted the parallel test code from hadoop-aws, it will depend on the final fix for HADOOP-14696 being copied over to actually work on windows > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112954#comment-16112954 ] Steve Loughran commented on HADOOP-14553: - patch 009 # rebased onto trunk # added a unified base class, {{AbstractWasbTestWithTimeout}}, which just extends Assert with test timeout and thread naming. This is now the base class for all tests, even those which don't extend {{AbstractWasbTestBase}} (which itself extends the class) # {{AbstractWasbTestBase}} structured better for subclassing, including creating the configuration for test accounts # {{AbstractAzureScaleTest}} direct child of {{AbstractWasbTestBase}} (i.e cut any intermediate IntegrationTest class) # {{ITestBlockBlobInputStream}} is a scale test, as is {{ITestAzureNativeContractDistCp}}. This moves the core bandwidth-heavy tests into the scale group # Also: misc other cleanups, fix javadocs so yetus won't complain. > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch, > HADOOP-14553-009.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083125#comment-16083125 ] Hadoop QA commented on HADOOP-14553: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 90 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 26s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 35s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 35s{color} | {color:red} root generated 2 new + 1356 unchanged - 2 fixed = 1358 total (was 1358) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 3s{color} | {color:orange} root: The patch generated 145 new + 206 unchanged - 172 fixed = 351 total (was 378) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 11 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 11s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 8s{color} | {color:green} hadoop-azure in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 40s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 99m 24s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.security.TestRaceWhenRelogin | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HADOOP-14553 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12876701/HADOOP-14553-008.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux 6e2cf2d80b1d 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / d670c3a | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/12763/artifact/patchprocess/diff-compile-javac-root.txt | | checkstyle | https://builds.apache.org/job/PreCommi
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16082120#comment-16082120 ] Steve Loughran commented on HADOOP-14553: - * The checkstyle complaints are arising not from new code, but from the files being renamed. Hence "121 new + 209 unchanged - 147 fixed". This is fixing 26 of the complaints from before. That should be enough for now, with the later improvements doing the rest. * Javac is on about deprecation of {{cleanup(Log,Closeable...) in IOUtils}}; it'd need a move to SLF4J for that, which could/should be done across all of the project except those tests which get at the logs. > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch, > HADOOP-14553-006.patch, HADOOP-14553-007.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078205#comment-16078205 ] Hadoop QA commented on HADOOP-14553: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 88 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 7s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 7s{color} | {color:red} root generated 2 new + 1357 unchanged - 2 fixed = 1359 total (was 1359) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 1s{color} | {color:orange} root: The patch generated 121 new + 209 unchanged - 147 fixed = 330 total (was 356) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 36s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 11 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 4s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 7s{color} | {color:green} hadoop-azure in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 39s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 84m 26s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.security.TestKDiag | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HADOOP-14553 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12876085/HADOOP-14553-007.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux 16c242a1a673 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 8153fe2 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/12731/artifact/patchprocess/diff-compile-javac-root.txt | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-B
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077847#comment-16077847 ] Steve Loughran commented on HADOOP-14553: - To be precise, it was the FS contract base test subclasses which were timing out; this means the timeout logic in the hadoop-common base class needs to be made overrideable. This would be of benefit for other object store tests too {code} testWriteReadAndDeleteEmptyFile(org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive) Time elapsed: 30.006 sec <<< ERROR! java.lang.Exception: test timed out after 3 milliseconds at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:48 {code} > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077314#comment-16077314 ] Hadoop QA commented on HADOOP-14553: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s{color} | {color:blue} Docker mode activated. {color} | | {color:green} 1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green} 1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 73 new or modified test files. {color} | | {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 13m 13s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} compile {color} | {color:green} 0m 18s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} findbugs {color} | {color:green} 0m 27s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 0m 13s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} compile {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} javac {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 15s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 123 new 193 unchanged - 114 fixed = 316 total (was 307) {color} | | {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 16 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s{color} | {color:red} The patch 1 line(s) with tabs. {color} | | {color:green} 1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green} 1{color} | {color:green} findbugs {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} unit {color} | {color:green} 0m 42s{color} | {color:green} hadoop-azure in the patch passed. {color} | | {color:green} 1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 19m 7s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HADOOP-14553 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12875991/HADOOP-14553-005.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle | | uname | Linux f13b37e8bf35 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 7576a68 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/12725/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt | | whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/12725/artifact/patchprocess/whitespace-eol.txt | | whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/12725/artifact/patchprocess/whitespace-tabs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/12725/testReport/ | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/12725/console | | Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Add (parallelized) integration tes
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076504#comment-16076504 ] Steve Loughran commented on HADOOP-14553: - {code} -public class TestNativeAzureFileSystemContractMocked extends +/** + * Mocked testing of FileSystemContractBaseTest. + * This isn't an IT, but making it so makes it a lot faster for now. + */ +public class ITestNativeAzureFileSystemContractMocked extends {code} bq. why is it faster as ITest? its not that the test finishes fast, it's just as something slow, running it in parallel meant the test run took less time. I want to do another iteration of this and * rename Test* which requires credentails to being an ITest —but just list them in the sequential section * leave the other tests alone * change the test profile in the POM to run the normal test profile without looking for an auth-keys file Goal: Jenkins/yetus to run the unit tests; move everything else to integration tests sooner rather than later, and so allow for 1+ followup which parallelised the remaining tests, or in the case of the big native test suite, split it up. Regarding commonality between S3A test runner and the new stuff, yes, I did copy and past S3ATestUtils in, which you would have noticed. Trouble is: I don't know what commonality we really have right now. > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075717#comment-16075717 ] Aaron Fabbri commented on HADOOP-14553: --- Integration tests passed in US West via `mvn -T 1C clean verify`: {noformat} Tests run: 323, Failures: 0, Errors: 0, Skipped: 70 [INFO] [INFO] --- maven-enforcer-plugin:1.4.1:enforce (depcheck) @ hadoop-azure --- [INFO] [INFO] --- maven-failsafe-plugin:2.17:verify (default) @ hadoop-azure --- [INFO] Failsafe report directory: /Users/fabbri/Code/hadoop/hadoop-tools/hadoop-azure/target/failsafe-reports [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 51:46 min (Wall Clock) {noformat} I am +1 (nonbinding) provided we link some followup JIRAs for the common code refactoring and any other work you think needs to be done next. > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075714#comment-16075714 ] Aaron Fabbri commented on HADOOP-14553: --- Thank you for doing this work [~ste...@apache.org]. I'm running the tests in US West now. Overall this patch contains a lot of goodness. Besides the ITest refactoring I see a lot of improved test cleanup and some reduced code duplication. I wonder if some of the common code pasted from S3A could be factored out (huge files and test utils classes). That said, this is an improvement overall so I'd be fine with some lower-priority JIRAs to follow up on factoring out the common hugefiles and testutils stuff. {noformat} + + 1 + + false + ${maven-surefire-plugin.argLine} -DminiClusterDedicatedDirs=true {noformat} Is this a temporary workaround for a unit-test parallelization issue? {noformat} +public class ITestNativeAzureFileSystemContractLive extends - @Before +@Before public void setUp() throws Exception { {noformat} Formatting nit at Before annotation. {noformat} -public class TestNativeAzureFileSystemContractMocked extends +/** + * Mocked testing of FileSystemContractBaseTest. + * This isn't an IT, but making it so makes it a lot faster for now. + */ +public class ITestNativeAzureFileSystemContractMocked extends {noformat} Just curious, why is it faster as ITest? > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071027#comment-16071027 ] Hadoop QA commented on HADOOP-14553: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color} | {color:red} HADOOP-14553 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HADOOP-14553 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12875303/HADOOP-14553-004.patch | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/12690/console | | Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070996#comment-16070996 ] Hadoop QA commented on HADOOP-14553: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color} | {color:red} HADOOP-14553 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HADOOP-14553 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12875303/HADOOP-14553-004.patch | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/12689/console | | Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch, > HADOOP-14553-003.patch, HADOOP-14553-004.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070670#comment-16070670 ] Steve Loughran commented on HADOOP-14553: - Tested, Azure ireland. Test run time now down to 23 minutes, with some obvious tests taking 1-5 minutes each: parallelise them and most of the heavy lifting is done. But two of those are the native tests: metadata heavy, but a long set of operations {code} --- --- T E S T S --- Running org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 96.646 sec - in org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.385 sec - in org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater Running org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.829 sec - in org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.234 sec - in org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage Running org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.482 sec - in org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions Running org.apache.hadoop.fs.azure.TestBlobDataValidation Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.656 sec - in org.apache.hadoop.fs.azure.TestBlobDataValidation Running org.apache.hadoop.fs.azure.TestBlobMetadata Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.83 sec - in org.apache.hadoop.fs.azure.TestBlobMetadata Running org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.547 sec - in org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference Running org.apache.hadoop.fs.azure.TestContainerChecks Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.368 sec - in org.apache.hadoop.fs.azure.TestContainerChecks Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.276 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling Running org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.692 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded Running org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 416.005 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.004 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization Tests run: 21, Failures: 0, Errors: 0, Skipped: 21, Time elapsed: 3.397 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner Tests run: 24, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: 4.036 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.884 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.062 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.883 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.862 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 355.53 sec - in org.apache.hadoop.fs.azure.TestNa
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070617#comment-16070617 ] Steve Loughran commented on HADOOP-14553: - Patch 004 # parallelise some more with better isolation # Clean up some of the tests with: try with resources, use of ContracTestUtil asserts about the state of the FS. Metric of work, the non-scale IT tests run in under 3 minutes {code} T E S T S --- Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode Running org.apache.hadoop.fs.azure.integration.ITestFileSystemOperationExceptionMessage Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete Running org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.643 sec - in org.apache.hadoop.fs.azure.integration.ITestFileSystemOperationExceptionMessage Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAppend Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 22.088 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemClientLogging Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 22.189 sec - in org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractEmulator Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 1.068 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractEmulator Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.535 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractMocked Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 2.207 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractMocked Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractPageBlobLive Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.855 sec - in org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.192 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemClientLogging Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemMocked Running org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.493 sec - in org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.95 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemMocked Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 58.926 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 30.771 sec - in org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.445 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 68.468 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.282 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 82.534 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 94.839 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 87.913 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractPageBlobLive Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.172 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemAppend Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 106.015 sec - in org.apache.hadoop.fs.azure.ITestN
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16061562#comment-16061562 ] Hadoop QA commented on HADOOP-14553: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 35 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 10s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 61 new + 23 unchanged - 25 fixed = 84 total (was 48) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 17s{color} | {color:green} hadoop-azure in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 18m 51s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HADOOP-14553 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12874320/HADOOP-14553-002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle | | uname | Linux cd21834d60a1 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / c6bd73c | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/12618/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt | | whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/12618/artifact/patchprocess/whitespace-eol.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/12618/testReport/ | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/12618/console | | Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Com
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16061505#comment-16061505 ] Steve Loughran commented on HADOOP-14553: - patch 002 There's some careful parallelization of a few tests, enough to show that it does work if you do it case by case, the use of shared/hard coded paths being the enemy. I've moved over all the contract tests as we know they parallize well once you set up the contract to return fork-specific paths. I've uprated a limited set oftests so far, enough to show things work and to achieve measurable speeups already. Here is a test where the total time to run these tests is 3 minutes, rather than the 5 minutes it would take otherwise. Tagging the huge files and read and seek as scale tests means that non-scale test runs would be even faster. Moved: * all the contract tests in {{org.apache.hadoop.fs.azure.contract}}, tweaking {{NativeAzureFileSystemContract}} to use the fork ID sysprop to generate unique paths. * {{ITestFileSystemOperationExceptionMessage}}. This was taking so long as it was needlessly subclassing {{NativeAzureFileSystemBaseTest}} and its 50 tests. Moving the single method to a new IT test fixed that. * {{ITestNativeAzureFileSystemContractLive}}: again, known to parallelize. * {{ITestReadAndSeekPageBlobAfterWrite}} self-contained, slow (it's been tagged as a scale test), and validates the ability for pageblob paths to be per-fork too. Added * docs in `test_azure.md` {code} > mvit -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dtest=none --- T E S T S --- Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDistCp Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 11.567 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractAppend Running org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.605 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractOpen Running org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.765 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDelete Running org.apache.hadoop.fs.azure.integration.ITestFileSystemOperationExceptionMessage Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.948 sec - in org.apache.hadoop.fs.azure.integration.ITestFileSystemOperationExceptionMessage Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.087 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractRename Running org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive Running org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.702 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractCreate Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.769 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractMkdir Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.872 sec - in org.apache.hadoop.fs.azure.integration.ITestAzureHugeFiles Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 62.163 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractSeek Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 79.653 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractGetFileStatus Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 98.776 sec - in org.apache.hadoop.fs.azure.ITestNativeAzureFileSystemContractLive Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 150.19 sec - in org.apache.hadoop.fs.azure.contract.ITestAzureNativeContractDistCp Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 165.022 sec - in org.apache.hadoop.fs.azure.ITestReadAndSeekPageBlobAfterWrite Results : Tests run: 138, Failures: 0, Errors: 0, Skipped: 6 [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 03:14 min (Wall Clock) [INFO] Finished at: 201
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060921#comment-16060921 ] Steve Loughran commented on HADOOP-14553: - this patch includes the testing.md file I'd copied from s3; thought I'd kept that out pending reworking, but clearly not. will revise checkstyle complaints mostly spurious (naming of {{_1MB}} constants, tests given explicit {{test_010_upload()}} names to ensure ordering in junit,... And complaining about .* static imports which is compliant with our guidelines. > Add (parallelized) integration tests to hadoop-azure > > > Key: HADOOP-14553 > URL: https://issues.apache.org/jira/browse/HADOOP-14553 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14553-001.patch > > > The Azure tests are slow to run as they are serialized, as they are all > called Test* there's no clear differentiation from unit tests which Jenkins > can run, and integration tests which it can't. > Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize > (which includes having separate paths for every test suite). The code in > hadoop-aws's POM &c show what to do. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org