[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16165663#comment-16165663
]
Aaron Fabbri commented on HADOOP-14553:
---------------------------------------
Just noticed your comment above which answers one of my questions
(CleanupTestContainers must be explicitly specified to run).
I applied the v16 patch and ran all tests in hadoop-azure and hit a bunch of
failures.. Not sure if I've done something stupid but this is what it looks
like:
{noformat}
Failed tests:
TestNativeAzureFileSystemConcurrency.testNoTempBlobsVisible:100->Assert.assertEquals:144->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88
expected:<wasb://[email protected]/inProgress>
but was:<wasb://mockAccount.blob.core.windows.net/mockContainer/inProgress>
TestNativeAzureFileSystemContractMocked>FileSystemContractBaseTest.testListStatusRootDir:866->FileSystemContractBaseTest.assertListStatusFinds:896
Path
wasb://[email protected]/FileSystemContractBaseTest
not found in directory
wasb://[email protected]/:FileStatus{path=wasb://mockAccount.blob.core.windows.net/mockContainer/FileSystemContractBaseTest;
isDirectory=false; length=2048; replication=1; blocksize=536870912;
modification_time=1505351566081; access_time=0; owner=fabbri; group=supergroup;
permission=rw-r--r--; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false}
TestNativeAzureFileSystemContractMocked>FileSystemContractBaseTest.testListStatus:330
expected:<1> but was:<2>
TestNativeAzureFileSystemContractMocked>FileSystemContractBaseTest.testLSRootDir:856->FileSystemContractBaseTest.assertListFilesFinds:881
Path
wasb://[email protected]/FileSystemContractBaseTest
not found in directory
wasb://[email protected]/:LocatedFileStatus{path=wasb://mockAccount.blob.core.windows.net/mockContainer/FileSystemContractBaseTest;
isDirectory=false; length=2048; replication=1; blocksize=536870912;
modification_time=1505351566450; access_time=0; owner=fabbri; group=supergroup;
permission=rw-r--r--; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false}
TestNativeAzureFileSystemContractMocked>FileSystemContractBaseTest.testDeleteRecursively:452
File doesn't exist
TestNativeAzureFileSystemFileNameCheck.testWasbFsck:120->Assert.assertTrue:52->Assert.assertTrue:41->Assert.fail:86
null
TestNativeAzureFileSystemMocked>NativeAzureFileSystemBaseTest.testDeepFileCreation:228->NativeAzureFileSystemBaseTest.testDeepFileCreationBase:218->NativeAzureFileSystemBaseTest.assertPathDoesNotExist:87->Assert.fail:88
deleted file: unexpectedly found deep/file/creation/test as
FileStatus{path=wasb://[email protected]/user/fabbri/deep/file/creation/test;
isDirectory=false; length=0; replication=1; blocksize=536870912;
modification_time=1505351569092; access_time=0; owner=fabbri; group=supergroup;
permission=rw-r--r--; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false}
TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testListStatus:312
expected:<1> but was:<2>
TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testListStatusFilterWithSomeMatches:369
null
TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusSomeMatchesInDirectories:426
expected:<2> but was:<4>
TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testGlobStatusWithMultipleWildCardMatches:451
expected:<4> but was:<8>
TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testDeleteRecursively:777
File doesn't exist
TestOutOfBandAzureBlobOperations.testImplicitFolderDeleted:94->Assert.assertFalse:74->Assert.assertFalse:64->Assert.assertTrue:41->Assert.fail:86
null
TestOutOfBandAzureBlobOperations.testImplicitFolderListed:80->Assert.assertEquals:144->Assert.assertEquals:115
expected:</[]root/b> but was:</[mockContainer/]root/b>
TestWasbFsck.testDelete:130->Assert.assertEquals:542->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88
expected:<0> but was:<1>
TestNativeAzureFileSystemMocked>NativeAzureFileSystemBaseTest.testListDirectory:358->Assert.assertEquals:542->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88
expected:<1> but was:<2>
TestNativeAzureFileSystemMocked>NativeAzureFileSystemBaseTest.testRedoRenameFolder:780->Assert.assertTrue:52->Assert.assertTrue:41->Assert.fail:86
null
TestNativeAzureFileSystemMocked>NativeAzureFileSystemBaseTest.testStoreDeleteFolder:137->NativeAzureFileSystemBaseTest.assertPathDoesNotExist:87->Assert.fail:88
inner file: unexpectedly found
wasb://[email protected]/user/fabbri/fork-2/testStoreDeleteFolder/innerFile
as
FileStatus{path=wasb://[email protected]/user/fabbri/fork-2/testStoreDeleteFolder/innerFile;
isDirectory=false; length=0; replication=1; blocksize=536870912;
modification_time=1505351584792; access_time=0; owner=fabbri; group=supergroup;
permission=rw-r--r--; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false}
TestNativeAzureFileSystemMocked>NativeAzureFileSystemBaseTest.testRename:273->Assert.assertTrue:52->Assert.assertTrue:41->Assert.fail:86
null
Tests in error:
TestNativeAzureFileSystemContractMocked>FileSystemContractBaseTest.testRenameDirectoryMoveToExistingDirectory:540->FileSystemContractBaseTest.rename:616
» Azure
TestNativeAzureFileSystemContractMocked>FileSystemContractBaseTest.testRenameDirectoryAsExistingDirectory:573->FileSystemContractBaseTest.rename:616
» Azure
TestNativeAzureFileSystemContractMocked>FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed:782->FileSystemContractBaseTest.rename:616
» Azure
TestNativeAzureFileSystemMocked>NativeAzureFileSystemBaseTest.testChineseCharactersFolderRename:441
» Azure
TestNativeAzureFileSystemMocked>NativeAzureFileSystemBaseTest.testRedoRenameFolderInFolderListingWithZeroByteRenameMetadata:954
» FileNotFound
TestNativeAzureFileSystemMocked>NativeAzureFileSystemBaseTest.testRedoRenameFolderInFolderListing:839
» FileNotFound
TestNativeAzureFileSystemMocked>NativeAzureFileSystemBaseTest.testUriEncoding:392
» Azure
TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testRenameDirectoryAsEmptyDirectory:1033->FSMainOperationsBaseTest.rename:1154
» Azure
TestNativeAzureFileSystemOperationsMocked>FSMainOperationsBaseTest.testRenameDirectoryAsNonExistentDirectory:984->FSMainOperationsBaseTest.doTestRenameDirectoryAsNonExistentDirectory:1001->FSMainOperationsBaseTest.rename:1154
» Azure
TestOutOfBandAzureBlobOperations.testCreatingDeepFileCreatesExplicitFolder:141
» FileAlreadyExists
TestNativeAzureFileSystemMocked>NativeAzureFileSystemBaseTest.testRedoRenameFolderRenameInProgress:906
» FileNotFound
TestNativeAzureFileSystemMocked>NativeAzureFileSystemBaseTest.testRenameFolder:312
» Azure
TestNativeAzureFileSystemMocked>NativeAzureFileSystemBaseTest.testRenameImplicitFolder:285
» Azure
Tests run: 217, Failures: 19, Errors: 13, Skipped: 37
{noformat}
I will try to re-apply and retest.
> Add (parallelized) integration tests to hadoop-azure
> ----------------------------------------------------
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch,
> HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch,
> HADOOP-14553-009.patch, HADOOP-14553-010.patch, HADOOP-14553-011.patch,
> HADOOP-14553-012.patch, HADOOP-14553-014.patch, HADOOP-14553-015.patch,
> HADOOP-14553-016.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM &c show what to do.
> *UPDATE August 4, 2017*: Adding a list of requirements to clarify the
> acceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii)
> IntegrationTests - requires Azure Storage account, executed by engineers
> prior to check-in, and if needed, iii) ScaleTests – long running performance
> and scalability tests.
> # Define configuration profiles to run tests with different settings. Allows
> an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and
> false. Need to review settings to see what else would benefit.
> # Maven commands to run b) and c). Turns out it is not easy to do with
> Maven, so we might have to run it multiple times to run with different
> configuration settings.
> # Document how to add and run tests and the process for contributing to
> Apache Hadoop. Steve shared an example at
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> # UnitTests should run in under 2 minutes and IntegrationTests should run in
> under 15 minutes, even on slower network connections. (These are rough goals)
> # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large
> persistent content used repeatedly to expedite test execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
