[jira] [Updated] (HADOOP-16205) Backporting ABFS driver from trunk to branch 2.0
[ https://issues.apache.org/jira/browse/HADOOP-16205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-16205: - Description: Back porting ABFS driver from trunk to 2.0 (was: Commit the core code of the ABFS connector (HADOOP-15407) to its development branch) > Backporting ABFS driver from trunk to branch 2.0 > > > Key: HADOOP-16205 > URL: https://issues.apache.org/jira/browse/HADOOP-16205 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.0.0-alpha >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > > Back porting ABFS driver from trunk to 2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16205) Backporting ABFS driver from trunk to branch 2.0
[ https://issues.apache.org/jira/browse/HADOOP-16205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-16205: - Target Version/s: 2.0.0-alpha (was: 3.2.0) > Backporting ABFS driver from trunk to branch 2.0 > > > Key: HADOOP-16205 > URL: https://issues.apache.org/jira/browse/HADOOP-16205 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.0.0-alpha >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > > Back porting ABFS driver from trunk to 2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16205) Backporting ABFS driver from trunk to branch 2.0
[ https://issues.apache.org/jira/browse/HADOOP-16205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-16205: - Affects Version/s: (was: 3.2.0) 2.0.0-alpha > Backporting ABFS driver from trunk to branch 2.0 > > > Key: HADOOP-16205 > URL: https://issues.apache.org/jira/browse/HADOOP-16205 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.0.0-alpha >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > > Commit the core code of the ABFS connector (HADOOP-15407) to its development > branch -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16205) Backporting ABFS driver from trunk to branch 2.0
[ https://issues.apache.org/jira/browse/HADOOP-16205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-16205: - Fix Version/s: (was: HADOOP-15407) > Backporting ABFS driver from trunk to branch 2.0 > > > Key: HADOOP-16205 > URL: https://issues.apache.org/jira/browse/HADOOP-16205 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > > Commit the core code of the ABFS connector (HADOOP-15407) to its development > branch -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16205) Backporting ABFS driver from trunk to branch 2.0
[ https://issues.apache.org/jira/browse/HADOOP-16205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii reassigned HADOOP-16205: Assignee: Esfandiar Manii (was: Da Zhou) > Backporting ABFS driver from trunk to branch 2.0 > > > Key: HADOOP-16205 > URL: https://issues.apache.org/jira/browse/HADOOP-16205 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Fix For: HADOOP-15407 > > > Commit the core code of the ABFS connector (HADOOP-15407) to its development > branch -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16205) Backporting ABFS driver from trunk to branch 2.0
Esfandiar Manii created HADOOP-16205: Summary: Backporting ABFS driver from trunk to branch 2.0 Key: HADOOP-16205 URL: https://issues.apache.org/jira/browse/HADOOP-16205 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Affects Versions: 3.2.0 Reporter: Esfandiar Manii Assignee: Da Zhou Fix For: HADOOP-15407 Commit the core code of the ABFS connector (HADOOP-15407) to its development branch -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15547) WASB: listStatus performance
[ https://issues.apache.org/jira/browse/HADOOP-15547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549616#comment-16549616 ] Esfandiar Manii commented on HADOOP-15547: -- Thanks [~ste...@apache.org] Could you please submit this to the branch? > WASB: listStatus performance > > > Key: HADOOP-15547 > URL: https://issues.apache.org/jira/browse/HADOOP-15547 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.9.1, 3.0.2 >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Attachments: HADOOP-15547-004.patch, HADOOP-15547-004.patch, > HADOOP-15547.001.patch, HADOOP-15547.002.patch, HADOOP-15547.003.patch > > > The WASB implementation of Filesystem.listStatus is very slow due to O(n!) > algorithm to remove duplicates and uses too much memory due to the extra > conversion from BlobListItem to FileMetadata to FileStatus. It takes over 30 > minutes to list 700,000 files. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15533) Make WASB listStatus messages consistent
[ https://issues.apache.org/jira/browse/HADOOP-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16517334#comment-16517334 ] Esfandiar Manii commented on HADOOP-15533: -- Thanks, [~chris.douglas]! > Make WASB listStatus messages consistent > > > Key: HADOOP-15533 > URL: https://issues.apache.org/jira/browse/HADOOP-15533 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Trivial > Fix For: 2.10.0, 3.2.0, 3.1.1, 3.0.4 > > Attachments: HADOOP-15533-001.patch, HADOOP-15533-branch-2-001.patch > > > - This change make WASB listStatus error messages to be consistent with the > rest of the listStatus error messages. > - Inconsistent error messages cause a few WASB tests to fail only in > branch-2. The test bug was introduced in > "https://issues.apache.org/jira/browse/HADOOP-15506";. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15533) Making WASB listStatus messages consistent
[ https://issues.apache.org/jira/browse/HADOOP-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512765#comment-16512765 ] Esfandiar Manii commented on HADOOP-15533: -- Tested this against a Microsoft test storage account for both branch-2 and trunk. > Making WASB listStatus messages consistent > -- > > Key: HADOOP-15533 > URL: https://issues.apache.org/jira/browse/HADOOP-15533 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Trivial > Attachments: HADOOP-15533-001.patch, HADOOP-15533-branch-2-001.patch > > > - This change make WASB listStatus error messages to be consistent with the > rest of the listStatus error messages. > - Inconsistent error messages cause a few WASB tests to fail only in > branch-2. The test bug was introduced in > "https://issues.apache.org/jira/browse/HADOOP-15506";. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15533) Making WASB listStatus error codes consistent with other list status error messages
[ https://issues.apache.org/jira/browse/HADOOP-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15533: - Description: - This change make WASB listStatus error messages to be consistent with the rest of the listStatus error messages. - Inconsistent error messages cause a few WASB tests to fail only in branch-2. The test bug was introduced in "https://issues.apache.org/jira/browse/HADOOP-15506";. was:* Very small string updated to make the error messages consistent > Making WASB listStatus error codes consistent with other list status error > messages > --- > > Key: HADOOP-15533 > URL: https://issues.apache.org/jira/browse/HADOOP-15533 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Trivial > Attachments: HADOOP-15533-001.patch, HADOOP-15533-branch-2-001.patch > > > - This change make WASB listStatus error messages to be consistent with the > rest of the listStatus error messages. > - Inconsistent error messages cause a few WASB tests to fail only in > branch-2. The test bug was introduced in > "https://issues.apache.org/jira/browse/HADOOP-15506";. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15533) Making WASB listStatus messages consistent
[ https://issues.apache.org/jira/browse/HADOOP-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15533: - Summary: Making WASB listStatus messages consistent (was: Making WASB listStatus error codes consistent with other list status error messages) > Making WASB listStatus messages consistent > -- > > Key: HADOOP-15533 > URL: https://issues.apache.org/jira/browse/HADOOP-15533 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Trivial > Attachments: HADOOP-15533-001.patch, HADOOP-15533-branch-2-001.patch > > > - This change make WASB listStatus error messages to be consistent with the > rest of the listStatus error messages. > - Inconsistent error messages cause a few WASB tests to fail only in > branch-2. The test bug was introduced in > "https://issues.apache.org/jira/browse/HADOOP-15506";. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15533) Making WASB listStatus error codes consistent
[ https://issues.apache.org/jira/browse/HADOOP-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15533: - Description: * Very small string updated to make the error messages consistent (was: Very small string updated to make the error messages consistent) > Making WASB listStatus error codes consistent > - > > Key: HADOOP-15533 > URL: https://issues.apache.org/jira/browse/HADOOP-15533 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Trivial > Attachments: HADOOP-15533-001.patch, HADOOP-15533-branch-2-001.patch > > > * Very small string updated to make the error messages consistent -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15506) Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks
[ https://issues.apache.org/jira/browse/HADOOP-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511711#comment-16511711 ] Esfandiar Manii commented on HADOOP-15506: -- Yes. Sure, I will update it now, thanks! > Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code > blocks > --- > > Key: HADOOP-15506 > URL: https://issues.apache.org/jira/browse/HADOOP-15506 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Minor > Fix For: 2.10.0, 3.2.0, 3.1.1, 3.0.4 > > Attachments: HADOOP-15506-001.patch > > > - Upgraded Azure Storage Sdk to 7.0.0 > - Fixed code issues and couple of tests -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15533) Making WASB listStatus error codes consistent with other list status error messages
[ https://issues.apache.org/jira/browse/HADOOP-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15533: - Summary: Making WASB listStatus error codes consistent with other list status error messages (was: Making WASB listStatus error codes consistent) > Making WASB listStatus error codes consistent with other list status error > messages > --- > > Key: HADOOP-15533 > URL: https://issues.apache.org/jira/browse/HADOOP-15533 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Trivial > Attachments: HADOOP-15533-001.patch, HADOOP-15533-branch-2-001.patch > > > * Very small string updated to make the error messages consistent -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15506) Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks
[ https://issues.apache.org/jira/browse/HADOOP-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511502#comment-16511502 ] Esfandiar Manii commented on HADOOP-15506: -- [~chris.douglas] Thanks so much for all the helps. Unfortunately it seems my cache was not cleaned when I ran the tests while one error message needs update. It only affects 3 wasb tests in trunk-2. But sent out a JIRA and made it consistent with trunk. Thanks! https://issues.apache.org/jira/browse/HADOOP-15533 > Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code > blocks > --- > > Key: HADOOP-15506 > URL: https://issues.apache.org/jira/browse/HADOOP-15506 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Minor > Fix For: 2.10.0, 3.2.0, 3.1.1, 3.0.4 > > Attachments: HADOOP-15506-001.patch > > > - Upgraded Azure Storage Sdk to 7.0.0 > - Fixed code issues and couple of tests -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15533) Making WASB listStatus error codes consistent
[ https://issues.apache.org/jira/browse/HADOOP-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15533: - Attachment: HADOOP-15533-branch-2-001.patch HADOOP-15533-001.patch > Making WASB listStatus error codes consistent > - > > Key: HADOOP-15533 > URL: https://issues.apache.org/jira/browse/HADOOP-15533 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Trivial > Attachments: HADOOP-15533-001.patch, HADOOP-15533-branch-2-001.patch > > > Very small string updated to make the error messages consistent -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15533) Making WASB listStatus error codes consistent
Esfandiar Manii created HADOOP-15533: Summary: Making WASB listStatus error codes consistent Key: HADOOP-15533 URL: https://issues.apache.org/jira/browse/HADOOP-15533 Project: Hadoop Common Issue Type: Bug Components: fs/azure Reporter: Esfandiar Manii Assignee: Esfandiar Manii Very small string updated to make the error messages consistent -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15521) Upgrading Azure Storage Sdk version to 7.0.0 and updating corresponding code blocks
[ https://issues.apache.org/jira/browse/HADOOP-15521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16508813#comment-16508813 ] Esfandiar Manii commented on HADOOP-15521: -- After I synced offline with Thomas I figured that I dont need to create a separate JIRA for the back porting. The original change is linked to this JIRA (HADOOP-15506 Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks). Here both of the patches are identical and targeting branch-2 but I had to add the branch name to have the tests run against branch2. Sorry for the confusion. > Upgrading Azure Storage Sdk version to 7.0.0 and updating corresponding code > blocks > --- > > Key: HADOOP-15521 > URL: https://issues.apache.org/jira/browse/HADOOP-15521 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.10.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Minor > Attachments: HADOOP-15521-001.patch, HADOOP-15521-branch-2-001.patch > > > Upgraded Azure Storage Sdk to 7.0.0 > Fixed code issues and couple of tests -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15521) Upgrading Azure Storage Sdk version to 7.0.0 and updating corresponding code blocks
[ https://issues.apache.org/jira/browse/HADOOP-15521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506400#comment-16506400 ] Esfandiar Manii commented on HADOOP-15521: -- Only the Azure SDK, initially I forgot to add "branch-2" so it caused merge conflict with trunk > Upgrading Azure Storage Sdk version to 7.0.0 and updating corresponding code > blocks > --- > > Key: HADOOP-15521 > URL: https://issues.apache.org/jira/browse/HADOOP-15521 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.10.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Minor > Attachments: HADOOP-15521-001.patch, HADOOP-15521-branch-2-001.patch > > > Upgraded Azure Storage Sdk to 7.0.0 > Fixed code issues and couple of tests -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15521) Upgrading Azure Storage Sdk version to 7.0.0 and updating corresponding code blocks
[ https://issues.apache.org/jira/browse/HADOOP-15521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15521: - Attachment: HADOOP-15521-branch-2-001.patch > Upgrading Azure Storage Sdk version to 7.0.0 and updating corresponding code > blocks > --- > > Key: HADOOP-15521 > URL: https://issues.apache.org/jira/browse/HADOOP-15521 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.10.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Minor > Attachments: HADOOP-15521-001.patch, HADOOP-15521-branch-2-001.patch > > > Upgraded Azure Storage Sdk to 7.0.0 > Fixed code issues and couple of tests -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15521) Upgrading Azure Storage Sdk version to 7.0.0 and updating corresponding code blocks
[ https://issues.apache.org/jira/browse/HADOOP-15521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15521: - Affects Version/s: 2.10.0 > Upgrading Azure Storage Sdk version to 7.0.0 and updating corresponding code > blocks > --- > > Key: HADOOP-15521 > URL: https://issues.apache.org/jira/browse/HADOOP-15521 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 2.10.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Minor > Attachments: HADOOP-15521-001.patch > > > Upgraded Azure Storage Sdk to 7.0.0 > Fixed code issues and couple of tests -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15521) Upgrading Azure Storage Sdk version to 7.0.0 and updating corresponding code blocks
[
https://issues.apache.org/jira/browse/HADOOP-15521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505507#comment-16505507
]
Esfandiar Manii commented on HADOOP-15521:
--
{code:java}
[INFO] Executed tasks
[INFO]
[INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ hadoop-azure ---
[INFO]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.fs.azure.TestBlobOperationDescriptor
[INFO] Running org.apache.hadoop.fs.azure.TestWasbFsck
[INFO] Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
[INFO] Running
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.148 s
- in org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.703
s - in org.apache.hadoop.fs.azure.TestWasbFsck
[INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
[INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
[INFO] Tests run: 35, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.321 s
- in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
[INFO] Running org.apache.hadoop.fs.azure.TestBlobMetadata
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.124 s
- in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.817 s
- in org.apache.hadoop.fs.azure.TestBlobMetadata
[INFO] Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
[INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.065
s - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.752 s
- in org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
[INFO] Running
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockCompaction
[INFO] Running org.apache.hadoop.fs.azure.TestClientThrottlingAnalyzer
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.151 s
- in org.apache.hadoop.fs.azure.TestBlobOperationDescriptor
[INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.698 s
- in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
[INFO] Running
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.626 s
- in org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
[INFO] Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.332 s
- in org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
[INFO] Running
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
[INFO] Tests run: 49, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.364 s
- in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
[INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
[INFO] Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.197
s - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.129 s
- in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockCompaction
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.754 s
- in org.apache.hadoop.fs.azure.TestClientThrottlingAnalyzer
[INFO] Tests run: 59, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 231.325
s - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 232, Failures: 0, Errors: 0, Skipped: 4
[INFO]
[INFO]
[INFO] --- maven-surefire-plugin:2.21.0:test (serialized-test) @ hadoop-azure
---
[INFO]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.203 s
- in org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-azure ---
[INFO] Deleting /home/esmanii/trunk-2/hadoop/hadoop-tools/hadoop-azure/target
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-azur
[jira] [Updated] (HADOOP-15521) Upgrading Azure Storage Sdk version to 7.0.0 and updating corresponding code blocks
[ https://issues.apache.org/jira/browse/HADOOP-15521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15521: - Attachment: HADOOP-15521-001.patch > Upgrading Azure Storage Sdk version to 7.0.0 and updating corresponding code > blocks > --- > > Key: HADOOP-15521 > URL: https://issues.apache.org/jira/browse/HADOOP-15521 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Minor > Attachments: HADOOP-15521-001.patch > > > Upgraded Azure Storage Sdk to 7.0.0 > Fixed code issues and couple of tests -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15521) Upgrading Azure Storage Sdk version to 7.0.0 and updating corresponding code blocks
Esfandiar Manii created HADOOP-15521: Summary: Upgrading Azure Storage Sdk version to 7.0.0 and updating corresponding code blocks Key: HADOOP-15521 URL: https://issues.apache.org/jira/browse/HADOOP-15521 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Reporter: Esfandiar Manii Assignee: Esfandiar Manii Upgraded Azure Storage Sdk to 7.0.0 Fixed code issues and couple of tests -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15407) Support Windows Azure Storage - Blob file system in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16503951#comment-16503951
]
Esfandiar Manii commented on HADOOP-15407:
--
Thanks [~fabbri]. Yes, htrace was very useful (credits to Steve for letting us
know to use it :) ).
> Support Windows Azure Storage - Blob file system in Hadoop
> --
>
> Key: HADOOP-15407
> URL: https://issues.apache.org/jira/browse/HADOOP-15407
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Esfandiar Manii
>Assignee: Esfandiar Manii
>Priority: Major
> Attachments: HADOOP-15407-001.patch, HADOOP-15407-002.patch,
> HADOOP-15407-003.patch, HADOOP-15407-004.patch,
> HADOOP-15407-HADOOP-15407.006.patch, HADOOP-15407-HADOOP-15407.007.patch
>
>
> *{color:#212121}Description{color}*
> This JIRA adds a new file system implementation, ABFS, for running Big Data
> and Analytics workloads against Azure Storage. This is a complete rewrite of
> the previous WASB driver with a heavy focus on optimizing both performance
> and cost.
> {color:#212121} {color}
> *{color:#212121}High level design{color}*
> At a high level, the code here extends the FileSystem class to provide an
> implementation for accessing blobs in Azure Storage. The scheme abfs is used
> for accessing it over HTTP, and abfss for accessing over HTTPS. The following
> URI scheme is used to address individual paths:
> {color:#212121} {color}
>
> {color:#212121}abfs[s]://@.dfs.core.windows.net/{color}
> {color:#212121} {color}
> {color:#212121}ABFS is intended as a replacement to WASB. WASB is not
> deprecated but is in pure maintenance mode and customers should upgrade to
> ABFS once it hits General Availability later in CY18.{color}
> {color:#212121}Benefits of ABFS include:{color}
> {color:#212121}· Higher scale (capacity, throughput, and IOPS) Big
> Data and Analytics workloads by allowing higher limits on storage
> accounts{color}
> {color:#212121}· Removing any ramp up time with Storage backend
> partitioning; blocks are now automatically sharded across partitions in the
> Storage backend{color}
> {color:#212121} . This avoids the need for using
> temporary/intermediate files, increasing the cost (and framework complexity
> around committing jobs/tasks){color}
> {color:#212121}· Enabling much higher read and write throughput on
> single files (tens of Gbps by default){color}
> {color:#212121}· Still retaining all of the Azure Blob features
> customers are familiar with and expect, and gaining the benefits of future
> Blob features as well{color}
> {color:#212121}ABFS incorporates Hadoop Filesystem metrics to monitor the
> file system throughput and operations. Ambari metrics are not currently
> implemented for ABFS, but will be available soon.{color}
> {color:#212121} {color}
> *{color:#212121}Credits and history{color}*
> Credit for this work goes to (hope I don't forget anyone): Shane Mainali,
> {color:#212121}Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar
> Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant,
> and James Baker. {color}
> {color:#212121} {color}
> *Test*
> ABFS has gone through many test procedures including Hadoop file system
> contract tests, unit testing, functional testing, and manual testing. All the
> Junit tests provided with the driver are capable of running in both
> sequential/parallel fashion in order to reduce the testing time.
> {color:#212121}Besides unit tests, we have used ABFS as the default file
> system in Azure HDInsight. Azure HDInsight will very soon offer ABFS as a
> storage option. (HDFS is also used but not as default file system.) Various
> different customer and test workloads have been run against clusters with
> such configurations for quite some time. Benchmarks such as Tera*, TPC-DS,
> Spark Streaming and Spark SQL, and others have been run to do scenario,
> performance, and functional testing. Third parties and customers have also
> done various testing of ABFS.{color}
> {color:#212121}The current version reflects to the version of the code
> tested and used in our production environment.{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15506) Upgrading Azure Storage Sdk version and updated corresponding code blocks
[
https://issues.apache.org/jira/browse/HADOOP-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497236#comment-16497236
]
Esfandiar Manii commented on HADOOP-15506:
--
{code:java}
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.fs.azure.TestWasbFsck
[INFO] Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
[INFO] Running org.apache.hadoop.fs.azure.TestBlobOperationDescriptor
[INFO] Running
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.157
s - in org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.736
s - in org.apache.hadoop.fs.azure.TestWasbFsck
[INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
[INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
[WARNING] Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed:
1.493 s - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
[INFO] Running org.apache.hadoop.fs.azure.TestBlobMetadata
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.11 s -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.668 s
- in org.apache.hadoop.fs.azure.TestBlobMetadata
[INFO] Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
[INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
[WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.058
s - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.735 s
- in org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
[INFO] Running
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockCompaction
[INFO] Running org.apache.hadoop.fs.azure.TestClientThrottlingAnalyzer
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.103 s
- in org.apache.hadoop.fs.azure.TestBlobOperationDescriptor
[INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.649 s
- in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
[INFO] Running
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.596 s
- in org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
[INFO] Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.287 s
- in org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
[INFO] Running
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
[INFO] Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.335 s
- in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
[INFO] Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
[INFO] Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.206
s - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.12 s
- in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockCompaction
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.733 s
- in org.apache.hadoop.fs.azure.TestClientThrottlingAnalyzer
[INFO] Tests run: 59, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 230.7 s
- in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 241, Failures: 0, Errors: 0, Skipped: 11
[INFO]
[INFO]
[INFO] --- maven-surefire-plugin:2.21.0:test (serialized-test) @ hadoop-azure
---
[INFO]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.197 s
- in org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-azure ---
[INFO] Deleting /home/esmanii/hadoop/hadoop-tools/hadoop-azure/target
[INFO] Deleting /home/esmanii/hadoop/hadoop-tools/hadoop-azure (includes =
[dependency-reduced-pom.xml], excludes = [])
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-azure
[jira] [Updated] (HADOOP-15506) Upgrading Azure Storage Sdk version and updated corresponding code blocks
[ https://issues.apache.org/jira/browse/HADOOP-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15506: - Attachment: HADOOP-15506-001.patch > Upgrading Azure Storage Sdk version and updated corresponding code blocks > - > > Key: HADOOP-15506 > URL: https://issues.apache.org/jira/browse/HADOOP-15506 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Minor > Attachments: HADOOP-15506-001.patch > > > - Upgraded Azure Storage Sdk to 7.0.0 > - Fixed code issues and couple of tests -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15506) Upgrading Azure Storage Sdk version and updated corresponding code blocks
Esfandiar Manii created HADOOP-15506: Summary: Upgrading Azure Storage Sdk version and updated corresponding code blocks Key: HADOOP-15506 URL: https://issues.apache.org/jira/browse/HADOOP-15506 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Affects Versions: 3.2.0 Reporter: Esfandiar Manii Assignee: Esfandiar Manii - Upgraded Azure Storage Sdk to 7.0.0 - Fixed code issues and couple of tests -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15407) Support Windows Azure Storage - Blob file system in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii updated HADOOP-15407:
-
Attachment: HADOOP-15407-004.patch
> Support Windows Azure Storage - Blob file system in Hadoop
> --
>
> Key: HADOOP-15407
> URL: https://issues.apache.org/jira/browse/HADOOP-15407
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Esfandiar Manii
>Assignee: Esfandiar Manii
>Priority: Major
> Attachments: HADOOP-15407-001.patch, HADOOP-15407-002.patch,
> HADOOP-15407-003.patch, HADOOP-15407-004.patch
>
>
> *{color:#212121}Description{color}*
> This JIRA adds a new file system implementation, ABFS, for running Big Data
> and Analytics workloads against Azure Storage. This is a complete rewrite of
> the previous WASB driver with a heavy focus on optimizing both performance
> and cost.
> {color:#212121} {color}
> *{color:#212121}High level design{color}*
> At a high level, the code here extends the FileSystem class to provide an
> implementation for accessing blobs in Azure Storage. The scheme abfs is used
> for accessing it over HTTP, and abfss for accessing over HTTPS. The following
> URI scheme is used to address individual paths:
> {color:#212121} {color}
>
> {color:#212121}abfs[s]://@.dfs.core.windows.net/{color}
> {color:#212121} {color}
> {color:#212121}ABFS is intended as a replacement to WASB. WASB is not
> deprecated but is in pure maintenance mode and customers should upgrade to
> ABFS once it hits General Availability later in CY18.{color}
> {color:#212121}Benefits of ABFS include:{color}
> {color:#212121}· Higher scale (capacity, throughput, and IOPS) Big
> Data and Analytics workloads by allowing higher limits on storage
> accounts{color}
> {color:#212121}· Removing any ramp up time with Storage backend
> partitioning; blocks are now automatically sharded across partitions in the
> Storage backend{color}
> {color:#212121} . This avoids the need for using
> temporary/intermediate files, increasing the cost (and framework complexity
> around committing jobs/tasks){color}
> {color:#212121}· Enabling much higher read and write throughput on
> single files (tens of Gbps by default){color}
> {color:#212121}· Still retaining all of the Azure Blob features
> customers are familiar with and expect, and gaining the benefits of future
> Blob features as well{color}
> {color:#212121}ABFS incorporates Hadoop Filesystem metrics to monitor the
> file system throughput and operations. Ambari metrics are not currently
> implemented for ABFS, but will be available soon.{color}
> {color:#212121} {color}
> *{color:#212121}Credits and history{color}*
> Credit for this work goes to (hope I don't forget anyone): Shane Mainali,
> {color:#212121}Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar
> Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant,
> and James Baker. {color}
> {color:#212121} {color}
> *Test*
> ABFS has gone through many test procedures including Hadoop file system
> contract tests, unit testing, functional testing, and manual testing. All the
> Junit tests provided with the driver are capable of running in both
> sequential/parallel fashion in order to reduce the testing time.
> {color:#212121}Besides unit tests, we have used ABFS as the default file
> system in Azure HDInsight. Azure HDInsight will very soon offer ABFS as a
> storage option. (HDFS is also used but not as default file system.) Various
> different customer and test workloads have been run against clusters with
> such configurations for quite some time. Benchmarks such as Tera*, TPC-DS,
> Spark Streaming and Spark SQL, and others have been run to do scenario,
> performance, and functional testing. Third parties and customers have also
> done various testing of ABFS.{color}
> {color:#212121}The current version reflects to the version of the code
> tested and used in our production environment.{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15432) AzureBlobFS - Base package classes and configuration files
[
https://issues.apache.org/jira/browse/HADOOP-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16468114#comment-16468114
]
Esfandiar Manii commented on HADOOP-15432:
--
[~mackrorysd] Thank you for your comment. The big patch is broken into 6 parts
(sub JIRAs). I divided all the code by the functionality type, i.e. the patches
won't necessary be compile-able. To help understanding the different parts I
would suggest to review in the following order: Contracts, Constants, Base
Package, Diagnostics, Services, and finally tests.
- I definitely would like to move towards that path. But at the same time we
wanted to be very careful about configurations. Our goal was to have the new
driver be "out of the box" usable for the existing customers which have
configured their clusters with the old version of driver (WASB), so there won't
be any hassles for them.
- Regarding LOG, we only use the LOG in the logging service. There were reasons
we encapsulated it into logging service, e.g. the implementation details and
the way loggers are constructed will be hidden from the callers, also they will
be much easier to replace for testing and etc.
- Regarding those libraries, I will take a look to see if I can upgrade. Our
code had been initially built on top of a local Hadoop 2.7 and I ported into
trunk.
- Regarding shading, I am fine either way. Will convert them to be more
specific.
- The auto generated code is generated by an open source tool (AutoRest) and if
you take a look over the generated code in the other JIRA, you'll notice the
amount of generated code is quite big. Updating the style for that amount code
which is subject to change anytime would not be relatively easy. But we are
very open to suggestions! :)
- IsSecure is called in AbfsHttpClientFactoryImpl.java in the services patch.
if (abfs.isSecure() {
scheme = FileSystemUriSchemes.HTTPS_SCHEME;
}
- I agree, we followed the same pattern as WASB configuration for back
compatibility reasons. Once we pass WASB deprecation phases, we will remove
extra ones.
- Thanks for letting me know about it, those tests were not implemented in
Hadoop 2.7. I will implement it them in the next change.
- If you are interested in running tests yourself, you can just directly
contact me and I will provide the access info through email.
esma...@microsoft.com
> AzureBlobFS - Base package classes and configuration files
> --
>
> Key: HADOOP-15432
> URL: https://issues.apache.org/jira/browse/HADOOP-15432
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Esfandiar Manii
>Assignee: Esfandiar Manii
>Priority: Major
> Attachments: HADOOP-15432-001.patch, HADOOP-15432-003.patch
>
>
> Patch contains:
> - AzureBlobFileSystem and SecureAzureBlobFileSystem classes which are the
> main interfaces Hadoop interacts with.
> - Updated Azure pom.xml with updated dependencies, updated parallel tests
> configurations and maven shader plugin.
> - Checkstyle suppression file. Since http layer is generated automatically by
> another libraries, it will not follow hadoop coding guidelines. Therefore a
> few rules for checkstyles have been disabled.
> - Added test configuration file template to be used by the consumers. Similar
> to wasb, all the configurations will go into this file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15432) AzureBlobFS - Base package classes and configuration files
[ https://issues.apache.org/jira/browse/HADOOP-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15432: - Description: Patch contains: - AzureBlobFileSystem and SecureAzureBlobFileSystem classes which are the main interfaces Hadoop interacts with. - Updated Azure pom.xml with updated dependencies, updated parallel tests configurations and maven shader plugin. - Checkstyle suppression file. Since http layer is generated automatically by another libraries, it will not follow hadoop coding guidelines. Therefore a few rules for checkstyles have been disabled. - Added test configuration file template to be used by the consumers. Similar to wasb, all the configurations will go into this file. > AzureBlobFS - Base package classes and configuration files > -- > > Key: HADOOP-15432 > URL: https://issues.apache.org/jira/browse/HADOOP-15432 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15432-001.patch, HADOOP-15432-003.patch > > > Patch contains: > - AzureBlobFileSystem and SecureAzureBlobFileSystem classes which are the > main interfaces Hadoop interacts with. > - Updated Azure pom.xml with updated dependencies, updated parallel tests > configurations and maven shader plugin. > - Checkstyle suppression file. Since http layer is generated automatically by > another libraries, it will not follow hadoop coding guidelines. Therefore a > few rules for checkstyles have been disabled. > - Added test configuration file template to be used by the consumers. Similar > to wasb, all the configurations will go into this file. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15435) AzureBlobFS - Constants
[ https://issues.apache.org/jira/browse/HADOOP-15435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15435: - Attachment: HADOOP-15435-003.patch > AzureBlobFS - Constants > --- > > Key: HADOOP-15435 > URL: https://issues.apache.org/jira/browse/HADOOP-15435 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15435-001.patch, HADOOP-15435-003.patch > > > AzureBlobFS constants used across the driver. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15438) AzureBlobFS - Tests
[ https://issues.apache.org/jira/browse/HADOOP-15438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15438: - Attachment: HADOOP-15438-003.patch > AzureBlobFS - Tests > --- > > Key: HADOOP-15438 > URL: https://issues.apache.org/jira/browse/HADOOP-15438 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15438-001.patch, HADOOP-15438-003.patch > > > AzureBlobFS functional and contract tests -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15437) AzureBlobFS - Services
[ https://issues.apache.org/jira/browse/HADOOP-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15437: - Attachment: HADOOP-15437-003.patch > AzureBlobFS - Services > -- > > Key: HADOOP-15437 > URL: https://issues.apache.org/jira/browse/HADOOP-15437 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15437-003.patch, HADOOP-15437.patch > > > AzureBlobFS services and factories in the driver. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15436) AzureBlobFS - Diagnostics and Utils
[ https://issues.apache.org/jira/browse/HADOOP-15436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15436: - Attachment: HADOOP-15436-003.patch > AzureBlobFS - Diagnostics and Utils > --- > > Key: HADOOP-15436 > URL: https://issues.apache.org/jira/browse/HADOOP-15436 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15436-001.patch, HADOOP-15436-003.patch > > > AzureBlobFS Diagnostics and Utils classes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15432) AzureBlobFS - Base package classes and configuration files
[ https://issues.apache.org/jira/browse/HADOOP-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15432: - Attachment: HADOOP-15432-003.patch > AzureBlobFS - Base package classes and configuration files > -- > > Key: HADOOP-15432 > URL: https://issues.apache.org/jira/browse/HADOOP-15432 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15432-001.patch, HADOOP-15432-003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15433) AzureBlobFS - Contracts
[ https://issues.apache.org/jira/browse/HADOOP-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15433: - Attachment: HADOOP-15433-003.patch > AzureBlobFS - Contracts > --- > > Key: HADOOP-15433 > URL: https://issues.apache.org/jira/browse/HADOOP-15433 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15433-001.patch, HADOOP-15433-003.patch > > > All the internal, external contracts for the AzureBlobFS driver. > Contracts include: > - Configuration annotations > - Configuration validation contract > - Custom exceptions > - Service contracts -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15407) Support Windows Azure Storage - Blob file system in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii updated HADOOP-15407:
-
Attachment: HADOOP-15407-003.patch
> Support Windows Azure Storage - Blob file system in Hadoop
> --
>
> Key: HADOOP-15407
> URL: https://issues.apache.org/jira/browse/HADOOP-15407
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Esfandiar Manii
>Assignee: Esfandiar Manii
>Priority: Major
> Attachments: HADOOP-15407-001.patch, HADOOP-15407-002.patch,
> HADOOP-15407-003.patch
>
>
> *{color:#212121}Description{color}*
> This JIRA adds a new file system implementation, ABFS, for running Big Data
> and Analytics workloads against Azure Storage. This is a complete rewrite of
> the previous WASB driver with a heavy focus on optimizing both performance
> and cost.
> {color:#212121} {color}
> *{color:#212121}High level design{color}*
> At a high level, the code here extends the FileSystem class to provide an
> implementation for accessing blobs in Azure Storage. The scheme abfs is used
> for accessing it over HTTP, and abfss for accessing over HTTPS. The following
> URI scheme is used to address individual paths:
> {color:#212121} {color}
>
> {color:#212121}abfs[s]://@.dfs.core.windows.net/{color}
> {color:#212121} {color}
> {color:#212121}ABFS is intended as a replacement to WASB. WASB is not
> deprecated but is in pure maintenance mode and customers should upgrade to
> ABFS once it hits General Availability later in CY18.{color}
> {color:#212121}Benefits of ABFS include:{color}
> {color:#212121}· Higher scale (capacity, throughput, and IOPS) Big
> Data and Analytics workloads by allowing higher limits on storage
> accounts{color}
> {color:#212121}· Removing any ramp up time with Storage backend
> partitioning; blocks are now automatically sharded across partitions in the
> Storage backend{color}
> {color:#212121} . This avoids the need for using
> temporary/intermediate files, increasing the cost (and framework complexity
> around committing jobs/tasks){color}
> {color:#212121}· Enabling much higher read and write throughput on
> single files (tens of Gbps by default){color}
> {color:#212121}· Still retaining all of the Azure Blob features
> customers are familiar with and expect, and gaining the benefits of future
> Blob features as well{color}
> {color:#212121}ABFS incorporates Hadoop Filesystem metrics to monitor the
> file system throughput and operations. Ambari metrics are not currently
> implemented for ABFS, but will be available soon.{color}
> {color:#212121} {color}
> *{color:#212121}Credits and history{color}*
> Credit for this work goes to (hope I don't forget anyone): Shane Mainali,
> {color:#212121}Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar
> Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant,
> and James Baker. {color}
> {color:#212121} {color}
> *Test*
> ABFS has gone through many test procedures including Hadoop file system
> contract tests, unit testing, functional testing, and manual testing. All the
> Junit tests provided with the driver are capable of running in both
> sequential/parallel fashion in order to reduce the testing time.
> {color:#212121}Besides unit tests, we have used ABFS as the default file
> system in Azure HDInsight. Azure HDInsight will very soon offer ABFS as a
> storage option. (HDFS is also used but not as default file system.) Various
> different customer and test workloads have been run against clusters with
> such configurations for quite some time. Benchmarks such as Tera*, TPC-DS,
> Spark Streaming and Spark SQL, and others have been run to do scenario,
> performance, and functional testing. Third parties and customers have also
> done various testing of ABFS.{color}
> {color:#212121}The current version reflects to the version of the code
> tested and used in our production environment.{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15436) AzureBlobFS - Diagnostics and Utils
[ https://issues.apache.org/jira/browse/HADOOP-15436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii reassigned HADOOP-15436: Assignee: Esfandiar Manii > AzureBlobFS - Diagnostics and Utils > --- > > Key: HADOOP-15436 > URL: https://issues.apache.org/jira/browse/HADOOP-15436 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15436-001.patch > > > AzureBlobFS Diagnostics and Utils classes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15433) AzureBlobFS - Contracts
[ https://issues.apache.org/jira/browse/HADOOP-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15433: - Description: All the internal, external contracts for the AzureBlobFS driver. Contracts include: - Configuration annotations - Configuration validation contract - Custom exceptions - Service contracts was:All the internal, external contracts for the AzureBlobFS driver > AzureBlobFS - Contracts > --- > > Key: HADOOP-15433 > URL: https://issues.apache.org/jira/browse/HADOOP-15433 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15433-001.patch > > > All the internal, external contracts for the AzureBlobFS driver. > Contracts include: > - Configuration annotations > - Configuration validation contract > - Custom exceptions > - Service contracts -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15438) AzureBlobFS - Tests
Esfandiar Manii created HADOOP-15438: Summary: AzureBlobFS - Tests Key: HADOOP-15438 URL: https://issues.apache.org/jira/browse/HADOOP-15438 Project: Hadoop Common Issue Type: Sub-task Reporter: Esfandiar Manii Assignee: Esfandiar Manii Attachments: HADOOP-15438-001.patch AzureBlobFS functional and contract tests -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15438) AzureBlobFS - Tests
[ https://issues.apache.org/jira/browse/HADOOP-15438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15438: - Attachment: HADOOP-15438-001.patch > AzureBlobFS - Tests > --- > > Key: HADOOP-15438 > URL: https://issues.apache.org/jira/browse/HADOOP-15438 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15438-001.patch > > > AzureBlobFS functional and contract tests -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15437) AzureBlobFS - Services
Esfandiar Manii created HADOOP-15437: Summary: AzureBlobFS - Services Key: HADOOP-15437 URL: https://issues.apache.org/jira/browse/HADOOP-15437 Project: Hadoop Common Issue Type: Sub-task Reporter: Esfandiar Manii Assignee: Esfandiar Manii Attachments: HADOOP-15437.patch AzureBlobFS services and factories in the driver. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15437) AzureBlobFS - Services
[ https://issues.apache.org/jira/browse/HADOOP-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15437: - Attachment: HADOOP-15437.patch > AzureBlobFS - Services > -- > > Key: HADOOP-15437 > URL: https://issues.apache.org/jira/browse/HADOOP-15437 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15437.patch > > > AzureBlobFS services and factories in the driver. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15436) AzureBlobFS - Diagnostics and Utils
[ https://issues.apache.org/jira/browse/HADOOP-15436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15436: - Attachment: HADOOP-15436-001.patch > AzureBlobFS - Diagnostics and Utils > --- > > Key: HADOOP-15436 > URL: https://issues.apache.org/jira/browse/HADOOP-15436 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15436-001.patch > > > AzureBlobFS Diagnostics and Utils classes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15436) AzureBlobFS - Diagnostics and Utils
Esfandiar Manii created HADOOP-15436: Summary: AzureBlobFS - Diagnostics and Utils Key: HADOOP-15436 URL: https://issues.apache.org/jira/browse/HADOOP-15436 Project: Hadoop Common Issue Type: Sub-task Reporter: Esfandiar Manii Attachments: HADOOP-15436-001.patch AzureBlobFS Diagnostics and Utils classes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15435) AzureBlobFS - Constants
[ https://issues.apache.org/jira/browse/HADOOP-15435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15435: - Attachment: HADOOP-15435-001.patch > AzureBlobFS - Constants > --- > > Key: HADOOP-15435 > URL: https://issues.apache.org/jira/browse/HADOOP-15435 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15435-001.patch > > > AzureBlobFS constants used across the driver. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15435) AzureBlobFS - Constants
Esfandiar Manii created HADOOP-15435: Summary: AzureBlobFS - Constants Key: HADOOP-15435 URL: https://issues.apache.org/jira/browse/HADOOP-15435 Project: Hadoop Common Issue Type: Sub-task Reporter: Esfandiar Manii Assignee: Esfandiar Manii Attachments: HADOOP-15435-001.patch AzureBlobFS constants used across the driver. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15433) AzureBlobFS - Contracts
Esfandiar Manii created HADOOP-15433: Summary: AzureBlobFS - Contracts Key: HADOOP-15433 URL: https://issues.apache.org/jira/browse/HADOOP-15433 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.2.0 Reporter: Esfandiar Manii Assignee: Esfandiar Manii Attachments: HADOOP-15433-001.patch All the internal, external contracts for the AzureBlobFS driver -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15433) AzureBlobFS - Contracts
[ https://issues.apache.org/jira/browse/HADOOP-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15433: - Attachment: HADOOP-15433-001.patch > AzureBlobFS - Contracts > --- > > Key: HADOOP-15433 > URL: https://issues.apache.org/jira/browse/HADOOP-15433 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15433-001.patch > > > All the internal, external contracts for the AzureBlobFS driver -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15432) AzureBlobFS - Base package classes and configuration files
[ https://issues.apache.org/jira/browse/HADOOP-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15432: - Attachment: HADOOP-15432.patch > AzureBlobFS - Base package classes and configuration files > -- > > Key: HADOOP-15432 > URL: https://issues.apache.org/jira/browse/HADOOP-15432 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15432-001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15432) AzureBlobFS - Base package classes and configuration files
[ https://issues.apache.org/jira/browse/HADOOP-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15432: - Attachment: HADOOP-15432-001.patch > AzureBlobFS - Base package classes and configuration files > -- > > Key: HADOOP-15432 > URL: https://issues.apache.org/jira/browse/HADOOP-15432 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15432-001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15432) AzureBlobFS - Base package classes and configuration files
[ https://issues.apache.org/jira/browse/HADOOP-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esfandiar Manii updated HADOOP-15432: - Attachment: (was: HADOOP-15432.patch) > AzureBlobFS - Base package classes and configuration files > -- > > Key: HADOOP-15432 > URL: https://issues.apache.org/jira/browse/HADOOP-15432 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Esfandiar Manii >Assignee: Esfandiar Manii >Priority: Major > Attachments: HADOOP-15432-001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15432) AzureBlobFS - Base package classes and configuration files
Esfandiar Manii created HADOOP-15432: Summary: AzureBlobFS - Base package classes and configuration files Key: HADOOP-15432 URL: https://issues.apache.org/jira/browse/HADOOP-15432 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.2.0 Reporter: Esfandiar Manii Assignee: Esfandiar Manii -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15407) Support Windows Azure Storage - Blob file system in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449145#comment-16449145
]
Esfandiar Manii commented on HADOOP-15407:
--
My bad, the order of diff was incorrect. Updated with the correct one. :)
> Support Windows Azure Storage - Blob file system in Hadoop
> --
>
> Key: HADOOP-15407
> URL: https://issues.apache.org/jira/browse/HADOOP-15407
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Esfandiar Manii
>Assignee: Esfandiar Manii
>Priority: Major
> Attachments: HADOOP-15407-001.patch, HADOOP-15407-002.patch
>
>
> *{color:#212121}Description{color}*
> This JIRA adds a new file system implementation, ABFS, for running Big Data
> and Analytics workloads against Azure Storage. This is a complete rewrite of
> the previous WASB driver with a heavy focus on optimizing both performance
> and cost.
> {color:#212121} {color}
> *{color:#212121}High level design{color}*
> At a high level, the code here extends the FileSystem class to provide an
> implementation for accessing blobs in Azure Storage. The scheme abfs is used
> for accessing it over HTTP, and abfss for accessing over HTTPS. The following
> URI scheme is used to address individual paths:
> {color:#212121} {color}
>
> {color:#212121}abfs[s]://@.dfs.core.windows.net/{color}
> {color:#212121} {color}
> {color:#212121}ABFS is intended as a replacement to WASB. WASB is not
> deprecated but is in pure maintenance mode and customers should upgrade to
> ABFS once it hits General Availability later in CY18.{color}
> {color:#212121}Benefits of ABFS include:{color}
> {color:#212121}· Higher scale (capacity, throughput, and IOPS) Big
> Data and Analytics workloads by allowing higher limits on storage
> accounts{color}
> {color:#212121}· Removing any ramp up time with Storage backend
> partitioning; blocks are now automatically sharded across partitions in the
> Storage backend{color}
> {color:#212121} . This avoids the need for using
> temporary/intermediate files, increasing the cost (and framework complexity
> around committing jobs/tasks){color}
> {color:#212121}· Enabling much higher read and write throughput on
> single files (tens of Gbps by default){color}
> {color:#212121}· Still retaining all of the Azure Blob features
> customers are familiar with and expect, and gaining the benefits of future
> Blob features as well{color}
> {color:#212121}ABFS incorporates Hadoop Filesystem metrics to monitor the
> file system throughput and operations. Ambari metrics are not currently
> implemented for ABFS, but will be available soon.{color}
> {color:#212121} {color}
> *{color:#212121}Credits and history{color}*
> Credit for this work goes to (hope I don't forget anyone): Shane Mainali,
> {color:#212121}Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar
> Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant,
> and James Baker. {color}
> {color:#212121} {color}
> *Test*
> ABFS has gone through many test procedures including Hadoop file system
> contract tests, unit testing, functional testing, and manual testing. All the
> Junit tests provided with the driver are capable of running in both
> sequential/parallel fashion in order to reduce the testing time.
> {color:#212121}Besides unit tests, we have used ABFS as the default file
> system in Azure HDInsight. Azure HDInsight will very soon offer ABFS as a
> storage option. (HDFS is also used but not as default file system.) Various
> different customer and test workloads have been run against clusters with
> such configurations for quite some time. Benchmarks such as Tera*, TPC-DS,
> Spark Streaming and Spark SQL, and others have been run to do scenario,
> performance, and functional testing. Third parties and customers have also
> done various testing of ABFS.{color}
> {color:#212121}The current version reflects to the version of the code
> tested and used in our production environment.{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15407) Support Windows Azure Storage - Blob file system in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii updated HADOOP-15407:
-
Attachment: HADOOP-15407-002.patch
> Support Windows Azure Storage - Blob file system in Hadoop
> --
>
> Key: HADOOP-15407
> URL: https://issues.apache.org/jira/browse/HADOOP-15407
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Esfandiar Manii
>Assignee: Esfandiar Manii
>Priority: Major
> Attachments: HADOOP-15407-001.patch, HADOOP-15407-002.patch
>
>
> *{color:#212121}Description{color}*
> This JIRA adds a new file system implementation, ABFS, for running Big Data
> and Analytics workloads against Azure Storage. This is a complete rewrite of
> the previous WASB driver with a heavy focus on optimizing both performance
> and cost.
> {color:#212121} {color}
> *{color:#212121}High level design{color}*
> At a high level, the code here extends the FileSystem class to provide an
> implementation for accessing blobs in Azure Storage. The scheme abfs is used
> for accessing it over HTTP, and abfss for accessing over HTTPS. The following
> URI scheme is used to address individual paths:
> {color:#212121} {color}
>
> {color:#212121}abfs[s]://@.dfs.core.windows.net/{color}
> {color:#212121} {color}
> {color:#212121}ABFS is intended as a replacement to WASB. WASB is not
> deprecated but is in pure maintenance mode and customers should upgrade to
> ABFS once it hits General Availability later in CY18.{color}
> {color:#212121}Benefits of ABFS include:{color}
> {color:#212121}· Higher scale (capacity, throughput, and IOPS) Big
> Data and Analytics workloads by allowing higher limits on storage
> accounts{color}
> {color:#212121}· Removing any ramp up time with Storage backend
> partitioning; blocks are now automatically sharded across partitions in the
> Storage backend{color}
> {color:#212121} . This avoids the need for using
> temporary/intermediate files, increasing the cost (and framework complexity
> around committing jobs/tasks){color}
> {color:#212121}· Enabling much higher read and write throughput on
> single files (tens of Gbps by default){color}
> {color:#212121}· Still retaining all of the Azure Blob features
> customers are familiar with and expect, and gaining the benefits of future
> Blob features as well{color}
> {color:#212121}ABFS incorporates Hadoop Filesystem metrics to monitor the
> file system throughput and operations. Ambari metrics are not currently
> implemented for ABFS, but will be available soon.{color}
> {color:#212121} {color}
> *{color:#212121}Credits and history{color}*
> Credit for this work goes to (hope I don't forget anyone): Shane Mainali,
> {color:#212121}Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar
> Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant,
> and James Baker. {color}
> {color:#212121} {color}
> *Test*
> ABFS has gone through many test procedures including Hadoop file system
> contract tests, unit testing, functional testing, and manual testing. All the
> Junit tests provided with the driver are capable of running in both
> sequential/parallel fashion in order to reduce the testing time.
> {color:#212121}Besides unit tests, we have used ABFS as the default file
> system in Azure HDInsight. Azure HDInsight will very soon offer ABFS as a
> storage option. (HDFS is also used but not as default file system.) Various
> different customer and test workloads have been run against clusters with
> such configurations for quite some time. Benchmarks such as Tera*, TPC-DS,
> Spark Streaming and Spark SQL, and others have been run to do scenario,
> performance, and functional testing. Third parties and customers have also
> done various testing of ABFS.{color}
> {color:#212121}The current version reflects to the version of the code
> tested and used in our production environment.{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15407) Support Windows Azure Storage - Blob file system in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448976#comment-16448976
]
Esfandiar Manii commented on HADOOP-15407:
--
{code:java}
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCreate
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.924 s
- in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCreate
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCopy
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.623 s
- in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemCopy
[INFO] Running
org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemInitAndCreate
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.731 s
- in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemInitAndCreate
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 246.169
s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemE2EScale
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAppend
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.202 s
- in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAppend
[INFO] Running
org.apache.hadoop.fs.azurebfs.diagnostics.TestConfigurationValidators
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.805 s
- in org.apache.hadoop.fs.azurebfs.diagnostics.TestConfigurationValidators
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRename
[WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
27.916 s - in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRename
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestConfigurationServiceFieldsValidation
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.258 s
- in
org.apache.hadoop.fs.azurebfs.services.TestConfigurationServiceFieldsValidation
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestAbfsHttpServiceImpl
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.977 s
- in org.apache.hadoop.fs.azurebfs.services.ITestAbfsHttpServiceImpl
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestParameterizedLoggingServiceImpl
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.283 s
- in org.apache.hadoop.fs.azurebfs.services.TestParameterizedLoggingServiceImpl
[INFO] Running org.apache.hadoop.fs.azurebfs.services.TestLoggingServiceImpl
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.253 s
- in org.apache.hadoop.fs.azurebfs.services.TestLoggingServiceImpl
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestNetworkThroughputAnalysisServiceImpl
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.87 s
- in
org.apache.hadoop.fs.azurebfs.services.TestNetworkThroughputAnalysisServiceImpl
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestReadWriteAndSeek
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 244.85 s
- in org.apache.hadoop.fs.azurebfs.services.ITestReadWriteAndSeek
[INFO] Running
org.apache.hadoop.fs.azurebfs.services.TestAbfsStatisticsServiceImpl
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.195 s
- in org.apache.hadoop.fs.azurebfs.services.TestAbfsStatisticsServiceImpl
[INFO] Running org.apache.hadoop.fs.azurebfs.services.ITestTracingServiceImpl
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.893 s
- in org.apache.hadoop.fs.azurebfs.services.ITestTracingServiceImpl
[INFO] Running org.apache.hadoop.fs.azurebfs.utils.TestUriUtils
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.037 s
- in org.apache.hadoop.fs.azurebfs.utils.TestUriUtils
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility
[WARNING] Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
11.948 s - in org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility
[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFileStatus
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.894 s
- in org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemFileStatus
[INFO] Running
org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp
[WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.834
s - in org.apache.hadoop.fs.azurebfs.contract.ITestAbfsFileSystemContractDistCp
[INFO] Running
org.apache.hadoop.fs.azurebfs.contract.ITestAzureBlobFileSystemContract
[INFO] Tests run: 45, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.694
s -
[jira] [Updated] (HADOOP-15407) Support Windows Azure Storage - Blob file system in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii updated HADOOP-15407:
-
Attachment: HADOOP-15407-001.patch
> Support Windows Azure Storage - Blob file system in Hadoop
> --
>
> Key: HADOOP-15407
> URL: https://issues.apache.org/jira/browse/HADOOP-15407
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Esfandiar Manii
>Assignee: Esfandiar Manii
>Priority: Major
> Attachments: HADOOP-15407-001.patch
>
>
> *{color:#212121}Description{color}*
> This JIRA adds a new file system implementation, ABFS, for running Big Data
> and Analytics workloads against Azure Storage. This is a complete rewrite of
> the previous WASB driver with a heavy focus on optimizing both performance
> and cost.
> {color:#212121} {color}
> *{color:#212121}High level design{color}*
> At a high level, the code here extends the FileSystem class to provide an
> implementation for accessing blobs in Azure Storage. The scheme abfs is used
> for accessing it over HTTP, and abfss for accessing over HTTPS. The following
> URI scheme is used to address individual paths:
> {color:#212121} {color}
>
> {color:#212121}abfs[s]://@.dfs.core.windows.net/{color}
> {color:#212121} {color}
> {color:#212121}ABFS is intended as a replacement to WASB. WASB is not
> deprecated but is in pure maintenance mode and customers should upgrade to
> ABFS once it hits General Availability later in CY18.{color}
> {color:#212121}Benefits of ABFS include:{color}
> {color:#212121}· Higher scale (capacity, throughput, and IOPS) Big
> Data and Analytics workloads by allowing higher limits on storage
> accounts{color}
> {color:#212121}· Removing any ramp up time with Storage backend
> partitioning; blocks are now automatically sharded across partitions in the
> Storage backend{color}
> {color:#212121} . This avoids the need for using
> temporary/intermediate files, increasing the cost (and framework complexity
> around committing jobs/tasks){color}
> {color:#212121}· Enabling much higher read and write throughput on
> single files (tens of Gbps by default){color}
> {color:#212121}· Still retaining all of the Azure Blob features
> customers are familiar with and expect, and gaining the benefits of future
> Blob features as well{color}
> {color:#212121}ABFS incorporates Hadoop Filesystem metrics to monitor the
> file system throughput and operations. Ambari metrics are not currently
> implemented for ABFS, but will be available soon.{color}
> {color:#212121} {color}
> *{color:#212121}Credits and history{color}*
> Credit for this work goes to (hope I don't forget anyone): Shane Mainali,
> {color:#212121}Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar
> Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant,
> and James Baker. {color}
> {color:#212121} {color}
> *Test*
> ABFS has gone through many test procedures including Hadoop file system
> contract tests, unit testing, functional testing, and manual testing. All the
> Junit tests provided with the driver are capable of running in both
> sequential/parallel fashion in order to reduce the testing time.
> {color:#212121}Besides unit tests, we have used ABFS as the default file
> system in Azure HDInsight. Azure HDInsight will very soon offer ABFS as a
> storage option. (HDFS is also used but not as default file system.) Various
> different customer and test workloads have been run against clusters with
> such configurations for quite some time. Benchmarks such as Tera*, TPC-DS,
> Spark Streaming and Spark SQL, and others have been run to do scenario,
> performance, and functional testing. Third parties and customers have also
> done various testing of ABFS.{color}
> {color:#212121}The current version reflects to the version of the code
> tested and used in our production environment.{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15407) Support Windows Azure Storage - Blob file system in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii updated HADOOP-15407:
-
Description:
*{color:#212121}Description{color}*
This JIRA adds a new file system implementation, ABFS, for running Big Data
and Analytics workloads against Azure Storage. This is a complete rewrite of
the previous WASB driver with a heavy focus on optimizing both performance and
cost.
{color:#212121} {color}
*{color:#212121}High level design{color}*
At a high level, the code here extends the FileSystem class to provide an
implementation for accessing blobs in Azure Storage. The scheme abfs is used
for accessing it over HTTP, and abfss for accessing over HTTPS. The following
URI scheme is used to address individual paths:
{color:#212121} {color}
{color:#212121}abfs[s]://@.dfs.core.windows.net/{color}
{color:#212121} {color}
{color:#212121}ABFS is intended as a replacement to WASB. WASB is not
deprecated but is in pure maintenance mode and customers should upgrade to ABFS
once it hits General Availability later in CY18.{color}
{color:#212121}Benefits of ABFS include:{color}
{color:#212121}· Higher scale (capacity, throughput, and IOPS) Big
Data and Analytics workloads by allowing higher limits on storage
accounts{color}
{color:#212121}· Removing any ramp up time with Storage backend
partitioning; blocks are now automatically sharded across partitions in the
Storage backend{color}
{color:#212121} . This avoids the need for using
temporary/intermediate files, increasing the cost (and framework complexity
around committing jobs/tasks){color}
{color:#212121}· Enabling much higher read and write throughput on
single files (tens of Gbps by default){color}
{color:#212121}· Still retaining all of the Azure Blob features
customers are familiar with and expect, and gaining the benefits of future Blob
features as well{color}
{color:#212121}ABFS incorporates Hadoop Filesystem metrics to monitor the file
system throughput and operations. Ambari metrics are not currently implemented
for ABFS, but will be available soon.{color}
{color:#212121} {color}
*{color:#212121}Credits and history{color}*
Credit for this work goes to (hope I don't forget anyone): Shane Mainali,
{color:#212121}Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar Manii,
Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant, and
James Baker. {color}
{color:#212121} {color}
*Test*
ABFS has gone through many test procedures including Hadoop file system
contract tests, unit testing, functional testing, and manual testing. All the
Junit tests provided with the driver are capable of running in both
sequential/parallel fashion in order to reduce the testing time.
{color:#212121}Besides unit tests, we have used ABFS as the default file
system in Azure HDInsight. Azure HDInsight will very soon offer ABFS as a
storage option. (HDFS is also used but not as default file system.) Various
different customer and test workloads have been run against clusters with such
configurations for quite some time. Benchmarks such as Tera*, TPC-DS, Spark
Streaming and Spark SQL, and others have been run to do scenario, performance,
and functional testing. Third parties and customers have also done various
testing of ABFS.{color}
{color:#212121}The current version reflects to the version of the code tested
and used in our production environment.{color}
was:
{color:#212121}Description{color}
This JIRA adds a new file system implementation, ABFS, for running Big Data
and Analytics workloads against Azure Storage. This is a complete rewrite of
the previous WASB driver with a heavy focus on optimizing both performance and
cost.
{color:#212121} {color}
{color:#212121}High level design{color}
At a high level, the code here extends the FileSystem class to provide an
implementation for accessing blobs in Azure Storage. The scheme abfs is used
for accessing it over HTTP, and abfss for accessing over HTTPS. The following
URI scheme is used to address individual paths:
{color:#212121} {color}
{color:#212121}abfs[s]://@.dfs.core.windows.net/{color}
{color:#212121} {color}
{color:#212121}ABFS is intended as a replacement to WASB. WASB is not
deprecated but is in pure maintenance mode and customers should upgrade to ABFS
once it hits General Availability later in CY18.{color}
{color:#212121}Benefits of ABFS include:{color}
{color:#212121}· Higher scale (capacity, throughput, and IOPS) Big
Data and Analytics workloads by allowing higher limits on storage
accounts{color}
{color:#212121}· Removing any ramp up time with Storage backend
partitioning; blocks are now automatically sharded across partitions in the
Storage backend{color}
{color:#212121}oThis avoids the need for using temporary/intermediate
files, increasing
[jira] [Updated] (HADOOP-15407) Support Windows Azure Storage - Blob file system in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii updated HADOOP-15407:
-
Description:
{color:#212121}Description{color}
This JIRA adds a new file system implementation, ABFS, for running Big Data
and Analytics workloads against Azure Storage. This is a complete rewrite of
the previous WASB driver with a heavy focus on optimizing both performance and
cost.
{color:#212121} {color}
{color:#212121}High level design{color}
At a high level, the code here extends the FileSystem class to provide an
implementation for accessing blobs in Azure Storage. The scheme abfs is used
for accessing it over HTTP, and abfss for accessing over HTTPS. The following
URI scheme is used to address individual paths:
{color:#212121} {color}
{color:#212121}abfs[s]://@.dfs.core.windows.net/{color}
{color:#212121} {color}
{color:#212121}ABFS is intended as a replacement to WASB. WASB is not
deprecated but is in pure maintenance mode and customers should upgrade to ABFS
once it hits General Availability later in CY18.{color}
{color:#212121}Benefits of ABFS include:{color}
{color:#212121}· Higher scale (capacity, throughput, and IOPS) Big
Data and Analytics workloads by allowing higher limits on storage
accounts{color}
{color:#212121}· Removing any ramp up time with Storage backend
partitioning; blocks are now automatically sharded across partitions in the
Storage backend{color}
{color:#212121}oThis avoids the need for using temporary/intermediate
files, increasing the cost (and framework complexity around committing
jobs/tasks){color}
{color:#212121}· Enabling much higher read and write throughput on
single files (tens of Gbps by default){color}
{color:#212121}· Still retaining all of the Azure Blob features
customers are familiar with and expect, and gaining the benefits of future Blob
features as well{color}
{color:#212121}ABFS incorporates Hadoop Filesystem metrics to monitor the file
system throughput and operations. Ambari metrics are not currently implemented
for ABFS, but will be available soon.{color}
{color:#212121} {color}
{color:#212121}Credits and history{color}
Credit for this work goes to (hope I don't forget anyone): Shane Mainali,
{color:#212121}Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar Manii,
Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant, and
James Baker. {color}
{color:#212121} {color}
Test
ABFS has gone through many test procedures including Hadoop file system
contract tests, unit testing, functional testing, and manual testing. All the
Junit tests provided with the driver are capable of running in both
sequential/parallel fashion in order to reduce the testing time.
{color:#212121}Besides unit tests, we have used ABFS as the default file
system in Azure HDInsight. Azure HDInsight will very soon offer ABFS as a
storage option. (HDFS is also used but not as default file system.) Various
different customer and test workloads have been run against clusters with such
configurations for quite some time. Benchmarks such as Tera*, TPC-DS, Spark
Streaming and Spark SQL, and others have been run to do scenario, performance,
and functional testing. Third parties and customers have also done various
testing of ABFS.{color}
{color:#212121}The current version reflects to the version of the code tested
and used in our production environment.{color}
was:
{color:#33}Description{color}
This JIRA adds a new file system implementation, ABFS, for running Big Data and
Analytics workloads against Azure Storage. This is a complete rewrite of the
previous WASB driver with a heavy focus on optimizing both performance and cost.
{color:#33}High level design{color}
At a high level, the code here extends the FileSystem class to provide an
implementation for accessing blobs in Azure Storage. The scheme abfs is used
for accessing it over HTTP, and abfss for accessing over HTTPS. The following
URI scheme is used to address individual paths:
abfs[s]://@.dfs.core.windows.net/
{color:#33} {color}
ABFS is intended as a replacement to WASB. WASB is not deprecated but is in
pure maintenance mode and customers should upgrade to ABFS once it hits General
Availability later in CY18.
Benefits of ABFS include: * Higher scale (capacity, throughput, and IOPS) Big
Data and Analytics workloads by allowing higher limits on storage accounts
* Removing any ramp up time with Storage backend partitioning; blocks are now
automatically sharded across partitions in the Storage backend
** This avoids the need for using temporary/intermediate files, increasing the
cost (and framework complexity around committing jobs/tasks)
* Enabling much higher read and write throughput on single files (tens of Gbps
by default)
* Still retaining all of the Azure Blob features customers are fami
[jira] [Updated] (HADOOP-15407) Support Windows Azure Storage - Blob file system in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii updated HADOOP-15407:
-
Description:
{color:#33}Description{color}
This JIRA adds a new file system implementation, ABFS, for running Big Data and
Analytics workloads against Azure Storage. This is a complete rewrite of the
previous WASB driver with a heavy focus on optimizing both performance and cost.
{color:#33}High level design{color}
At a high level, the code here extends the FileSystem class to provide an
implementation for accessing blobs in Azure Storage. The scheme abfs is used
for accessing it over HTTP, and abfss for accessing over HTTPS. The following
URI scheme is used to address individual paths:
abfs[s]://@.dfs.core.windows.net/
{color:#33} {color}
ABFS is intended as a replacement to WASB. WASB is not deprecated but is in
pure maintenance mode and customers should upgrade to ABFS once it hits General
Availability later in CY18.
Benefits of ABFS include: * Higher scale (capacity, throughput, and IOPS) Big
Data and Analytics workloads by allowing higher limits on storage accounts
* Removing any ramp up time with Storage backend partitioning; blocks are now
automatically sharded across partitions in the Storage backend
** This avoids the need for using temporary/intermediate files, increasing the
cost (and framework complexity around committing jobs/tasks)
* Enabling much higher read and write throughput on single files (tens of Gbps
by default)
* Still retaining all of the Azure Blob features customers are familiar with
and expect, and gaining the benefits of future Blob features as well
ABFS incorporates Hadoop Filesystem metrics to monitor the file system
throughput and operations. Ambari metrics are not currently implemented for
ABFS, but will be available soon.
{color:#33}Credits and history{color}
Credit for this work goes to .
{color:#33}Test{color}
ABFS has gone through many test procedures including Hadoop file system
contract tests, unit testing, functional testing, and manual testing. All the
Junit tests provided with the driver are capable of running in both
sequential/parallel fashion in order to reduce the testing time.
Besides unit tests, we have used ABFS as the default file system in Azure
HDInsight. Azure HDInsight will very soon offer ABFS as a storage option. (HDFS
is also used but not as default file system.) Various different customer and
test workloads have been run against clusters with such configurations for
quite some time. Benchmarks such as Tera*, TPC-DS, Spark Streaming and Spark
SQL, and others have been run to do scenario, performance, and functional
testing. Third parties and customers have also done various testing of ABFS.
The current version reflects to the version of the code tested and used in our
production environment.
was:
{color:#212121}{color:#33}Description{color}{color}
{color:#212121}This JIRA adds a new file system implementation, ABFS, for
running Big Data and Analytics workloads against Azure Storage. This is a
complete rewrite of the previous WASB driver with a heavy focus on optimizing
both performance and cost.{color}
{color:#212121} {color}
{color:#212121}{color:#33}High level design{color}{color}
{color:#212121}At a high level, the code here extends the FileSystem class to
provide an implementation for accessing blobs in Azure Storage. The scheme abfs
is used for accessing it over HTTP, and abfss for accessing over HTTPS. The
following URI scheme is used to address individual paths:{color}
{color:#212121} {color}
{color:#212121}abfs[s]://@.dfs.core.windows.net/{color}
{color:#212121} {color}
{color:#212121}ABFS is intended as a replacement to WASB. WASB is not
deprecated but is in pure maintenance mode and customers should upgrade to ABFS
once it hits General Availability later in CY18.{color}
{color:#212121}Benefits of ABFS include:{color}
{color:#212121}· Higher scale (capacity, throughput, and IOPS) Big Data
and Analytics workloads by allowing higher limits on storage accounts{color}
{color:#212121}· Removing any ramp up time with Storage backend
partitioning; blocks are now automatically sharded across partitions in the
Storage backend{color}
{color:#212121}oThis avoids the need for using temporary/intermediate
files, increasing the cost (and framework complexity around committing
jobs/tasks){color}
{color:#212121}· Enabling much higher read and write throughput on
single files (tens of Gbps by default){color}
{color:#212121}· Still retaining all of the Azure Blob features
customers are familiar with and expect, and gaining the benefits of future Blob
features as well{color}
{color:#212121}ABFS incorporates Hadoop Filesystem metrics to monitor the file
system throughput and operations. Ambari metrics are not currently implemented
for ABFS
[jira] [Created] (HADOOP-15407) Support Windows Azure Storage - Blob file system in Hadoop
Esfandiar Manii created HADOOP-15407:
Summary: Support Windows Azure Storage - Blob file system in Hadoop
Key: HADOOP-15407
URL: https://issues.apache.org/jira/browse/HADOOP-15407
Project: Hadoop Common
Issue Type: New Feature
Components: fs/azure
Affects Versions: 3.2.0
Reporter: Esfandiar Manii
Assignee: Esfandiar Manii
{color:#212121}{color:#33}Description{color}{color}
{color:#212121}This JIRA adds a new file system implementation, ABFS, for
running Big Data and Analytics workloads against Azure Storage. This is a
complete rewrite of the previous WASB driver with a heavy focus on optimizing
both performance and cost.{color}
{color:#212121} {color}
{color:#212121}{color:#33}High level design{color}{color}
{color:#212121}At a high level, the code here extends the FileSystem class to
provide an implementation for accessing blobs in Azure Storage. The scheme abfs
is used for accessing it over HTTP, and abfss for accessing over HTTPS. The
following URI scheme is used to address individual paths:{color}
{color:#212121} {color}
{color:#212121}abfs[s]://@.dfs.core.windows.net/{color}
{color:#212121} {color}
{color:#212121}ABFS is intended as a replacement to WASB. WASB is not
deprecated but is in pure maintenance mode and customers should upgrade to ABFS
once it hits General Availability later in CY18.{color}
{color:#212121}Benefits of ABFS include:{color}
{color:#212121}· Higher scale (capacity, throughput, and IOPS) Big Data
and Analytics workloads by allowing higher limits on storage accounts{color}
{color:#212121}· Removing any ramp up time with Storage backend
partitioning; blocks are now automatically sharded across partitions in the
Storage backend{color}
{color:#212121}oThis avoids the need for using temporary/intermediate
files, increasing the cost (and framework complexity around committing
jobs/tasks){color}
{color:#212121}· Enabling much higher read and write throughput on
single files (tens of Gbps by default){color}
{color:#212121}· Still retaining all of the Azure Blob features
customers are familiar with and expect, and gaining the benefits of future Blob
features as well{color}
{color:#212121}ABFS incorporates Hadoop Filesystem metrics to monitor the file
system throughput and operations. Ambari metrics are not currently implemented
for ABFS, but will be available soon.{color}
{color:#212121} {color}
{color:#212121}{color:#33}Credits and history{color}{color}
{color:#212121}Credit for this work goes to (hope I don't forget anyone): Shane
Mainali, {color}{color:#212121}Thomas Marquardt, Zichen Sun, Georgi Chalakov,
Esfandiar Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal,
Saurabh Pant, and James Baker. {color}
{color:#212121}{color:#33} {color}{color}
{color:#212121}{color:#33}Test{color}{color}
{color:#212121}ABFS has gone through many test procedures including Hadoop file
system contract tests, unit testing, functional testing, and manual testing.
All the Junit tests provided with the driver are capable of running in both
sequential/parallel fashion in order to reduce the testing time.{color}
{color:#212121}Besides unit tests, we have used ABFS as the default file system
in Azure HDInsight. Azure HDInsight will very soon offer ABFS as a storage
option. (HDFS is also used but not as default file system.) Various different
customer and test workloads have been run against clusters with such
configurations for quite some time. Benchmarks such as Tera*, TPC-DS, Spark
Streaming and Spark SQL, and others have been run to do scenario, performance,
and functional testing. Third parties and customers have also done various
testing of ABFS.{color}
{color:#212121}The current version reflects to the version of the code tested
and used in our production environment.{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14583) wasb throws an exception if you try to create a file and there's no parent directory
[
https://issues.apache.org/jira/browse/HADOOP-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii updated HADOOP-14583:
-
Attachment: HADOOP-14583-branch-2-001.patch
> wasb throws an exception if you try to create a file and there's no parent
> directory
>
>
> Key: HADOOP-14583
> URL: https://issues.apache.org/jira/browse/HADOOP-14583
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Esfandiar Manii
>Priority: Minor
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14583-001.patch, HADOOP-14583-002.patch,
> HADOOP-14583-003.patch, HADOOP-14583-branch-2-001.patch
>
>
> It's a known defect of the Hadoop FS API (and one we don't explicitly test
> for enough), but you can create a file on a path which doesn't exist. In that
> situation, the create() logic is expectd to create the entries.
> Wasb appears to raise an exception if you try to call {{create(filepath)}}
> without calling {{mkdirs(filepath.getParent()}} first. That's the semantics
> expected of {{createNonRecursive()}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14583) wasb throws an exception if you try to create a file and there's no parent directory
[
https://issues.apache.org/jira/browse/HADOOP-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16170635#comment-16170635
]
Esfandiar Manii commented on HADOOP-14583:
--
Merged with branch-2 and attached the patch. Also here is the run result
{code:java}
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.431 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Running
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
Tests run: 29, Failures: 0, Errors: 0, Skipped: 29, Time elapsed: 3.263 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.918 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.147 sec - in
org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
Running org.apache.hadoop.fs.azure.TestWasbFsck
Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.704 sec - in
org.apache.hadoop.fs.azure.TestWasbFsck
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.195 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
Running org.apache.hadoop.fs.azure.TestContainerChecks
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.223 sec - in
org.apache.hadoop.fs.azure.TestContainerChecks
Running org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 202.591 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
Running org.apache.hadoop.fs.azure.TestBlobOperationDescriptor
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.033 sec - in
org.apache.hadoop.fs.azure.TestBlobOperationDescriptor
Running org.apache.hadoop.fs.azure.TestBlockBlobInputStream
Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 103.874 sec -
in org.apache.hadoop.fs.azure.TestBlockBlobInputStream
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.084 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 214.652 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Running org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
Tests run: 19, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 7.978 sec - in
org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
Running org.apache.hadoop.fs.azure.TestBlobDataValidation
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.261 sec - in
org.apache.hadoop.fs.azure.TestBlobDataValidation
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.047 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
Running org.apache.hadoop.fs.azure.TestBlobMetadata
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.777 sec - in
org.apache.hadoop.fs.azure.TestBlobMetadata
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.531 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Running
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.643 sec -
in
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 1.666 sec -
in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
Tests run: 35, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.668 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.432 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Running org.apache.hadoop.fs.azu
[jira] [Commented] (HADOOP-14583) wasb throws an exception if you try to create a file and there's no parent directory
[
https://issues.apache.org/jira/browse/HADOOP-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129174#comment-16129174
]
Esfandiar Manii commented on HADOOP-14583:
--
Thank you! In patch 002 I addressed the comments,
{code:java}
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.081 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Running
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
Tests run: 27, Failures: 0, Errors: 0, Skipped: 27, Time elapsed: 2.686 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.865 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.08 sec - in
org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
Running org.apache.hadoop.fs.azure.TestWasbFsck
Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.677 sec - in
org.apache.hadoop.fs.azure.TestWasbFsck
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 1.226 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
Running org.apache.hadoop.fs.azure.TestContainerChecks
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.937 sec - in
org.apache.hadoop.fs.azure.TestContainerChecks
Running org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 194.244 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
Running org.apache.hadoop.fs.azure.TestBlobOperationDescriptor
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.109 sec - in
org.apache.hadoop.fs.azure.TestBlobOperationDescriptor
Running org.apache.hadoop.fs.azure.TestBlockBlobInputStream
Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapse// Some comments
here
public String getFoo()
{
return foo;
}d: 91.494 sec - in org.apache.hadoop.fs.azure.TestBlockBlobInputStream
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.671 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 216.221 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Running org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
Tests run: 19, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 7.94 sec - in
org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
Running org.apache.hadoop.fs.azure.TestBlobDataValidation
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.532 sec - in
org.apache.hadoop.fs.azure.TestBlobDataValidation
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.897 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
Running org.apache.hadoop.fs.azure.TestBlobMetadata
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.617 sec - in
org.apache.hadoop.fs.azure.TestBlobMetadata
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.596 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Running
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.689 sec -
in
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 6.853 sec -
in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 34.944 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.249 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFil
[jira] [Updated] (HADOOP-14583) wasb throws an exception if you try to create a file and there's no parent directory
[
https://issues.apache.org/jira/browse/HADOOP-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii updated HADOOP-14583:
-
Attachment: HADOOP-14583-002.patch
> wasb throws an exception if you try to create a file and there's no parent
> directory
>
>
> Key: HADOOP-14583
> URL: https://issues.apache.org/jira/browse/HADOOP-14583
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Esfandiar Manii
>Priority: Minor
> Attachments: HADOOP-14583-001.patch, HADOOP-14583-002.patch
>
>
> It's a known defect of the Hadoop FS API (and one we don't explicitly test
> for enough), but you can create a file on a path which doesn't exist. In that
> situation, the create() logic is expectd to create the entries.
> Wasb appears to raise an exception if you try to call {{create(filepath)}}
> without calling {{mkdirs(filepath.getParent()}} first. That's the semantics
> expected of {{createNonRecursive()}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14583) wasb throws an exception if you try to create a file and there's no parent directory
[
https://issues.apache.org/jira/browse/HADOOP-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii updated HADOOP-14583:
-
Attachment: (was: HADOOP-14583-001.patch)
> wasb throws an exception if you try to create a file and there's no parent
> directory
>
>
> Key: HADOOP-14583
> URL: https://issues.apache.org/jira/browse/HADOOP-14583
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Esfandiar Manii
>Priority: Minor
> Attachments: HADOOP-14583-001.patch
>
>
> It's a known defect of the Hadoop FS API (and one we don't explicitly test
> for enough), but you can create a file on a path which doesn't exist. In that
> situation, the create() logic is expectd to create the entries.
> Wasb appears to raise an exception if you try to call {{create(filepath)}}
> without calling {{mkdirs(filepath.getParent()}} first. That's the semantics
> expected of {{createNonRecursive()}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14583) wasb throws an exception if you try to create a file and there's no parent directory
[
https://issues.apache.org/jira/browse/HADOOP-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii updated HADOOP-14583:
-
Attachment: HADOOP-14583-001.patch
> wasb throws an exception if you try to create a file and there's no parent
> directory
>
>
> Key: HADOOP-14583
> URL: https://issues.apache.org/jira/browse/HADOOP-14583
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Esfandiar Manii
>Priority: Minor
> Attachments: HADOOP-14583-001.patch
>
>
> It's a known defect of the Hadoop FS API (and one we don't explicitly test
> for enough), but you can create a file on a path which doesn't exist. In that
> situation, the create() logic is expectd to create the entries.
> Wasb appears to raise an exception if you try to call {{create(filepath)}}
> without calling {{mkdirs(filepath.getParent()}} first. That's the semantics
> expected of {{createNonRecursive()}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14583) wasb throws an exception if you try to create a file and there's no parent directory
[
https://issues.apache.org/jira/browse/HADOOP-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128044#comment-16128044
]
Esfandiar Manii commented on HADOOP-14583:
--
This is not related to if parent directory does not exist. The issue is
concurrency. If there are multiple threads trying to create and delete the same
file over and over, you will be having a scenario where:
ThreadA -> Creates File
ThreadB -> Looks up the file and it exists
ThreadA -> Removes file
ThreadB -> Looks up for metadata before creation and throws exception
The logic to make the metadata lookup safe was missed from the create function
where most of the other ones have this logic. I updated the code and added a
test to ensure this wont happen under over 100 threads.
Please take a look over the patch and let me know if you have comments.
> wasb throws an exception if you try to create a file and there's no parent
> directory
>
>
> Key: HADOOP-14583
> URL: https://issues.apache.org/jira/browse/HADOOP-14583
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-14583-001.patch
>
>
> It's a known defect of the Hadoop FS API (and one we don't explicitly test
> for enough), but you can create a file on a path which doesn't exist. In that
> situation, the create() logic is expectd to create the entries.
> Wasb appears to raise an exception if you try to call {{create(filepath)}}
> without calling {{mkdirs(filepath.getParent()}} first. That's the semantics
> expected of {{createNonRecursive()}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14583) wasb throws an exception if you try to create a file and there's no parent directory
[
https://issues.apache.org/jira/browse/HADOOP-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii reassigned HADOOP-14583:
Assignee: Esfandiar Manii
> wasb throws an exception if you try to create a file and there's no parent
> directory
>
>
> Key: HADOOP-14583
> URL: https://issues.apache.org/jira/browse/HADOOP-14583
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Esfandiar Manii
>Priority: Minor
> Attachments: HADOOP-14583-001.patch
>
>
> It's a known defect of the Hadoop FS API (and one we don't explicitly test
> for enough), but you can create a file on a path which doesn't exist. In that
> situation, the create() logic is expectd to create the entries.
> Wasb appears to raise an exception if you try to call {{create(filepath)}}
> without calling {{mkdirs(filepath.getParent()}} first. That's the semantics
> expected of {{createNonRecursive()}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14583) wasb throws an exception if you try to create a file and there's no parent directory
[
https://issues.apache.org/jira/browse/HADOOP-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128048#comment-16128048
]
Esfandiar Manii commented on HADOOP-14583:
--
{code:java}
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.299 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Running
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
Tests run: 27, Failures: 0, Errors: 0, Skipped: 27, Time elapsed: 2.643 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.891 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.085 sec - in
org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
Running org.apache.hadoop.fs.azure.TestWasbFsck
Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.704 sec - in
org.apache.hadoop.fs.azure.TestWasbFsck
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 1.138 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
Running org.apache.hadoop.fs.azure.TestContainerChecks
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.349 sec - in
org.apache.hadoop.fs.azure.TestContainerChecks
Running org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 199.932 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
Running org.apache.hadoop.fs.azure.TestBlobOperationDescriptor
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.918 sec - in
org.apache.hadoop.fs.azure.TestBlobOperationDescriptor
Running org.apache.hadoop.fs.azure.TestBlockBlobInputStream
Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 92.134 sec -
in org.apache.hadoop.fs.azure.TestBlockBlobInputStream
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.767 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 209.062 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
Running org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
Tests run: 19, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 7.902 sec - in
org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
Running org.apache.hadoop.fs.azure.TestBlobDataValidation
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.181 sec - in
org.apache.hadoop.fs.azure.TestBlobDataValidation
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.868 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
Running org.apache.hadoop.fs.azure.TestBlobMetadata
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.742 sec - in
org.apache.hadoop.fs.azure.TestBlobMetadata
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.375 sec - in
org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
Running
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.124 sec -
in
org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 1.514 sec -
in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 31.019 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.53 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
Tests run: 6, Failures: 0, Errors
[jira] [Updated] (HADOOP-14583) wasb throws an exception if you try to create a file and there's no parent directory
[
https://issues.apache.org/jira/browse/HADOOP-14583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii updated HADOOP-14583:
-
Attachment: HADOOP-14583-001.patch
> wasb throws an exception if you try to create a file and there's no parent
> directory
>
>
> Key: HADOOP-14583
> URL: https://issues.apache.org/jira/browse/HADOOP-14583
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-14583-001.patch
>
>
> It's a known defect of the Hadoop FS API (and one we don't explicitly test
> for enough), but you can create a file on a path which doesn't exist. In that
> situation, the create() logic is expectd to create the entries.
> Wasb appears to raise an exception if you try to call {{create(filepath)}}
> without calling {{mkdirs(filepath.getParent()}} first. That's the semantics
> expected of {{createNonRecursive()}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14769) WASB: delete recursive should not fail if a file is deleted
[
https://issues.apache.org/jira/browse/HADOOP-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16127831#comment-16127831
]
Esfandiar Manii commented on HADOOP-14769:
--
+1 with few comments:
AzureNativeFileSystemStore.java L2503-2505: Not sure how much we want to invest
on this but there are many of this code everywhere, I wish there was only one
method doing this.
NativeAzureFileSystem.java L2099-2108: instead of nested ifs please rewrite it
to be like (for better code clarity):
if (!store.delete(path)) {
return false;
}
if (isDir) {
}
else {
}
return true;
TestFileSystemOperationsWithThreads.java L592-594: nit: Please fix indentation
> WASB: delete recursive should not fail if a file is deleted
> ---
>
> Key: HADOOP-14769
> URL: https://issues.apache.org/jira/browse/HADOOP-14769
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Thomas Marquardt
> Attachments: HADOOP-14769-001.patch
>
>
> FileSystem.delete(Path path) and delete(Path path, boolean recursive) return
> false if the path does not exist. The WASB implementation of recursive
> delete currently fails if one of the entries is deleted by an external agent
> while a recursive delete is in progress. For example, if you try to delete
> all of the files in a directory, which can be a very long process, and one of
> the files contained within is deleted by an external agent, the recursive
> directory delete operation will fail if it tries to delete that file and
> discovers that it does not exist. This is not desirable. A recursive
> directory delete operation should succeeed if the directory initially exists
> and when the operation completes, the directory and all of its entries do not
> exist.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124090#comment-16124090
]
Esfandiar Manii commented on HADOOP-14553:
--
Looked at Allen's work. The fix is only required for your patch. Its building
for windows without your change and the only reason is the embedded JS code :)
> Add (parallelized) integration tests to hadoop-azure
>
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch,
> HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch,
> HADOOP-14553-009.patch, HADOOP-14553-010.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM &c show what to do.
> *UPDATE August 4, 2017*: Adding a list of requirements to clarify the
> acceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii)
> IntegrationTests - requires Azure Storage account, executed by engineers
> prior to check-in, and if needed, iii) ScaleTests – long running performance
> and scalability tests.
> # Define configuration profiles to run tests with different settings. Allows
> an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and
> false. Need to review settings to see what else would benefit.
> # Maven commands to run b) and c). Turns out it is not easy to do with
> Maven, so we might have to run it multiple times to run with different
> configuration settings.
> # Document how to add and run tests and the process for contributing to
> Apache Hadoop. Steve shared an example at
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> # UnitTests should run in under 2 minutes and IntegrationTests should run in
> under 15 minutes, even on slower network connections. (These are rough goals)
> # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large
> persistent content used repeatedly to expedite test execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123802#comment-16123802
]
Esfandiar Manii commented on HADOOP-14553:
--
around 8 mins and 30 seconds :)
> Add (parallelized) integration tests to hadoop-azure
>
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch,
> HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch,
> HADOOP-14553-009.patch, HADOOP-14553-010.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM &c show what to do.
> *UPDATE August 4, 2017*: Adding a list of requirements to clarify the
> acceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii)
> IntegrationTests - requires Azure Storage account, executed by engineers
> prior to check-in, and if needed, iii) ScaleTests – long running performance
> and scalability tests.
> # Define configuration profiles to run tests with different settings. Allows
> an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and
> false. Need to review settings to see what else would benefit.
> # Maven commands to run b) and c). Turns out it is not easy to do with
> Maven, so we might have to run it multiple times to run with different
> configuration settings.
> # Document how to add and run tests and the process for contributing to
> Apache Hadoop. Steve shared an example at
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> # UnitTests should run in under 2 minutes and IntegrationTests should run in
> under 15 minutes, even on slower network connections. (These are rough goals)
> # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large
> persistent content used repeatedly to expedite test execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123728#comment-16123728
]
Esfandiar Manii commented on HADOOP-14553:
--
Ok, found the solution, instead of direct placeholders we can use
project.getProperty and that will ensure backslashes are interpreted correctly.
Please update your JavaScript to the following and everything works smoothly.
{code:javascript}
{code}
> Add (parallelized) integration tests to hadoop-azure
>
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch,
> HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch,
> HADOOP-14553-009.patch, HADOOP-14553-010.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM &c show what to do.
> *UPDATE August 4, 2017*: Adding a list of requirements to clarify the
> acceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii)
> IntegrationTests - requires Azure Storage account, executed by engineers
> prior to check-in, and if needed, iii) ScaleTests – long running performance
> and scalability tests.
> # Define configuration profiles to run tests with different settings. Allows
> an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and
> false. Need to review settings to see what else would benefit.
> # Maven commands to run b) and c). Turns out it is not easy to do with
> Maven, so we might have to run it multiple times to run with different
> configuration settings.
> # Document how to add and run tests and the process for contributing to
> Apache Hadoop. Steve shared an example at
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> # UnitTests should run in under 2 minutes and IntegrationTests should run in
> under 15 minutes, even on slower network connections. (These are rough goals)
> # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large
> persistent content used repeatedly to expedite test execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123593#comment-16123593
]
Esfandiar Manii commented on HADOOP-14553:
--
All the three have issues since the paths are including one backslash for file
separator and JavaScript interprets them as escape sequence characters.
So whichever code is taking the JavaScript string between the target tags in
the pom.xml and replacing those ${test.build.data}, ... variables with their
values, should make sure that test.build.data has two back slashes instead of
one.
> Add (parallelized) integration tests to hadoop-azure
>
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch,
> HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch,
> HADOOP-14553-009.patch, HADOOP-14553-010.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM &c show what to do.
> *UPDATE August 4, 2017*: Adding a list of requirements to clarify the
> acceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii)
> IntegrationTests - requires Azure Storage account, executed by engineers
> prior to check-in, and if needed, iii) ScaleTests – long running performance
> and scalability tests.
> # Define configuration profiles to run tests with different settings. Allows
> an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and
> false. Need to review settings to see what else would benefit.
> # Maven commands to run b) and c). Turns out it is not easy to do with
> Maven, so we might have to run it multiple times to run with different
> configuration settings.
> # Document how to add and run tests and the process for contributing to
> Apache Hadoop. Steve shared an example at
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> # UnitTests should run in under 2 minutes and IntegrationTests should run in
> under 15 minutes, even on slower network connections. (These are rough goals)
> # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large
> persistent content used repeatedly to expedite test execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122497#comment-16122497
]
Esfandiar Manii edited comment on HADOOP-14553 at 8/10/17 11:05 PM:
Actually
{code:java}
"${test.build.data.replace('\\', '')}",
"${test.build.dir.replace('\\', '')}",
"${hadoop.tmp.dir.replace('\\', '')}"];
{code}
is not working well. It seems it created wrong directories under hadoop-azure.
Is there any easy way to print two back slashes through pom.xml configurations?
Currently the issue is you are doing
{code:java}
var baseDirs = [
"${test.build.data}",
"${test.build.dir}",
"${hadoop.tmp.dir}"]
{code}
in windows if test.build.data is f:\a\b
in JS you will have
baseDirs = [ 'f'] since \a \b are escape sequence chars and the path is invalid
in Windows.
was (Author: esmanii):
Actually "${test.build.data.replace('\\', '')}",
"${test.build.dir.replace('\\', '')}",
"${hadoop.tmp.dir.replace('\\', '')}"];
is not working well. It seems it created wrong directories under hadoop-azure.
Is there any easy way to print two back slashes through pom.xml configurations?
Currently the issue is you are doing
var baseDirs = [
"${test.build.data}",
"${test.build.dir}",
"${hadoop.tmp.dir}"]
in windows if test.build.data is f:\a\b
in JS you will have
baseDirs = [ 'f'] since \a \b are escape sequence chars and the path is invalid
in Windows.
> Add (parallelized) integration tests to hadoop-azure
>
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch,
> HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch,
> HADOOP-14553-009.patch, HADOOP-14553-010.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM &c show what to do.
> *UPDATE August 4, 2017*: Adding a list of requirements to clarify the
> acceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii)
> IntegrationTests - requires Azure Storage account, executed by engineers
> prior to check-in, and if needed, iii) ScaleTests – long running performance
> and scalability tests.
> # Define configuration profiles to run tests with different settings. Allows
> an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and
> false. Need to review settings to see what else would benefit.
> # Maven commands to run b) and c). Turns out it is not easy to do with
> Maven, so we might have to run it multiple times to run with different
> configuration settings.
> # Document how to add and run tests and the process for contributing to
> Apache Hadoop. Steve shared an example at
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> # UnitTests should run in under 2 minutes and IntegrationTests should run in
> under 15 minutes, even on slower network connections. (These are rough goals)
> # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large
> persistent content used repeatedly to expedite test execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122497#comment-16122497
]
Esfandiar Manii commented on HADOOP-14553:
--
Actually "${test.build.data.replace('\\', '')}",
"${test.build.dir.replace('\\', '')}",
"${hadoop.tmp.dir.replace('\\', '')}"];
is not working well. It seems it created wrong directories under hadoop-azure.
Is there any easy way to print two back slashes through pom.xml configurations?
Currently the issue is you are doing
var baseDirs = [
"${test.build.data}",
"${test.build.dir}",
"${hadoop.tmp.dir}"]
in windows if test.build.data is f:\a\b
in JS you will have
baseDirs = [ 'f'] since \a \b are escape sequence chars and the path is invalid
in Windows.
> Add (parallelized) integration tests to hadoop-azure
>
>
> Key: HADOOP-14553
> URL: https://issues.apache.org/jira/browse/HADOOP-14553
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14553-001.patch, HADOOP-14553-002.patch,
> HADOOP-14553-003.patch, HADOOP-14553-004.patch, HADOOP-14553-005.patch,
> HADOOP-14553-006.patch, HADOOP-14553-007.patch, HADOOP-14553-008.patch,
> HADOOP-14553-009.patch, HADOOP-14553-010.patch
>
>
> The Azure tests are slow to run as they are serialized, as they are all
> called Test* there's no clear differentiation from unit tests which Jenkins
> can run, and integration tests which it can't.
> Move the azure tests {{Test*}} to integration tests {{ITest*}}, parallelize
> (which includes having separate paths for every test suite). The code in
> hadoop-aws's POM &c show what to do.
> *UPDATE August 4, 2017*: Adding a list of requirements to clarify the
> acceptance criteria for this JIRA:
> # Parallelize test execution
> # Define test groups: i) UnitTests - self-contained, executed by Jenkins, ii)
> IntegrationTests - requires Azure Storage account, executed by engineers
> prior to check-in, and if needed, iii) ScaleTests – long running performance
> and scalability tests.
> # Define configuration profiles to run tests with different settings. Allows
> an engineer to run “IntegrationTests” with fs.azure.secure.mode = true and
> false. Need to review settings to see what else would benefit.
> # Maven commands to run b) and c). Turns out it is not easy to do with
> Maven, so we might have to run it multiple times to run with different
> configuration settings.
> # Document how to add and run tests and the process for contributing to
> Apache Hadoop. Steve shared an example at
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
>
> # UnitTests should run in under 2 minutes and IntegrationTests should run in
> under 15 minutes, even on slower network connections. (These are rough goals)
> # Ensure test data (containers/blobs/etc) is deleted. Exceptions for large
> persistent content used repeatedly to expedite test execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure
[
https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122404#comment-16122404
]
Esfandiar Manii commented on HADOOP-14553:
--
All the tests passed in windows after I applied a fix to the pom.xml file
{code:java}
var baseDirs = [
"${test.build.data.replace('\\', '')}",
"${test.build.dir.replace('\\', '')}",
"${hadoop.tmp.dir.replace('\\', '')}"];
for (var i in baseDirs) {
for (var j = 1; j <= ${testsThreadCount}; ++j) {
var mkdir = project.createTask("mkdir");
mkdir.setDir(new java.io.File(baseDirs[i], j));
mkdir.perform();
}
}
{code}
{code:java}
$ mvn -T 1C -Dparallel-tests clean verify
[INFO] Scanning for projects...
[INFO]
[INFO] Using the MultiThreadedBuilder implementation with a thread count of 8
[INFO]
[INFO]
[INFO] Building Apache Hadoop Azure support 3.0.0-beta1-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-azure ---
[INFO] Deleting F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\target
[INFO] Deleting F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure (includes =
[dependency-reduced-pom.xml], excludes = [])
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-azure ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\target\test-dir
[INFO] Executed tasks
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-azure
---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hadoop-azure ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-azure
---
[INFO] Compiling 62 source files to
F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\target\classes
[WARNING]
/F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java:
Some input files use or override a deprecated API.
[WARNING]
/F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/StorageInterfaceImpl.java:
Recompile with -Xlint:deprecation for details.
[WARNING]
/F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java:
F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\src\main\java\org\apache\hadoop\fs\azure\security\TokenUtils.java
uses unchecked or unsafe operations.
[WARNING]
/F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java:
Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-dependency-plugin:2.10:list (deplist) @ hadoop-azure ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hadoop-azure ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 5 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hadoop-azure ---
[INFO] Compiling 69 source files to
F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\target\test-classes
[WARNING]
/F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java:
F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\src\test\java\org\apache\hadoop\fs\azure\NativeAzureFileSystemBaseTest.java
uses or overrides a deprecated API.
[WARNING]
/F:/hadoop-trunk/hadoop/hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java:
Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-parallel-tests-dirs) @
hadoop-azure ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-azure ---
[INFO] Surefire report directory:
F:\hadoop-trunk\hadoop\hadoop-tools\hadoop-azure\target\surefire-reports
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.458 sec - in
org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Running
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.339 sec - in
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAve
[jira] [Commented] (HADOOP-14183) Remove service loader config file for wasb fs
[ https://issues.apache.org/jira/browse/HADOOP-14183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16120638#comment-16120638 ] Esfandiar Manii commented on HADOOP-14183: -- --- T E S T S --- --- T E S T S --- Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.768 sec - in org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner Tests run: 27, Failures: 0, Errors: 0, Skipped: 27, Time elapsed: 3.028 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.996 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider Tests run: 2, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 0.107 sec - in org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider Running org.apache.hadoop.fs.azure.TestWasbFsck Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.608 sec - in org.apache.hadoop.fs.azure.TestWasbFsck Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 1.282 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked Running org.apache.hadoop.fs.azure.TestContainerChecks Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.104 sec - in org.apache.hadoop.fs.azure.TestContainerChecks Running org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 195.661 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive Running org.apache.hadoop.fs.azure.TestBlobOperationDescriptor Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.018 sec - in org.apache.hadoop.fs.azure.TestBlobOperationDescriptor Running org.apache.hadoop.fs.azure.TestBlockBlobInputStream Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 109.495 sec - in org.apache.hadoop.fs.azure.TestBlockBlobInputStream Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.735 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 210.089 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive Running org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration Tests run: 19, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 7.997 sec - in org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration Running org.apache.hadoop.fs.azure.TestBlobDataValidation Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.15 sec - in org.apache.hadoop.fs.azure.TestBlobDataValidation Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.901 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency Running org.apache.hadoop.fs.azure.TestBlobMetadata Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.725 sec - in org.apache.hadoop.fs.azure.TestBlobMetadata Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.471 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging Running org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.597 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 1.499 sec - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 35.392 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.083 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename Tests run: 6, Failures: 0, Errors: 0, Skipped
[jira] [Updated] (HADOOP-14183) Remove service loader config file for wasb fs
[
https://issues.apache.org/jira/browse/HADOOP-14183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii updated HADOOP-14183:
-
Attachment: HADOOP-14183.001.patch
> Remove service loader config file for wasb fs
> -
>
> Key: HADOOP-14183
> URL: https://issues.apache.org/jira/browse/HADOOP-14183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.7.3
>Reporter: John Zhuge
>Assignee: Esfandiar Manii
>Priority: Minor
> Labels: newbie
> Attachments: HADOOP-14183.001.patch
>
>
> Per discussion in HADOOP-14132. Remove the service loader config file
> hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> and add property {{fs.wasb.impl}} to {{core-default.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14183) Remove service loader config file for wasb fs
[
https://issues.apache.org/jira/browse/HADOOP-14183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii reassigned HADOOP-14183:
Assignee: Esfandiar Manii
> Remove service loader config file for wasb fs
> -
>
> Key: HADOOP-14183
> URL: https://issues.apache.org/jira/browse/HADOOP-14183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.7.3
>Reporter: John Zhuge
>Assignee: Esfandiar Manii
>Priority: Minor
> Labels: newbie
>
> Per discussion in HADOOP-14132. Remove the service loader config file
> hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
> and add property {{fs.wasb.impl}} to {{core-default.xml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14748) Wasb input streams to implement CanUnbuffer
[
https://issues.apache.org/jira/browse/HADOOP-14748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii reassigned HADOOP-14748:
Assignee: Esfandiar Manii
> Wasb input streams to implement CanUnbuffer
> ---
>
> Key: HADOOP-14748
> URL: https://issues.apache.org/jira/browse/HADOOP-14748
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Esfandiar Manii
>Priority: Minor
>
> HBase relies on FileSystems implementing CanUnbuffer.unbuffer() to force
> input streams to free up remote connections (HBASE-9393Link). This works for
> HDFS, but not elsewhere.
> WASB {{BlockBlobInputStream}} can implement this by closing the stream
> in ({{closeBlobInputStream}}, so it will be re-opened elsewhere.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14748) Wasb input streams to implement CanUnbuffer
[
https://issues.apache.org/jira/browse/HADOOP-14748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii reassigned HADOOP-14748:
Assignee: (was: Esfandiar Manii)
> Wasb input streams to implement CanUnbuffer
> ---
>
> Key: HADOOP-14748
> URL: https://issues.apache.org/jira/browse/HADOOP-14748
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
>
> HBase relies on FileSystems implementing CanUnbuffer.unbuffer() to force
> input streams to free up remote connections (HBASE-9393Link). This works for
> HDFS, but not elsewhere.
> WASB {{BlockBlobInputStream}} can implement this by closing the stream
> in ({{closeBlobInputStream}}, so it will be re-opened elsewhere.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14715) TestWasbRemoteCallHelper failing
[
https://issues.apache.org/jira/browse/HADOOP-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118826#comment-16118826
]
Esfandiar Manii commented on HADOOP-14715:
--
All the tests ran against:
wasb://testcontai...@xhdfs.blob.core.windows.net
*When secure mode is on and authorization caching is enabled in azure-test.xml*
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 137.797 sec -
in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Results :
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 02:20 min
[INFO] Finished at: 2017-08-08T18:18:36+00:00
[INFO] Final Memory: 22M/315M
[INFO]
*When secure mode is on and authorization caching is disabled in azure-test.xml*
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 147.801 sec -
in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Results :
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 02:32 min
[INFO] Finished at: 2017-08-08T18:24:54+00:00
[INFO] Final Memory: 35M/283M
[INFO]
> TestWasbRemoteCallHelper failing
>
>
> Key: HADOOP-14715
> URL: https://issues.apache.org/jira/browse/HADOOP-14715
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0, 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Esfandiar Manii
> Attachments: HADOOP-14715-001.patch, HADOOP-14715-002.patch,
> HADOOP-14715-003.patch
>
>
> {{org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.testWhenOneInstanceIsDown}}
> is failing for me on trunk
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14715) TestWasbRemoteCallHelper failing
[
https://issues.apache.org/jira/browse/HADOOP-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117441#comment-16117441
]
Esfandiar Manii commented on HADOOP-14715:
--
Updated comments from Thomas
> TestWasbRemoteCallHelper failing
>
>
> Key: HADOOP-14715
> URL: https://issues.apache.org/jira/browse/HADOOP-14715
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0, 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Esfandiar Manii
> Attachments: HADOOP-14715-001.patch, HADOOP-14715-002.patch,
> HADOOP-14715-003.patch
>
>
> {{org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.testWhenOneInstanceIsDown}}
> is failing for me on trunk
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14715) TestWasbRemoteCallHelper failing
[
https://issues.apache.org/jira/browse/HADOOP-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117441#comment-16117441
]
Esfandiar Manii edited comment on HADOOP-14715 at 8/7/17 10:26 PM:
---
Updated wrt comments from Thomas
was (Author: esmanii):
Updated comments from Thomas
> TestWasbRemoteCallHelper failing
>
>
> Key: HADOOP-14715
> URL: https://issues.apache.org/jira/browse/HADOOP-14715
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0, 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Esfandiar Manii
> Attachments: HADOOP-14715-001.patch, HADOOP-14715-002.patch,
> HADOOP-14715-003.patch
>
>
> {{org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.testWhenOneInstanceIsDown}}
> is failing for me on trunk
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14715) TestWasbRemoteCallHelper failing
[
https://issues.apache.org/jira/browse/HADOOP-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii updated HADOOP-14715:
-
Attachment: HADOOP-14715-003.patch
> TestWasbRemoteCallHelper failing
>
>
> Key: HADOOP-14715
> URL: https://issues.apache.org/jira/browse/HADOOP-14715
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0, 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Esfandiar Manii
> Attachments: HADOOP-14715-001.patch, HADOOP-14715-002.patch,
> HADOOP-14715-003.patch
>
>
> {{org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.testWhenOneInstanceIsDown}}
> is failing for me on trunk
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14715) TestWasbRemoteCallHelper failing
[
https://issues.apache.org/jira/browse/HADOOP-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esfandiar Manii updated HADOOP-14715:
-
Attachment: HADOOP-14715-002.patch
Updated the patch to handle scenarios where authorization caching is
enabled/disabled and set secure mode to be disabled by default.
> TestWasbRemoteCallHelper failing
>
>
> Key: HADOOP-14715
> URL: https://issues.apache.org/jira/browse/HADOOP-14715
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0, 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Esfandiar Manii
> Attachments: HADOOP-14715-001.patch, HADOOP-14715-002.patch
>
>
> {{org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.testWhenOneInstanceIsDown}}
> is failing for me on trunk
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14598) Wasb connection failing: FsUrlConnection cannot be cast to HttpURLConnection
[
https://issues.apache.org/jira/browse/HADOOP-14598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114683#comment-16114683
]
Esfandiar Manii commented on HADOOP-14598:
--
+1, thanks!
> Wasb connection failing: FsUrlConnection cannot be cast to HttpURLConnection
>
>
> Key: HADOOP-14598
> URL: https://issues.apache.org/jira/browse/HADOOP-14598
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, test
>Affects Versions: 2.9.0, 3.0.0-alpha4
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-14598-002.patch, HADOOP-14598-003.patch,
> HADOOP-14598-004.patch, HADOOP-14598-005.patch
>
>
> my downstream-of-spark cloud integration tests (where I haven't been running
> the azure ones for a while) now have a few of the tests failing
> {code}
> org.apache.hadoop.fs.azure.AzureException:
> com.microsoft.azure.storage.StorageException:
> org.apache.hadoop.fs.FsUrlConnection cannot be cast to
> java.net.HttpURLConnection
> {code}
> No obvious cause, and it's only apparently happening in some of the
> (scalatest) tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14722) Azure: BlockBlobInputStream position incorrect after seek
[ https://issues.apache.org/jira/browse/HADOOP-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114615#comment-16114615 ] Esfandiar Manii commented on HADOOP-14722: -- +1 On my side too, thanks! > Azure: BlockBlobInputStream position incorrect after seek > - > > Key: HADOOP-14722 > URL: https://issues.apache.org/jira/browse/HADOOP-14722 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt > Attachments: HADOOP-14722-001.patch, HADOOP-14722-002.patch, > HADOOP-14722-003.patch > > > The seek, skip, and getPos methods of BlockBlobInputStream do not correctly > account for the stream's internal buffer. This results in invalid stream > positions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14598) Wasb connection failing: FsUrlConnection cannot be cast to HttpURLConnection
[
https://issues.apache.org/jira/browse/HADOOP-14598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113688#comment-16113688
]
Esfandiar Manii commented on HADOOP-14598:
--
FsUrlStreamHandlerFactory L73-74, Could you please add a few lines of comment
on why the protocols are added there so the reason wont be forgotten in the
future.
FsUrlStreamHandlerFactory L73-74, I would create a util/private method which
gets/(exists in) the factory and call put on all the list of protocols.
TestUrlStreamHandler.java, do we also need to include a test for invalid
protocols?
> Wasb connection failing: FsUrlConnection cannot be cast to HttpURLConnection
>
>
> Key: HADOOP-14598
> URL: https://issues.apache.org/jira/browse/HADOOP-14598
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, test
>Affects Versions: 2.9.0, 3.0.0-alpha4
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-14598-002.patch, HADOOP-14598-003.patch,
> HADOOP-14598-004.patch
>
>
> my downstream-of-spark cloud integration tests (where I haven't been running
> the azure ones for a while) now have a few of the tests failing
> {code}
> org.apache.hadoop.fs.azure.AzureException:
> com.microsoft.azure.storage.StorageException:
> org.apache.hadoop.fs.FsUrlConnection cannot be cast to
> java.net.HttpURLConnection
> {code}
> No obvious cause, and it's only apparently happening in some of the
> (scalatest) tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14715) TestWasbRemoteCallHelper failing
[
https://issues.apache.org/jira/browse/HADOOP-14715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113462#comment-16113462
]
Esfandiar Manii commented on HADOOP-14715:
--
This is a regression, I am preparing a fix
> TestWasbRemoteCallHelper failing
>
>
> Key: HADOOP-14715
> URL: https://issues.apache.org/jira/browse/HADOOP-14715
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0, 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14715-001.patch
>
>
> {{org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.testWhenOneInstanceIsDown}}
> is failing for me on trunk
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14722) Azure: BlockBlobInputStream position incorrect after seek
[
https://issues.apache.org/jira/browse/HADOOP-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113101#comment-16113101
]
Esfandiar Manii edited comment on HADOOP-14722 at 8/3/17 5:34 PM:
--
BlockBlobInputStream.java: L92-94: streamPosition - streamBufferLength +
streamBufferPosition, can this become negative?
BlockBlobInputStream.java: L133: don't we need to nullify streamBuffer too?
BlockBlobInputStream.java: L321-323: Why dont you throw the exception right at
the beginning?
BlockBlobInputStream.java: L314: Overall I am not a big fan of having nested if
and elses because its making code more complicated that needed. lets just
return instead of creating else.
For example
public synchronized long skip(long n) throws IOException {
checkState();
long skipped;
if (blobInputStream != null) {
skipped = blobInputStream.skip(n);
streamPosition += skipped;
return skipped;
}
if (n < 0 || n > streamLength - streamPosition) {
throw new IndexOutOfBoundsException("skip range");
}
if (streamBuffer == null) {
streamPosition += n;
return n;
}
if (n < streamBufferLength - streamBufferPosition) {
streamBufferPosition += (int) n;
} else {
streamBufferPosition = 0;
streamBufferLength = 0;
streamPosition = getPos() + n;
}
return skipped;
}
BlockBlobInputStream.java: L330: I'd suggest create a private method which
clears the buffer and get rid of all the custom streamBufferPosition = 0;
streamBufferLength = 0 and etc.
was (Author: esmanii):
BlockBlobInputStream.java: L92-94: streamPosition - streamBufferLength +
streamBufferPosition, can this become negative?
BlockBlobInputStream.java: L133: don't we need to nullify streamBuffer too?
BlockBlobInputStream.java: L321-323: Why dont you throw the exception right at
the beginning?
BlockBlobInputStream.java: L314: Overall I am not a big fan of having nested if
and elses because its making code more complicated that needed. lets just
return instead of creating else.
For example
public synchronized long skip(long n) throws IOException {
checkState();
long skipped;
if (blobInputStream != null) {
skipped = blobInputStream.skip(n);
streamPosition += skipped;
return skipped;
}
if (n < 0 || n > streamLength - streamPosition) {
throw new IndexOutOfBoundsException("skip range");
}
if (streamBuffer == null) {
streamPosition += n;
return n;
}
if (n < streamBufferLength - streamBufferPosition) {
streamBufferPosition += (int) n;
} else {
streamBufferPosition = 0;
streamBufferLength = 0;
streamPosition = getPos() + n;
}
return skipped;
}
BlockBlobInputStream.java: L330: I'd suggest clear a private method which
clears the buffer and get rid of all the custom streamBufferPosition = 0;
streamBufferLength = 0 and etc.
> Azure: BlockBlobInputStream position incorrect after seek
> -
>
> Key: HADOOP-14722
> URL: https://issues.apache.org/jira/browse/HADOOP-14722
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Thomas Marquardt
> Attachments: HADOOP-14722-001.patch, HADOOP-14722-002.patch
>
>
> The seek, skip, and getPos methods of BlockBlobInputStream do not correctly
> account for the stream's internal buffer. This results in invalid stream
> positions.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
