[jira] [Commented] (HADOOP-17215) ABFS: Support for conditional overwrite
[ https://issues.apache.org/jira/browse/HADOOP-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17703465#comment-17703465 ] Thomas Marqardt commented on HADOOP-17215: -- [~ste...@apache.org] it might be worth revisiting why this change was made and whether it is necessary. [~snvijaya] may have more context. > ABFS: Support for conditional overwrite > --- > > Key: HADOOP-17215 > URL: https://issues.apache.org/jira/browse/HADOOP-17215 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: abfsactive > Fix For: 3.3.1, 3.4.0 > > > Filesystem Create APIs that do not accept an argument for overwrite flag end > up defaulting it to true. > We are observing that request count of creates with overwrite=true is more > and primarily because of the default setting of the flag is true of the > called Create API. When a create with overwrite ends up timing out, we have > observed that it could lead to race conditions between the first create and > retried one running almost parallel. > To avoid this scenario for create with overwrite=true request, ABFS driver > will always attempt to create without overwrite. If the create fails due to > fileAlreadyPresent, it will resend the request with overwrite=true. > > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17704) HADOOP-16916 changed interface SASTokenProvider fields, breaking compatibility between 3.3.0 and 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-17704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17346535#comment-17346535 ] Thomas Marqardt commented on HADOOP-17704: -- It's unfortunate that an error was made and HADOOP-16730 was committed in 3.3.0 instead of 3.3.1. The SASTokenProvider interface and implementation was a collaboration between Microsoft and Cloudera, and CDP has a dependency on the latest version of the SASTokenProvider interface (not the initial one). The interface is attributed with @InterfaceStability.Unstable, and other than CDP I'm not aware of it being used. The Apache Ranger source code does not use this interface, as far as I can tell, probably the source used by CDP has not yet been shared with the community. Also, prior to HADOOP-16916 the implementation had a few issues, so it is extremely unlikely that anyone took a dependency. You pointed out the interface change in this JIRA, but the underlying implementation was also changed in HADOOP-16916. Attempting to fix the breaking change would be quite ugly, resulting in two underlying code paths, two interfaces (SASTokenProvider and SASTokenProvider2), and two sets of tests. Since CDP needs the latest and I'm not aware of anyone else using this, I think the risk of breaking users is very, very low and we should not consider this a blocker for 3.3.1, but instead leave it as-is and resolve this JIRA. > HADOOP-16916 changed interface SASTokenProvider fields, breaking > compatibility between 3.3.0 and 3.3.1 > --- > > Key: HADOOP-17704 > URL: https://issues.apache.org/jira/browse/HADOOP-17704 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Wei-Chiu Chuang >Priority: Blocker > > I understand HADOOP-16730/HADOOP-16916 is specifically made for Ranger, but I > am not sure how Ranger consumes this feature. The interface SASTokenProvider > has a number of member fields that changed variable names in HADOOP-16916, > breaking the compatibility between 3.3.0 and 3.3.1. > As a matter of fact, the feature HADOOP-16730 itself was merged in 3.3.0 not > 3.3.1. I just corrected it today. > Raise this jira and mark it as a blocker for 3.3.1. But if this isn't a big > deal then we can downgrade, because, well, this feature was not officially in > the 3.3.0 release. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17422) ABFS: Set default ListMaxResults to max server limit
[
https://issues.apache.org/jira/browse/HADOOP-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt resolved HADOOP-17422.
--
Fix Version/s: 3.3.1
Release Note: ABFS: The default value for "fs.azure.list.max.results" was
changed from 500 to 5000.
Resolution: Fixed
commit a35fc387
> ABFS: Set default ListMaxResults to max server limit
>
>
> Key: HADOOP-17422
> URL: https://issues.apache.org/jira/browse/HADOOP-17422
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Sumangala Patki
>Assignee: Thomas Marqardt
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> {{Changing the default value of maximum size of }}{{results to be}}{{
> returned by ListStatus from 500 to 5000, since the maximum number of items
> supported by a listStatus server call is 5000.}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17422) ABFS: Set default ListMaxResults to max server limit
[
https://issues.apache.org/jira/browse/HADOOP-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt reassigned HADOOP-17422:
Assignee: Thomas Marqardt
> ABFS: Set default ListMaxResults to max server limit
>
>
> Key: HADOOP-17422
> URL: https://issues.apache.org/jira/browse/HADOOP-17422
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Sumangala Patki
>Assignee: Thomas Marqardt
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> {{Changing the default value of maximum size of }}{{results to be}}{{
> returned by ListStatus from 500 to 5000, since the maximum number of items
> supported by a listStatus server call is 5000.}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17397) ABFS: SAS Test updates for version and permission update
[ https://issues.apache.org/jira/browse/HADOOP-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt resolved HADOOP-17397. -- Resolution: Fixed Pushed to trunk (commit 717b8350) and branch-3.3 (commit a5695057). > ABFS: SAS Test updates for version and permission update > > > Key: HADOOP-17397 > URL: https://issues.apache.org/jira/browse/HADOOP-17397 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 1.5h > Remaining Estimate: 0h > > This Jira will track the below 2 updates to SAS test code: > # Upgrading the SAS version in Service SAS generator (test code) > # Updating the permission in Delegation SAS to "op" from "p" for ACL > operation as identities added as suoid/saoid added by tests are not owners of > test path (Again test code). > [Relevant public documentation: > https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview|https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17397) ABFS: SAS Test updates for version and permission update
[ https://issues.apache.org/jira/browse/HADOOP-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241253#comment-17241253 ] Thomas Marqardt commented on HADOOP-17397: -- PR 2504 fixes the previous commit. > ABFS: SAS Test updates for version and permission update > > > Key: HADOOP-17397 > URL: https://issues.apache.org/jira/browse/HADOOP-17397 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 1h > Remaining Estimate: 0h > > This Jira will track the below 2 updates to SAS test code: > # Upgrading the SAS version in Service SAS generator (test code) > # Updating the permission in Delegation SAS to "op" from "p" for ACL > operation as identities added as suoid/saoid added by tests are not owners of > test path (Again test code). > [Relevant public documentation: > https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview|https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17397) ABFS: SAS Test updates for version and permission update
[ https://issues.apache.org/jira/browse/HADOOP-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241058#comment-17241058 ] Thomas Marqardt commented on HADOOP-17397: -- [~ste...@apache.org] please note my previous comment. This also needs to be fixed in branch-3.3. > ABFS: SAS Test updates for version and permission update > > > Key: HADOOP-17397 > URL: https://issues.apache.org/jira/browse/HADOOP-17397 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 50m > Remaining Estimate: 0h > > This Jira will track the below 2 updates to SAS test code: > # Upgrading the SAS version in Service SAS generator (test code) > # Updating the permission in Delegation SAS to "op" from "p" for ACL > operation as identities added as suoid/saoid added by tests are not owners of > test path (Again test code). > [Relevant public documentation: > https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview|https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-17397) ABFS: SAS Test updates for version and permission update
[ https://issues.apache.org/jira/browse/HADOOP-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt reopened HADOOP-17397: -- See my previous comment on why this was reopened. > ABFS: SAS Test updates for version and permission update > > > Key: HADOOP-17397 > URL: https://issues.apache.org/jira/browse/HADOOP-17397 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 50m > Remaining Estimate: 0h > > This Jira will track the below 2 updates to SAS test code: > # Upgrading the SAS version in Service SAS generator (test code) > # Updating the permission in Delegation SAS to "op" from "p" for ACL > operation as identities added as suoid/saoid added by tests are not owners of > test path (Again test code). > [Relevant public documentation: > https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview|https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17397) ABFS: SAS Test updates for version and permission update
[ https://issues.apache.org/jira/browse/HADOOP-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241056#comment-17241056 ] Thomas Marqardt commented on HADOOP-17397: -- Unfortunately, this was not the correct fix. DelegationSASGenerator.getDelegationSAS should return sp=p for the set-permission and set-acl operations. The tests should be updated to do the following: # When saoid and suoid are not specified, skoid must have an RBAC role assignment which grants Microsoft.Storage/storageAccounts/blobServices/containers/blobs/modifyPermissions/action and sp=p in order to set permissions or set ACL. # When saoid or suiod is specified, same as 1) but furthermore the saoid or suoid must be an owner of the file or directory in order for the operation to succeed. # When saoid or suiod is specified, the ownership check is bypassed by also including 'o' (ownership) in the SAS permission (for example, sp=op). Note that 'o' grants the saoid or suoid the ability to change the file or directory owner to themself, and they can also change the owning group. Generally speaking, if a trusted authorizer would like to give a user the ability to change the permissions or ACL, then that user should be the file or directory owner. > ABFS: SAS Test updates for version and permission update > > > Key: HADOOP-17397 > URL: https://issues.apache.org/jira/browse/HADOOP-17397 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 50m > Remaining Estimate: 0h > > This Jira will track the below 2 updates to SAS test code: > # Upgrading the SAS version in Service SAS generator (test code) > # Updating the permission in Delegation SAS to "op" from "p" for ACL > operation as identities added as suoid/saoid added by tests are not owners of > test path (Again test code). > [Relevant public documentation: > https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview|https://docs.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#specify-a-signed-object-id-for-a-security-principal-preview] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17301) ABFS: read-ahead error reporting breaks buffer management
[ https://issues.apache.org/jira/browse/HADOOP-17301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17301: - Fix Version/s: 3.4.0 3.3.1 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to trunk in commit c4fff74 and backported to branch-3.3 in commit d5b4d04. All tests passing against my account in eastus2euap: namespace.enabled=true auth.type=SharedKey $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 24 Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 namespace.enabled=true auth.type=OAuth $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 66 Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 > ABFS: read-ahead error reporting breaks buffer management > - > > Key: HADOOP-17301 > URL: https://issues.apache.org/jira/browse/HADOOP-17301 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Critical > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > When reads done by readahead buffers failed, the exceptions where dropped and > the failure was not getting reported to the calling app. > Jira HADOOP-16852: Report read-ahead error back > tried to handle the scenario by reporting the error back to calling app. But > the commit has introduced a bug which can lead to ReadBuffer being injected > into read completed queue twice. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17279) ABFS: Test testNegativeScenariosForCreateOverwriteDisabled fails for non-HNS account
[ https://issues.apache.org/jira/browse/HADOOP-17279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17279: - Fix Version/s: 3.3.1 Backported to branch-3.3 in commit da5db6a. All tests passing against my account in eastus2euap: namespace.enabled=true auth.type=SharedKey $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 24 Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 namespace.enabled=true auth.type=OAuth $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 66 Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 > ABFS: Test testNegativeScenariosForCreateOverwriteDisabled fails for non-HNS > account > > > Key: HADOOP-17279 > URL: https://issues.apache.org/jira/browse/HADOOP-17279 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Test testNegativeScenariosForCreateOverwriteDisabled fails when run against a > non-HNS account. The test creates a mock AbfsClient to mimic negative > scenarios. > Mock is triggered for valid values that come in for permission and umask > while creating a file. Permission and umask get defaulted to null values with > driver when creating a file for a nonHNS account. The mock trigger was not > enabled for these null parameters. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17215) ABFS: Support for conditional overwrite
[ https://issues.apache.org/jira/browse/HADOOP-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17215: - Fix Version/s: 3.3.1 Backported to branch-3.3 in commit d166420. All tests passing against my account in eastus2euap: namespace.enabled=true auth.type=SharedKey $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 24 Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 namespace.enabled=true auth.type=OAuth $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 66 Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 > ABFS: Support for conditional overwrite > --- > > Key: HADOOP-17215 > URL: https://issues.apache.org/jira/browse/HADOOP-17215 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: abfsactive > Fix For: 3.3.1, 3.4.0 > > > Filesystem Create APIs that do not accept an argument for overwrite flag end > up defaulting it to true. > We are observing that request count of creates with overwrite=true is more > and primarily because of the default setting of the flag is true of the > called Create API. When a create with overwrite ends up timing out, we have > observed that it could lead to race conditions between the first create and > retried one running almost parallel. > To avoid this scenario for create with overwrite=true request, ABFS driver > will always attempt to create without overwrite. If the create fails due to > fileAlreadyPresent, it will resend the request with overwrite=true. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17166) ABFS: configure output stream thread pool
[ https://issues.apache.org/jira/browse/HADOOP-17166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17166: - Fix Version/s: 3.3.1 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to trunk in commit 8511926 and backported to branch-3.3 in commit f208da2. All tests passing against my account in eastus2euap: namespace.enabled=true auth.type=SharedKey $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 24 Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 namespace.enabled=true auth.type=OAuth $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 66 Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 > ABFS: configure output stream thread pool > - > > Key: HADOOP-17166 > URL: https://issues.apache.org/jira/browse/HADOOP-17166 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16915) ABFS: Test failure ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance
[ https://issues.apache.org/jira/browse/HADOOP-16915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt resolved HADOOP-16915. -- Fix Version/s: 3.4.0 3.3.1 Resolution: Fixed Pushed to trunk in commit 64f36b9 and backported to branch-3.3 in commit cc73503. All tests passing against my account in eastus2euap: namespace.enabled=true auth.type=SharedKey $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 24 Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 namespace.enabled=true auth.type=OAuth $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 66 Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 > ABFS: Test failure > ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance > --- > > Key: HADOOP-16915 > URL: https://issues.apache.org/jira/browse/HADOOP-16915 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Major > Labels: abfsactive > Fix For: 3.3.1, 3.4.0 > > > Ref: https://issues.apache.org/jira/browse/HADOOP-16890 > The following test fails randomly. This test compares the perf between Non > HNS account against WASB. > ITestAzureBlobFileSystemRandomRead.testRandomReadPerformance -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16966) ABFS: Upgrade Store REST API Version to 2019-12-12
[ https://issues.apache.org/jira/browse/HADOOP-16966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-16966: - Fix Version/s: 3.3.1 Backported to branch-3.3 in commit 4072323. All tests passing against my account in eastus2euap: namespace.enabled=true auth.type=SharedKey $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 24 Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 namespace.enabled=true auth.type=OAuth $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 66 Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 > ABFS: Upgrade Store REST API Version to 2019-12-12 > -- > > Key: HADOOP-16966 > URL: https://issues.apache.org/jira/browse/HADOOP-16966 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Ishani >Assignee: Sneha Vijayarajan >Priority: Major > Labels: abfsactive > Fix For: 3.3.1 > > > Store REST API version on the backend clusters has been upgraded to > 2019-12-12. This Jira will align the Driver requests to reflect this latest > API version. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17149) ABFS: Test failure: testFailedRequestWhenCredentialsNotCorrect fails when run with SharedKey
[ https://issues.apache.org/jira/browse/HADOOP-17149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17149: - Fix Version/s: 3.3.1 Backported to branch-3.3 in commit e481d01. All tests passing against my account in eastus2euap: namespace.enabled=true auth.type=SharedKey $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 24 Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 namespace.enabled=true auth.type=OAuth $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 66 Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 > ABFS: Test failure: testFailedRequestWhenCredentialsNotCorrect fails when run > with SharedKey > > > Key: HADOOP-17149 > URL: https://issues.apache.org/jira/browse/HADOOP-17149 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.1 >Reporter: Sneha Vijayarajan >Assignee: Bilahari T H >Priority: Minor > Labels: abfsactive > Fix For: 3.3.1, 3.4.0 > > > When authentication is set to SharedKey, below test fails. > > [ERROR] > ITestGetNameSpaceEnabled.testFailedRequestWhenCredentialsNotCorrect:161 > Expecting > org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException > with text "Server failed to authenticate the request. Make sure the value of > Authorization header is formed correctly including the signature.", 403 but > got : "void" > > This test fails when the newly introduced config > "fs.azure.account.hns.enabled" is set. This config will avoid network call to > check if namespace is enabled, whereas the test expects thsi call to be made. > > The assert in test to 403 needs check too. Should ideally be 401. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17163) ABFS: Add debug log for rename failures
[ https://issues.apache.org/jira/browse/HADOOP-17163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17163: - Fix Version/s: 3.3.1 Backported to branch-3.3 in commit f73c90f. All tests passing against my account in eastus2euap: +namespace.enabled=true+ +auth.type=SharedKey+ $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 24 Tests run: 208, Failures: 0, Errors: 0, Skipped: 24 +namespace.enabled=true+ +auth.type=OAuth+ $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 88, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 66 Tests run: 208, Failures: 0, Errors: 0, Skipped: 141 > ABFS: Add debug log for rename failures > --- > > Key: HADOOP-17163 > URL: https://issues.apache.org/jira/browse/HADOOP-17163 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Major > Fix For: 3.3.1, 3.4.0 > > > The JIRA [HADOOP-16281|https://issues.apache.org/jira/browse/HADOOP-16281] > has not yet been concluded. Untill then the logline could help debugging. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17137) ABFS: Tests ITestAbfsNetworkStatistics need to be config setting agnostic
[
https://issues.apache.org/jira/browse/HADOOP-17137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17137:
-
Fix Version/s: 3.3.1
Backported to branch-3.3 in commit fbf151ef. All tests passing against my
account in eastus2euap:
+namespace.enabled=true+
+auth.type=SharedKey+
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 88, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 24
Tests run: 208, Failures: 0, Errors: 0, Skipped: 24
+namespace.enabled=true+
+auth.type=OAuth+
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 88, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 66
Tests run: 208, Failures: 0, Errors: 0, Skipped: 141
> ABFS: Tests ITestAbfsNetworkStatistics need to be config setting agnostic
> -
>
> Key: HADOOP-17137
> URL: https://issues.apache.org/jira/browse/HADOOP-17137
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, test
>Affects Versions: 3.3.0
>Reporter: Sneha Vijayarajan
>Assignee: Bilahari T H
>Priority: Minor
> Labels: abfsactive
> Fix For: 3.3.1, 3.4.0
>
>
> Tess in ITestAbfsNetworkStatistics have asserts to a static number of
> network calls made from the start of fileystem instance creation. But this
> number of calls are dependent on the certain configs settings which allow
> creation of container or account is HNS enabled to avoid GetAcl call.
>
> The tests need to be modified to ensure that count asserts are made for the
> requests made by the tests alone.
>
> {code:java}
> [INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[INFO]
> Running org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR] Tests
> run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 4.148 s <<<
> FAILURE! - in org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics[ERROR]
> testAbfsHttpResponseStatistics(org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics)
> Time elapsed: 4.148 s <<< FAILURE!java.lang.AssertionError: Mismatch in
> get_responses expected:<8> but was:<7> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:645) at
> org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.assertAbfsStatistics(AbstractAbfsIntegrationTest.java:445)
> at
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics.testAbfsHttpResponseStatistics(ITestAbfsNetworkStatistics.java:207)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> [ERROR]
> testAbfsHttpSendStatistics(org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics)
> Time elapsed: 2.987 s <<< FAILURE!java.lang.AssertionError: Mismatch in
> connections_made expected:<6> but was:<5> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:645) at
> org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.assertAbfsStatistics(AbstractAbfsIntegrationTest.java:445)
> at
> org.apache.hadoop.fs.azurebfs.ITestAbfsNetworkStatistics.testAbfsHttpSendStatistics(ITestAbfsNetworkStatistics.java:91)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at
[jira] [Updated] (HADOOP-17279) ABFS: Test testNegativeScenariosForCreateOverwriteDisabled fails for non-HNS account
[ https://issues.apache.org/jira/browse/HADOOP-17279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17279: - Fix Version/s: 3.4.0 Resolution: Fixed Status: Resolved (was: Patch Available) lgtm commit c3a90dd9186b664594238131596ae2de17bf70fc Author: Sneha Vijayarajan Date: Tue Sep 22 20:58:12 2020 + HADOOP-17279: ABFS: testNegativeScenariosForCreateOverwriteDisabled fails for non-HNS account. Contributed by Sneha Vijayarajan Testing: namespace.enabled=false auth.type=SharedKey $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 246 Tests run: 207, Failures: 0, Errors: 0, Skipped: 24 namespace.enabled=true auth.type=SharedKey $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 33 Tests run: 207, Failures: 0, Errors: 0, Skipped: 24 namespace.enabled=true auth.type=OAuth $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 87, Failures: 0, Errors: 0, Skipped: 0 Tests run: 457, Failures: 0, Errors: 0, Skipped: 74 Tests run: 207, Failures: 0, Errors: 0, Skipped: 140 > ABFS: Test testNegativeScenariosForCreateOverwriteDisabled fails for non-HNS > account > > > Key: HADOOP-17279 > URL: https://issues.apache.org/jira/browse/HADOOP-17279 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Test testNegativeScenariosForCreateOverwriteDisabled fails when run against a > non-HNS account. The test creates a mock AbfsClient to mimic negative > scenarios. > Mock is triggered for valid values that come in for permission and umask > while creating a file. Permission and umask get defaulted to null values with > driver when creating a file for a nonHNS account. The mock trigger was not > enabled for these null parameters. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17215) ABFS: Support for conditional overwrite
[ https://issues.apache.org/jira/browse/HADOOP-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17215: - Fix Version/s: 3.4.0 Release Note: ABFS: Support for conditional overwrite. Resolution: Fixed Status: Resolved (was: Patch Available) commit e31a636e922a8fdbe0aa7cca53f6de7175e97254 Author: Sneha Vijayarajan Date: Wed Aug 26 00:31:35 2020 +0530 HADOOP-17215: Support for conditional overwrite. Contributed by Sneha Vijayarajan DETAILS: This change adds config key "fs.azure.enable.conditional.create.overwrite" with a default of true. When enabled, if create(path, overwrite: true) is invoked and the file exists, the ABFS driver will first obtain its etag and then attempt to overwrite the file on the condition that the etag matches. The purpose of this is to mitigate the non-idempotency of this method. Specifically, in the event of a network error or similar, the client will retry and this can result in the file being created more than once which may result in data loss. In essence this is like a poor man's file handle, and will be addressed more thoroughly in the future when support for lease is added to ABFS. > ABFS: Support for conditional overwrite > --- > > Key: HADOOP-17215 > URL: https://issues.apache.org/jira/browse/HADOOP-17215 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: abfsactive > Fix For: 3.4.0 > > > Filesystem Create APIs that do not accept an argument for overwrite flag end > up defaulting it to true. > We are observing that request count of creates with overwrite=true is more > and primarily because of the default setting of the flag is true of the > called Create API. When a create with overwrite ends up timing out, we have > observed that it could lead to race conditions between the first create and > retried one running almost parallel. > To avoid this scenario for create with overwrite=true request, ABFS driver > will always attempt to create without overwrite. If the create fails due to > fileAlreadyPresent, it will resend the request with overwrite=true. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17215) ABFS: Support for conditional overwrite
[ https://issues.apache.org/jira/browse/HADOOP-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17215: - Summary: ABFS: Support for conditional overwrite (was: ABFS: Excessive Create Overwrites leads to race conditions) > ABFS: Support for conditional overwrite > --- > > Key: HADOOP-17215 > URL: https://issues.apache.org/jira/browse/HADOOP-17215 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: abfsactive > > Filesystem Create APIs that do not accept an argument for overwrite flag end > up defaulting it to true. > We are observing that request count of creates with overwrite=true is more > and primarily because of the default setting of the flag is true of the > called Create API. When a create with overwrite ends up timing out, we have > observed that it could lead to race conditions between the first create and > retried one running almost parallel. > To avoid this scenario for create with overwrite=true request, ABFS driver > will always attempt to create without overwrite. If the create fails due to > fileAlreadyPresent, it will resend the request with overwrite=true. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17203) Test failures in ITestAzureBlobFileSystemCheckAccess in ABFS
[ https://issues.apache.org/jira/browse/HADOOP-17203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17203: - Fix Version/s: 3.4.0 Resolution: Fixed Status: Resolved (was: Patch Available) commit 0dc54d0247735ad744aad9a08c885c91b2be5c50 Author: ThomasMarquardt Date: Fri Sep 18 17:52:11 2020 -0700 HADOOP-17203: Revert HADOOP-17183. ABFS: Enabling checkaccess on ABFS This reverts commit a2610e21ed5289323d8a6f6359477a8ceb2db2eb. > Test failures in ITestAzureBlobFileSystemCheckAccess in ABFS > > > Key: HADOOP-17203 > URL: https://issues.apache.org/jira/browse/HADOOP-17203 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Mehakmeet Singh >Assignee: Thomas Marqardt >Priority: Major > Labels: abfsactive, pull-request-available > Fix For: 3.4.0 > > Time Spent: 20m > Remaining Estimate: 0h > > ITestAzureBlobFileSystemCheckAccess is giving test failures while running > both in parallel as well as in stand-alone(in IDE). > Tested by: mvn -T 1C -Dparallel-tests=abfs clean verify > Region: East US -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17183) ABFS: Enable checkaccess API
[ https://issues.apache.org/jira/browse/HADOOP-17183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198611#comment-17198611 ] Thomas Marqardt commented on HADOOP-17183: -- commit 0dc54d0247735ad744aad9a08c885c91b2be5c50 Author: ThomasMarquardt Date: Fri Sep 18 17:52:11 2020 -0700 HADOOP-17203: Revert HADOOP-17183. ABFS: Enabling checkaccess on ABFS This reverts commit a2610e21ed5289323d8a6f6359477a8ceb2db2eb. > ABFS: Enable checkaccess API > > > Key: HADOOP-17183 > URL: https://issues.apache.org/jira/browse/HADOOP-17183 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-17183) ABFS: Enable checkaccess API
[ https://issues.apache.org/jira/browse/HADOOP-17183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt reopened HADOOP-17183: -- Tests are failing so I am going to revert this to fix them. Sorry, I did not see a quick fix, so decided to revert for now. > ABFS: Enable checkaccess API > > > Key: HADOOP-17183 > URL: https://issues.apache.org/jira/browse/HADOOP-17183 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17203) Test failures in ITestAzureBlobFileSystemCheckAccess in ABFS
[
https://issues.apache.org/jira/browse/HADOOP-17203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198575#comment-17198575
]
Thomas Marqardt commented on HADOOP-17203:
--
Test Results for my account in East US 2 EUAP:
+namespace.enabled=true+
+auth.type=SharedKey+
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
{quote}Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 454, Failures: 0, Errors: 0, Skipped: 42
Tests run: 207, Failures: 0, Errors: 0, Skipped: 24
{quote}
+namespace.enabled=true+
+auth.type=OAuth+
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
{quote}Tests run: 63, Failures: 0, Errors: 0, Skipped: 0
Tests run: 432, Failures: 0, Errors: 1, Skipped: 74
Tests run: 206, Failures: 0, Errors: 0, Skipped: 140
{quote}
> Test failures in ITestAzureBlobFileSystemCheckAccess in ABFS
>
>
> Key: HADOOP-17203
> URL: https://issues.apache.org/jira/browse/HADOOP-17203
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Mehakmeet Singh
>Assignee: Thomas Marqardt
>Priority: Major
> Labels: abfsactive
>
> ITestAzureBlobFileSystemCheckAccess is giving test failures while running
> both in parallel as well as in stand-alone(in IDE).
> Tested by: mvn -T 1C -Dparallel-tests=abfs clean verify
> Region: East US
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17203) Test failures in ITestAzureBlobFileSystemCheckAccess in ABFS
[ https://issues.apache.org/jira/browse/HADOOP-17203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17203: - Status: Patch Available (was: Open) [https://github.com/apache/hadoop/pull/2316.patch] > Test failures in ITestAzureBlobFileSystemCheckAccess in ABFS > > > Key: HADOOP-17203 > URL: https://issues.apache.org/jira/browse/HADOOP-17203 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Mehakmeet Singh >Assignee: Thomas Marqardt >Priority: Major > Labels: abfsactive > > ITestAzureBlobFileSystemCheckAccess is giving test failures while running > both in parallel as well as in stand-alone(in IDE). > Tested by: mvn -T 1C -Dparallel-tests=abfs clean verify > Region: East US -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17203) Test failures in ITestAzureBlobFileSystemCheckAccess in ABFS
[ https://issues.apache.org/jira/browse/HADOOP-17203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt reassigned HADOOP-17203: Assignee: Thomas Marqardt (was: Bilahari T H) > Test failures in ITestAzureBlobFileSystemCheckAccess in ABFS > > > Key: HADOOP-17203 > URL: https://issues.apache.org/jira/browse/HADOOP-17203 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Mehakmeet Singh >Assignee: Thomas Marqardt >Priority: Major > Labels: abfsactive > > ITestAzureBlobFileSystemCheckAccess is giving test failures while running > both in parallel as well as in stand-alone(in IDE). > Tested by: mvn -T 1C -Dparallel-tests=abfs clean verify > Region: East US -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16948) ABFS: Support single writer dirs
[ https://issues.apache.org/jira/browse/HADOOP-16948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197243#comment-17197243 ] Thomas Marqardt commented on HADOOP-16948: -- This is not getting the attention it deserves. I'll see if I can find some free time to help move this along. > ABFS: Support single writer dirs > > > Key: HADOOP-16948 > URL: https://issues.apache.org/jira/browse/HADOOP-16948 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Minor > Labels: abfsactive > > This would allow some directories to be configured as single writer > directories. The ABFS driver would obtain a lease when creating or opening a > file for writing and would automatically renew the lease and release the > lease when closing the file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17215) ABFS: Excessive Create Overwrites leads to race conditions
[ https://issues.apache.org/jira/browse/HADOOP-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197204#comment-17197204 ] Thomas Marqardt commented on HADOOP-17215: -- Seems we should be doing something like https://issues.apache.org/jira/browse/HADOOP-16948 instead. Thoughts? I'd prefer to use leases so ABFS can uphold the single writer semantics of HDFS. This was something we planned to do early on, but was postponed. > ABFS: Excessive Create Overwrites leads to race conditions > -- > > Key: HADOOP-17215 > URL: https://issues.apache.org/jira/browse/HADOOP-17215 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Labels: abfsactive > > Filesystem Create APIs that do not accept an argument for overwrite flag end > up defaulting it to true. > We are observing that request count of creates with overwrite=true is more > and primarily because of the default setting of the flag is true of the > called Create API. When a create with overwrite ends up timing out, we have > observed that it could lead to race conditions between the first create and > retried one running almost parallel. > To avoid this scenario for create with overwrite=true request, ABFS driver > will always attempt to create without overwrite. If the create fails due to > fileAlreadyPresent, it will resend the request with overwrite=true. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17132) ABFS: Fix For Idempotency code
[
https://issues.apache.org/jira/browse/HADOOP-17132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17132:
-
Fix Version/s: 3.3.1
1ac5258 branch-3.3@\{2020-07-25 13:13:05 +}: cherry-pick: Hadoop 17132.
ABFS: Fix Rename and Delete Idempotency check trigger
> ABFS: Fix For Idempotency code
> --
>
> Key: HADOOP-17132
> URL: https://issues.apache.org/jira/browse/HADOOP-17132
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.4.0
>Reporter: Sneha Vijayarajan
>Assignee: Sneha Vijayarajan
>Priority: Major
> Labels: abfsactive
> Fix For: 3.3.1, 3.4.0
>
>
> Trigger to handle the idempotency code introduced in
> https://issues.apache.org/jira/browse/HADOOP-17015 is incomplete.
> This PR is to fix the issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17058) Support for Appendblob in abfs driver
[
https://issues.apache.org/jira/browse/HADOOP-17058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17058:
-
Fix Version/s: 3.3.1
f24e2ec branch-3.3@\{2020-07-25 13:12:32 +}: cherry-pick: HADOOP-17058.
ABFS: Support for AppendBlob in Hadoop ABFS Driver
> Support for Appendblob in abfs driver
> -
>
> Key: HADOOP-17058
> URL: https://issues.apache.org/jira/browse/HADOOP-17058
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Ishani
>Assignee: Ishani
>Priority: Major
> Fix For: 3.3.1
>
>
> add changes to support appendblob in the hadoop-azure abfs driver.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16961) ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics)
[
https://issues.apache.org/jira/browse/HADOOP-16961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-16961:
-
Fix Version/s: 3.3.1
7c9b459 branch-3.3@\{2020-07-25 13:12:09 +}: cherry-pick: HADOOP-16961.
ABFS: Adding metrics to AbfsInputStream (#2076)
> ABFS: Adding metrics to AbfsInputStream (AbfsInputStreamStatistics)
> ---
>
> Key: HADOOP-16961
> URL: https://issues.apache.org/jira/browse/HADOOP-16961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Gabor Bota
>Assignee: Mehakmeet Singh
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
>
> Adding metrics to AbfsInputStream (AbfsInputStreamStatistics) can improve the
> testing and diagnostics of the connector.
> Also adding some logging.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17065) Adding Network Counters in ABFS
[
https://issues.apache.org/jira/browse/HADOOP-17065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17065:
-
Fix Version/s: 3.3.1
0d0ed17 branch-3.3@\{2020-07-25 13:11:10 +}: cherry-pick: HADOOP-17065. Add
Network Counters to ABFS (#2056)
> Adding Network Counters in ABFS
> ---
>
> Key: HADOOP-17065
> URL: https://issues.apache.org/jira/browse/HADOOP-17065
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 3.3.0
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Fix For: 3.3.1
>
>
> Network Counters to be added in ABFS:
> |CONNECTIONS_MADE|Number of times connection was made with Azure Data Lake|
> |SEND_REQUESTS|Number of send requests|
> |GET_RESPONSE|Number of response gotten|
> |BYTES_SEND|Number of bytes send|
> |BYTES_RECEIVED|Number of bytes received|
> |READ_THROTTLE|Number of times throttled while read operation|
> |WRITE_THROTTLE|Number of times throttled while write operation|
> propose:
> * Adding these counters as part of AbfsStatistic already made in
> HADOOP-17016.
> * Increment of counters across Abfs Network services.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17057) ABFS driver enhancement - Allow customizable translation from AAD SPNs and security groups to Linux user and group
[
https://issues.apache.org/jira/browse/HADOOP-17057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17057:
-
Fix Version/s: 3.3.1
8b7e774 branch-3.3@\{2020-07-25 13:10:39 +}: cherry-pick: HDFS-15168: ABFS
enhancement to translate AAD to Linux identities. (#1978)
> ABFS driver enhancement - Allow customizable translation from AAD SPNs and
> security groups to Linux user and group
> --
>
> Key: HADOOP-17057
> URL: https://issues.apache.org/jira/browse/HADOOP-17057
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Reporter: Karthik Amarnath
>Assignee: Karthik Amarnath
>Priority: Major
> Fix For: 3.3.1
>
>
> ABFS driver does not support the translation of AAD Service principal (SPI)
> to Linux identities causing metadata operation failure. Hadoop MapReduce
> client
> [[JobSubmissionFiles|https://github.com/apache/hadoop/blob/d842dfffa53c8b565f3d65af44ccd7e1cc706733/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmissionFiles.java#L138]]
> expects the file owner permission to be the Linux identity, but the
> underlying ABFS driver returns the AAD Object identity. Hence need ABFS
> driver enhancement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17053) ABFS: FS initialize fails for incompatible account-agnostic Token Provider setting
[
https://issues.apache.org/jira/browse/HADOOP-17053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17053:
-
903935d branch-3.3@\{2020-07-25 13:10:30 +}: cherry-pick: HADOOP-17053.
ABFS: Fix Account-specific OAuth config setting parsing
> ABFS: FS initialize fails for incompatible account-agnostic Token Provider
> setting
> ---
>
> Key: HADOOP-17053
> URL: https://issues.apache.org/jira/browse/HADOOP-17053
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.1
>Reporter: Sneha Vijayarajan
>Assignee: Sneha Vijayarajan
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
>
> When AuthType and Auth token provider configs are set for both generic and
> account specific config, as below:
> // account agnostic
> fs.azure.account.auth.type=CUSTOM
> fs.azure.account.oauth.provider.type=ClassExtendingCustomTokenProviderAdapter
> // account specific
> fs.azure.account.auth.type.account_name=OAuth
> fs.azure.account.oauth.provider.type.account_name=ClassExtendingAccessTokenProvider
> For account_name, OAuth with provider as ClassExtendingAccessTokenProvider
> is expected to be in effect.
> When the token provider class is being read from the config, account agnostic
> config setting is read first in the assumption that it can serve as default
> if account-specific config setting is absent. But this logic leads to failure
> when AuthType set for account specific and otherwise are different as the
> Interface implementing the token provider is different for various Auth
> Types. This leads to a Runtime exception when trying to create the oAuth
> access token provider.
> This Jira is to track the fix for it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17053) ABFS: FS initialize fails for incompatible account-agnostic Token Provider setting
[
https://issues.apache.org/jira/browse/HADOOP-17053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17053:
-
Fix Version/s: 3.3.1
903935d branch-3.3@\{2020-07-25 13:10:30 +}: cherry-pick: HADOOP-17053.
ABFS: Fix Account-specific OAuth config setting parsing
> ABFS: FS initialize fails for incompatible account-agnostic Token Provider
> setting
> ---
>
> Key: HADOOP-17053
> URL: https://issues.apache.org/jira/browse/HADOOP-17053
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.1
>Reporter: Sneha Vijayarajan
>Assignee: Sneha Vijayarajan
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
>
> When AuthType and Auth token provider configs are set for both generic and
> account specific config, as below:
> // account agnostic
> fs.azure.account.auth.type=CUSTOM
> fs.azure.account.oauth.provider.type=ClassExtendingCustomTokenProviderAdapter
> // account specific
> fs.azure.account.auth.type.account_name=OAuth
> fs.azure.account.oauth.provider.type.account_name=ClassExtendingAccessTokenProvider
> For account_name, OAuth with provider as ClassExtendingAccessTokenProvider
> is expected to be in effect.
> When the token provider class is being read from the config, account agnostic
> config setting is read first in the assumption that it can serve as default
> if account-specific config setting is absent. But this logic leads to failure
> when AuthType set for account specific and otherwise are different as the
> Interface implementing the token provider is different for various Auth
> Types. This leads to a Runtime exception when trying to create the oAuth
> access token provider.
> This Jira is to track the fix for it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16852) ABFS: Send error back to client for Read Ahead request failure
[
https://issues.apache.org/jira/browse/HADOOP-16852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-16852:
-
Fix Version/s: 3.3.1
869a68b branch-3.3@\{2020-07-25 13:10:19 +}: cherry-pick: HADOOP-16852:
Report read-ahead error bac
> ABFS: Send error back to client for Read Ahead request failure
> --
>
> Key: HADOOP-16852
> URL: https://issues.apache.org/jira/browse/HADOOP-16852
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.3.1
>Reporter: Sneha Vijayarajan
>Assignee: Sneha Vijayarajan
>Priority: Major
> Fix For: 3.3.1
>
>
> Issue seen by a customer:
> The failed requests we were seeing in the AbfsClient logging actually never
> made it out over the wire. We have found that there’s an issue with ADLS
> passthrough and the 8 read ahead threads that ADLSv2 spawns in
> ReadBufferManager.java. We depend on thread local storage in order to get the
> right JWT token and those threads do not have the right information in their
> thread local storage. Thus, when they pick up a task from the read ahead
> queue they fail by throwing an AzureCredentialNotFoundException exception in
> AbfsRestOperation.executeHttpOperation() where it calls
> client.getAccessToken(). This exception is silently swallowed by the read
> ahead threads in ReadBufferWorker.run(). As a result, every read ahead
> attempt results in a failed executeHttpOperation(), but still calls
> AbfsClientThrottlingIntercept.updateMetrics() and contributes to throttling
> (despite not making it out over the wire). After the read aheads fail, the
> main task thread performs the read with the right thread local storage
> information and succeeds, but first sleeps for up to 10 seconds due to the
> throttling.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17054) ABFS: Fix idempotency test failures when SharedKey is set as AuthType
[
https://issues.apache.org/jira/browse/HADOOP-17054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17054:
-
Fix Version/s: 3.3.1
27b20f9 branch-3.3@\{2020-07-25 13:09:26 +}: cherry-pick: HADOOP-17054.
ABFS: Fix test AbfsClient authentication instance
> ABFS: Fix idempotency test failures when SharedKey is set as AuthType
> -
>
> Key: HADOOP-17054
> URL: https://issues.apache.org/jira/browse/HADOOP-17054
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.1
>Reporter: Sneha Vijayarajan
>Assignee: Sneha Vijayarajan
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
>
> Idempotency related tests added as part of
> https://issues.apache.org/jira/browse/HADOOP-17015
> create a test AbfsClient instance. This mock instance wrongly accepts valid
> sharedKey and oauth token provider instance. This leads to test failures with
> exceptions:
> [ERROR]
> testRenameRetryFailureAsHTTP404(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRename)
> Time elapsed: 9.133 s <<< ERROR!
> Invalid auth type: SharedKey is being used, expecting OAuth
> at
> org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getTokenProvider(AbfsConfiguration.java:643)
> This Jira is to fix these tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17015) ABFS: Make PUT and POST operations idempotent
[
https://issues.apache.org/jira/browse/HADOOP-17015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marqardt updated HADOOP-17015:
-
Fix Version/s: 3.3.1
eed06b4 branch-3.3@\{2020-07-25 13:08:01 +}: cherry-pick: Hadoop-17015.
ABFS: Handling Rename and Delete idempotency
> ABFS: Make PUT and POST operations idempotent
> -
>
> Key: HADOOP-17015
> URL: https://issues.apache.org/jira/browse/HADOOP-17015
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 3.2.1
>Reporter: Sneha Vijayarajan
>Assignee: Sneha Vijayarajan
>Priority: Major
> Fix For: 3.3.1, 3.4.0
>
>
> Currently when a PUT or POST operation timeouts and the server has already
> successfully executed the operation, there is no check in driver to see if
> the operation did succeed or not and just retries the same operation again.
> This can cause driver to through invalid user errors.
>
> Sample scenario:
> # Rename request times out. Though server has successfully executed the
> operation.
> # Driver retries rename and get source not found error.
> In the scenario, driver needs to check if rename is being retried and success
> if source if not found, but destination is present.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17089) WASB: Update azure-storage-java SDK
[ https://issues.apache.org/jira/browse/HADOOP-17089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145971#comment-17145971 ] Thomas Marqardt commented on HADOOP-17089: -- branch-2.10: commit 0d4f9c778967ce0f83663c63389987335d47c3ea Author: Thomas Marquardt Date: Wed Jun 24 18:37:25 2020 + > WASB: Update azure-storage-java SDK > --- > > Key: HADOOP-17089 > URL: https://issues.apache.org/jira/browse/HADOOP-17089 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.7.0, 2.8.0, 2.9.0, 3.0.0, 3.1.0, 3.2.0 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Critical > Fix For: 2.10.1, 3.3.1 > > > WASB depends on the Azure Storage Java SDK. There is a concurrency bug in > the Azure Storage Java SDK that can cause the results of a list blobs > operation to appear empty. This causes the Filesystem listStatus and similar > APIs to return empty results. This has been seen in Spark work loads when > jobs use more than one executor core. > See [https://github.com/Azure/azure-storage-java/pull/546] for details on the > bug in the Azure Storage SDK. > This issue can cause data loss. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17089) WASB: Update azure-storage-java SDK
[ https://issues.apache.org/jira/browse/HADOOP-17089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17089: - Fix Version/s: 2.10.1 > WASB: Update azure-storage-java SDK > --- > > Key: HADOOP-17089 > URL: https://issues.apache.org/jira/browse/HADOOP-17089 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.7.0, 2.8.0, 2.9.0, 3.0.0, 3.1.0, 3.2.0 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Critical > Fix For: 2.10.1, 3.3.1 > > > WASB depends on the Azure Storage Java SDK. There is a concurrency bug in > the Azure Storage Java SDK that can cause the results of a list blobs > operation to appear empty. This causes the Filesystem listStatus and similar > APIs to return empty results. This has been seen in Spark work loads when > jobs use more than one executor core. > See [https://github.com/Azure/azure-storage-java/pull/546] for details on the > bug in the Azure Storage SDK. > This issue can cause data loss. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17089) WASB: Update azure-storage-java SDK
[ https://issues.apache.org/jira/browse/HADOOP-17089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17089: - Description: WASB depends on the Azure Storage Java SDK. There is a concurrency bug in the Azure Storage Java SDK that can cause the results of a list blobs operation to appear empty. This causes the Filesystem listStatus and similar APIs to return empty results. This has been seen in Spark work loads when jobs use more than one executor core. See [https://github.com/Azure/azure-storage-java/pull/546] for details on the bug in the Azure Storage SDK. This issue can cause data loss. was: WASB depends on the Azure Storage Java SDK. There is a concurrency bug in the Azure Storage Java SDK that can cause the results of a list blobs operation to appear empty. This causes the Filesystem listStatus and similar APIs to return empty results. This has been seen in Spark work loads when jobs use more than one executor core. See [https://github.com/Azure/azure-storage-java/pull/546] for details on the bug in the Azure Storage SDK. Priority: Critical (was: Major) This issue can cause data loss. > WASB: Update azure-storage-java SDK > --- > > Key: HADOOP-17089 > URL: https://issues.apache.org/jira/browse/HADOOP-17089 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.7.0, 2.8.0, 2.9.0, 3.0.0, 3.1.0, 3.2.0 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Critical > Fix For: 3.3.1 > > > WASB depends on the Azure Storage Java SDK. There is a concurrency bug in > the Azure Storage Java SDK that can cause the results of a list blobs > operation to appear empty. This causes the Filesystem listStatus and similar > APIs to return empty results. This has been seen in Spark work loads when > jobs use more than one executor core. > See [https://github.com/Azure/azure-storage-java/pull/546] for details on the > bug in the Azure Storage SDK. > This issue can cause data loss. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17089) WASB: Update azure-storage-java SDK
[ https://issues.apache.org/jira/browse/HADOOP-17089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt resolved HADOOP-17089. -- Fix Version/s: 3.3.1 Release Note: Azure WASB bug fix that can cause list results to appear empty. Resolution: Fixed trunk: commit 4b5b54c73f2fd9146237087a59453e2b5d70f9ed Author: Thomas Marquardt Date: Wed Jun 24 18:37:25 2020 + branch-3.3 commit ee192c48265fe7dcf23bc33f6a6698bb41477ca9 Author: Thomas Marquardt Date: Wed Jun 24 18:37:25 2020 + > WASB: Update azure-storage-java SDK > --- > > Key: HADOOP-17089 > URL: https://issues.apache.org/jira/browse/HADOOP-17089 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.7.0, 2.8.0, 2.9.0, 3.0.0, 3.1.0, 3.2.0 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Major > Fix For: 3.3.1 > > > WASB depends on the Azure Storage Java SDK. There is a concurrency bug in > the Azure Storage Java SDK that can cause the results of a list blobs > operation to appear empty. This causes the Filesystem listStatus and similar > APIs to return empty results. This has been seen in Spark work loads when > jobs use more than one executor core. > See [https://github.com/Azure/azure-storage-java/pull/546] for details on the > bug in the Azure Storage SDK. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17089) WASB: Update azure-storage-java SDK
[ https://issues.apache.org/jira/browse/HADOOP-17089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144571#comment-17144571 ] Thomas Marqardt commented on HADOOP-17089: -- Updated with test at [https://github.com/apache/hadoop/pull/2099.patch|https://github.com/apache/hadoop/pull/2099]. > WASB: Update azure-storage-java SDK > --- > > Key: HADOOP-17089 > URL: https://issues.apache.org/jira/browse/HADOOP-17089 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.7.0, 2.8.0, 2.9.0, 3.0.0, 3.1.0, 3.2.0 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Major > > WASB depends on the Azure Storage Java SDK. There is a concurrency bug in > the Azure Storage Java SDK that can cause the results of a list blobs > operation to appear empty. This causes the Filesystem listStatus and similar > APIs to return empty results. This has been seen in Spark work loads when > jobs use more than one executor core. > See [https://github.com/Azure/azure-storage-java/pull/546] for details on the > bug in the Azure Storage SDK. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17015) ABFS: Make PUT and POST operations idempotent
[ https://issues.apache.org/jira/browse/HADOOP-17015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt resolved HADOOP-17015. -- Resolution: Fixed Sneha and I discussed this. The common Hadoop scenario is a case where you have one or more tasks, each operating on different source files, all attempting to rename to a common destination. In this scenario, the fix in PR 2021 is correct. There are scenarios where PR 2021 will lead to incorrect results, but they seem to be very contrived and unlikely in Hadoop. A work item will be opened to investigate the need to improve this on the server-side, for example by allowing an operation-id to be passed to the rename operation and persisted in the destination metadata, but for now we have this fix to the driver on the client-side. > ABFS: Make PUT and POST operations idempotent > - > > Key: HADOOP-17015 > URL: https://issues.apache.org/jira/browse/HADOOP-17015 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Fix For: 3.4.0 > > > Currently when a PUT or POST operation timeouts and the server has already > successfully executed the operation, there is no check in driver to see if > the operation did succeed or not and just retries the same operation again. > This can cause driver to through invalid user errors. > > Sample scenario: > # Rename request times out. Though server has successfully executed the > operation. > # Driver retries rename and get source not found error. > In the scenario, driver needs to check if rename is being retried and success > if source if not found, but destination is present. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17089) WASB: Update azure-storage-java SDK
Thomas Marqardt created HADOOP-17089: Summary: WASB: Update azure-storage-java SDK Key: HADOOP-17089 URL: https://issues.apache.org/jira/browse/HADOOP-17089 Project: Hadoop Common Issue Type: Bug Components: fs/azure Affects Versions: 3.2.0, 3.1.0, 3.0.0, 2.9.0, 2.8.0, 2.7.0 Reporter: Thomas Marqardt Assignee: Thomas Marqardt WASB depends on the Azure Storage Java SDK. There is a concurrency bug in the Azure Storage Java SDK that can cause the results of a list blobs operation to appear empty. This causes the Filesystem listStatus and similar APIs to return empty results. This has been seen in Spark work loads when jobs use more than one executor core. See [https://github.com/Azure/azure-storage-java/pull/546] for details on the bug in the Azure Storage SDK. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-17015) ABFS: Make PUT and POST operations idempotent
[ https://issues.apache.org/jira/browse/HADOOP-17015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt reopened HADOOP-17015: -- We should revisit PR 2021 and try to find a better solution for rename. Users expect Rename to be atomic. The service implementation is atomic, but we have this client-side idempotency issue. This fix relies on time and assumes that if the destination was recently updated while we are executing a retry policy, that we succeeded. This may not be the case. For example, users may rely on rename (with overwrite = false) of a file to synchronize or act like a distributed lock, so who ever renames successfully acquires the lock. With the fix in PR 2021, more than one caller could acquire this lock at the same time. Instead, I think we could allow the client to provide a UUID for the rename operation and persist this UUID in the metadata of the destination blob upon successful completion of a rename, then if we get into this idempotency issue and the client gets a 404 source does not exist, we can check the destination blob's metadata to see if the UUID is a match. > ABFS: Make PUT and POST operations idempotent > - > > Key: HADOOP-17015 > URL: https://issues.apache.org/jira/browse/HADOOP-17015 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Fix For: 3.4.0 > > > Currently when a PUT or POST operation timeouts and the server has already > successfully executed the operation, there is no check in driver to see if > the operation did succeed or not and just retries the same operation again. > This can cause driver to through invalid user errors. > > Sample scenario: > # Rename request times out. Though server has successfully executed the > operation. > # Driver retries rename and get source not found error. > In the scenario, driver needs to check if rename is being retried and success > if source if not found, but destination is present. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17054) ABFS: Fix idempotency test failures when SharedKey is set as AuthType
[ https://issues.apache.org/jira/browse/HADOOP-17054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt resolved HADOOP-17054. -- Resolution: Fixed Accidentally reactivated HADOOP-17015 but meant to reactivate HADOOP-17054. Please ignore previous comment. > ABFS: Fix idempotency test failures when SharedKey is set as AuthType > - > > Key: HADOOP-17054 > URL: https://issues.apache.org/jira/browse/HADOOP-17054 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Fix For: 3.4.0 > > > Idempotency related tests added as part of > https://issues.apache.org/jira/browse/HADOOP-17015 > create a test AbfsClient instance. This mock instance wrongly accepts valid > sharedKey and oauth token provider instance. This leads to test failures with > exceptions: > [ERROR] > testRenameRetryFailureAsHTTP404(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRename) > Time elapsed: 9.133 s <<< ERROR! > Invalid auth type: SharedKey is being used, expecting OAuth > at > org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getTokenProvider(AbfsConfiguration.java:643) > This Jira is to fix these tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17018) Intermittent failing of ITestAbfsStreamStatistics in ABFS
[ https://issues.apache.org/jira/browse/HADOOP-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17018: - Environment: (was: Committed to branch-3.3: commit a2f44344c346601607b9ac1de4598b754f9f2d72 Author: Mehakmeet Singh Date: Thu May 7 16:45:28 2020 +0530) Committed to branch-3.3: commit a2f44344c346601607b9ac1de4598b754f9f2d72 Author: Mehakmeet Singh Date: Thu May 7 16:45:28 2020 +0530 > Intermittent failing of ITestAbfsStreamStatistics in ABFS > - > > Key: HADOOP-17018 > URL: https://issues.apache.org/jira/browse/HADOOP-17018 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure, test >Affects Versions: 3.3.0 >Reporter: Mehakmeet Singh >Assignee: Mehakmeet Singh >Priority: Minor > Fix For: 3.3.1 > > > There are intermittent failures of a test inside ITestAbfsStreamStatistics in > ABFS. > Did consecutive runs of the test and failure seemed random. Stack Trace in > the comments. > Propose: > - Change the assertion of the test for it to be passed, Since the production > code seems fine. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17076) ABFS: Delegation SAS Generator Updates
[ https://issues.apache.org/jira/browse/HADOOP-17076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17076: - Fix Version/s: (was: 3.3.0) 3.3.1 Changing fix version from 3.3.0 to 3.3.1. Committed to branch-3.3: commit 63d236c019909d321f7824bb0e043c7bddd60bf0 Author: Thomas Marquardt Date: Wed Jun 17 23:12:22 2020 + > ABFS: Delegation SAS Generator Updates > -- > > Key: HADOOP-17076 > URL: https://issues.apache.org/jira/browse/HADOOP-17076 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Minor > Fix For: 3.3.1 > > > # The authentication version in the service has been updated from Dec19 to > Feb20, so need to update the client. > # Add support and test cases for getXattr and setXAttr. > # Update DelegationSASGenerator and related to use Duration instead of int > for time periods. > # Cleanup DelegationSASGenerator switch/case statement that maps operations > to permissions. > # Cleanup SASGenerator classes to use String.equals instead of ==. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17004) ABFS: Improve the ABFS driver documentation
[ https://issues.apache.org/jira/browse/HADOOP-17004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17004: - Fix Version/s: 3.3.1 Resolution: Fixed Status: Resolved (was: Patch Available) committed to branch-3.3: commit d639c119867cf382815da0e427776c59aba0f5c8 Author: bilaharith <52483117+bilahar...@users.noreply.github.com> Date: Thu May 21 00:21:48 2020 +0530 commit 11307f3be9be494ec880e036c78705c41ca8ceae Author: bilaharith <52483117+bilahar...@users.noreply.github.com> Date: Tue May 19 09:15:54 2020 +0530 > ABFS: Improve the ABFS driver documentation > --- > > Key: HADOOP-17004 > URL: https://issues.apache.org/jira/browse/HADOOP-17004 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0 >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Minor > Fix For: 3.3.1, 3.4.0 > > > * Add the missing configuration/settings details > * Mention the default vales -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16916) ABFS: Delegation SAS generator for integration with Ranger
[ https://issues.apache.org/jira/browse/HADOOP-16916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-16916: - Committed to branch-3.3: commit af98f32f7dbb9d71915690b66f12c33758011450 Author: Thomas Marquardt Date: Tue May 12 17:32:52 2020 + > ABFS: Delegation SAS generator for integration with Ranger > -- > > Key: HADOOP-16916 > URL: https://issues.apache.org/jira/browse/HADOOP-16916 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Minor > Fix For: 3.3.1 > > Attachments: HADOOP-16916.001.patch > > > HADOOP-16730 added support for Shared Access Signatures (SAS). Azure Data > Lake Storage Gen2 supports a new SAS type known as User Delegation SAS. This > Jira tracks an update to the ABFS driver that will include a Delegation SAS > generator and tests to validate that this SAS type is working correctly with > the driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17018) Intermittent failing of ITestAbfsStreamStatistics in ABFS
[ https://issues.apache.org/jira/browse/HADOOP-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17018: - Environment: Committed to branch-3.3: commit a2f44344c346601607b9ac1de4598b754f9f2d72 Author: Mehakmeet Singh Date: Thu May 7 16:45:28 2020 +0530 > Intermittent failing of ITestAbfsStreamStatistics in ABFS > - > > Key: HADOOP-17018 > URL: https://issues.apache.org/jira/browse/HADOOP-17018 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure, test >Affects Versions: 3.3.0 > Environment: Committed to branch-3.3: > commit a2f44344c346601607b9ac1de4598b754f9f2d72 > Author: Mehakmeet Singh > Date: Thu May 7 16:45:28 2020 +0530 >Reporter: Mehakmeet Singh >Assignee: Mehakmeet Singh >Priority: Minor > Fix For: 3.3.1 > > > There are intermittent failures of a test inside ITestAbfsStreamStatistics in > ABFS. > Did consecutive runs of the test and failure seemed random. Stack Trace in > the comments. > Propose: > - Change the assertion of the test for it to be passed, Since the production > code seems fine. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17002) ABFS: Avoid storage calls to check if the account is HNS enabled or not
[ https://issues.apache.org/jira/browse/HADOOP-17002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-17002: - Fix Version/s: 3.3.1 committed to branch-3.3. commit 76ee7e5494579b6f8adf1d86b17e97a63a8576ad Author: bilaharith <52483117+bilahar...@users.noreply.github.com> Date: Fri Apr 24 06:16:18 2020 +0530 > ABFS: Avoid storage calls to check if the account is HNS enabled or not > --- > > Key: HADOOP-17002 > URL: https://issues.apache.org/jira/browse/HADOOP-17002 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.4.0 >Reporter: Bilahari T H >Assignee: Bilahari T H >Priority: Minor > Fix For: 3.3.1, 3.4.0 > > > Each time an FS instance is created a Getacl call is made. If the call fails > with 400 Bad request, the account is determined to be a non-HNS account. > Recommendation is to create a config and be able to avoid store calls to > determine account HNS status, > If config is available, use that to determine account HNS status. If config > is not present in core-site, default behaviour will be calling getAcl. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-17054) ABFS: Fix idempotency test failures when SharedKey is set as AuthType
[ https://issues.apache.org/jira/browse/HADOOP-17054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt reopened HADOOP-17054: -- We should revisit this and try to find a better solution for rename. App developers expect our Rename to be atomic. The service implementation is atomic, but we have this client-side idempotency issue. Your fix relies on time and assumes that if the destination was recently updated while we are executing a retry policy, that we succeeded. This may not be the case. For example, app developers may rely on rename (with overwrite = false) of a file to synchronize or act like a distributed lock, so who ever renames successfully acquires the lock. With your fix, multiple callers could acquire this lock at the same time. Instead, I think we could allow the client to provide a UUID for the rename operation and persist this UUID in the metadata of the destination blob upon successful completion of a rename, then if we get into this idempotency issue and the client gets a 404 source does not exist, we can check the destination blob's metadata to see if the UUID is a match. > ABFS: Fix idempotency test failures when SharedKey is set as AuthType > - > > Key: HADOOP-17054 > URL: https://issues.apache.org/jira/browse/HADOOP-17054 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Fix For: 3.4.0 > > > Idempotency related tests added as part of > https://issues.apache.org/jira/browse/HADOOP-17015 > create a test AbfsClient instance. This mock instance wrongly accepts valid > sharedKey and oauth token provider instance. This leads to test failures with > exceptions: > [ERROR] > testRenameRetryFailureAsHTTP404(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemRename) > Time elapsed: 9.133 s <<< ERROR! > Invalid auth type: SharedKey is being used, expecting OAuth > at > org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getTokenProvider(AbfsConfiguration.java:643) > This Jira is to fix these tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17076) ABFS: Delegation SAS Generator Updates
[ https://issues.apache.org/jira/browse/HADOOP-17076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140746#comment-17140746 ] Thomas Marqardt commented on HADOOP-17076: -- Thanks Steve for letting me know. I will cherry-pick this into branch-3.3. I noticed that HADOOP-16916 also was not cherry-picked to branch-3.3, so I'll do that too. > ABFS: Delegation SAS Generator Updates > -- > > Key: HADOOP-17076 > URL: https://issues.apache.org/jira/browse/HADOOP-17076 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Minor > Fix For: 3.3.0 > > > # The authentication version in the service has been updated from Dec19 to > Feb20, so need to update the client. > # Add support and test cases for getXattr and setXAttr. > # Update DelegationSASGenerator and related to use Duration instead of int > for time periods. > # Cleanup DelegationSASGenerator switch/case statement that maps operations > to permissions. > # Cleanup SASGenerator classes to use String.equals instead of ==. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17076) ABFS: Delegation SAS Generator Updates
[ https://issues.apache.org/jira/browse/HADOOP-17076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt resolved HADOOP-17076. -- Fix Version/s: 3.3.0 Release Note: Azure Blob File System (ABFS) SAS Generator Update Resolution: Fixed commit caf3995ac2bbc3241896babb9a607272462f70ca Author: Thomas Marquardt Date: Wed Jun 17 23:12:22 2020 + HADOOP-17076: ABFS: Delegation SAS Generator Updates Contributed by Thomas Marquardt. > ABFS: Delegation SAS Generator Updates > -- > > Key: HADOOP-17076 > URL: https://issues.apache.org/jira/browse/HADOOP-17076 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Minor > Fix For: 3.3.0 > > > # The authentication version in the service has been updated from Dec19 to > Feb20, so need to update the client. > # Add support and test cases for getXattr and setXAttr. > # Update DelegationSASGenerator and related to use Duration instead of int > for time periods. > # Cleanup DelegationSASGenerator switch/case statement that maps operations > to permissions. > # Cleanup SASGenerator classes to use String.equals instead of ==. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17076) ABFS: Delegation SAS Generator Updates
[ https://issues.apache.org/jira/browse/HADOOP-17076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17138902#comment-17138902 ] Thomas Marqardt commented on HADOOP-17076: -- Created [https://github.com/apache/hadoop/pull/2081]. See PR description for details. > ABFS: Delegation SAS Generator Updates > -- > > Key: HADOOP-17076 > URL: https://issues.apache.org/jira/browse/HADOOP-17076 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Minor > > # The authentication version in the service has been updated from Dec19 to > Feb20, so need to update the client. > # Add support and test cases for getXattr and setXAttr. > # Update DelegationSASGenerator and related to use Duration instead of int > for time periods. > # Cleanup DelegationSASGenerator switch/case statement that maps operations > to permissions. > # Cleanup SASGenerator classes to use String.equals instead of ==. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17076) ABFS: Delegation SAS Generator Updates
Thomas Marqardt created HADOOP-17076: Summary: ABFS: Delegation SAS Generator Updates Key: HADOOP-17076 URL: https://issues.apache.org/jira/browse/HADOOP-17076 Project: Hadoop Common Issue Type: New Feature Components: fs/azure Affects Versions: 3.2.1 Reporter: Thomas Marqardt Assignee: Thomas Marqardt # The authentication version in the service has been updated from Dec19 to Feb20, so need to update the client. # Add support and test cases for getXattr and setXAttr. # Update DelegationSASGenerator and related to use Duration instead of int for time periods. # Cleanup DelegationSASGenerator switch/case statement that maps operations to permissions. # Cleanup SASGenerator classes to use String.equals instead of ==. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-17075) Improvement to the AccessControlException thrown by Azure abfs driver
[ https://issues.apache.org/jira/browse/HADOOP-17075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt reassigned HADOOP-17075: Assignee: Sneha Vijayarajan > Improvement to the AccessControlException thrown by Azure abfs driver > - > > Key: HADOOP-17075 > URL: https://issues.apache.org/jira/browse/HADOOP-17075 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Ramesh Mani >Assignee: Sneha Vijayarajan >Priority: Major > > Currently when an AccessControlException happens in Abfs driver call it > prints the entire stack trace. To be consistent with the HDFS way of showing > the Permission denied, could we modify this in ABFSClient? > > e.g: > >$ hdfs dfs -ls /hbase/mobdir > ls: Permission denied: user=user1, access=READ_EXECUTE, > inode="/hbase/mobdir":hbase:hbase:drwx-- -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16916) ABFS: Delegation SAS generator for integration with Ranger
[ https://issues.apache.org/jira/browse/HADOOP-16916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-16916: - Fix Version/s: 3.3.1 Release Note: Azure ABFS support for Shared Access Signatures (SAS) Resolution: Fixed Status: Resolved (was: Patch Available) commit b214bbd2d92a0c02b71d352dba85f3b87317933c Author: Thomas Marquardt Date: Tue May 12 17:32:52 2020 + Test Results: namespace.enabled=true auth.type=SharedKey --- $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 63, Failures: 0, Errors: 0, Skipped: 0 Tests run: 432, Failures: 0, Errors: 0, Skipped: 41 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 namespace.enabled=false auth.type=SharedKey --- $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 63, Failures: 0, Errors: 0, Skipped: 0 Tests run: 432, Failures: 0, Errors: 0, Skipped: 244 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 namespace.enabled=true auth.type=SharedKey sas.token.provider.type=MockDelegationSASTokenProvider enable.check.access=true --- $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 63, Failures: 0, Errors: 0, Skipped: 0 Tests run: 432, Failures: 0, Errors: 0, Skipped: 33 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 namespace.enabled=true auth.type=OAuth --- $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 63, Failures: 0, Errors: 0, Skipped: 0 Tests run: 432, Failures: 0, Errors: 1, Skipped: 74 Tests run: 206, Failures: 0, Errors: 0, Skipped: 140 > ABFS: Delegation SAS generator for integration with Ranger > -- > > Key: HADOOP-16916 > URL: https://issues.apache.org/jira/browse/HADOOP-16916 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Minor > Fix For: 3.3.1 > > Attachments: HADOOP-16916.001.patch > > > HADOOP-16730 added support for Shared Access Signatures (SAS). Azure Data > Lake Storage Gen2 supports a new SAS type known as User Delegation SAS. This > Jira tracks an update to the ABFS driver that will include a Delegation SAS > generator and tests to validate that this SAS type is working correctly with > the driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16916) ABFS: Delegation SAS generator for integration with Ranger
[ https://issues.apache.org/jira/browse/HADOOP-16916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105027#comment-17105027 ] Thomas Marqardt commented on HADOOP-16916: -- Updated PR 1965 to address feedback from Sneha. The configuration to set the REST version has been removed and the ordering of imports has been fixed. All tests are passing against my accounts. Hierarchical Namespace enabled: $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 63, Failures: 0, Errors: 0, Skipped: 0 Tests run: 432, Failures: 0, Errors: 0, Skipped: 41 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 Hierarchical Namespace disabled: $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 63, Failures: 0, Errors: 0, Skipped: 0 Tests run: 432, Failures: 0, Errors: 0, Skipped: 244 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 > ABFS: Delegation SAS generator for integration with Ranger > -- > > Key: HADOOP-16916 > URL: https://issues.apache.org/jira/browse/HADOOP-16916 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Minor > Attachments: HADOOP-16916.001.patch > > > HADOOP-16730 added support for Shared Access Signatures (SAS). Azure Data > Lake Storage Gen2 supports a new SAS type known as User Delegation SAS. This > Jira tracks an update to the ABFS driver that will include a Delegation SAS > generator and tests to validate that this SAS type is working correctly with > the driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16916) ABFS: Delegation SAS generator for integration with Ranger
[ https://issues.apache.org/jira/browse/HADOOP-16916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17100297#comment-17100297 ] Thomas Marqardt commented on HADOOP-16916: -- I updated PR 1965 with the following: 1. With hierarchical namespace disabled, some list operations were failing. It turns out that when it is disabled, the directory query parameter used by the List Paths API must not start with a forward slash '/'. The continuation token was also affected by this, so I have fixed both of these issues. 2. With hierarchical namespace disabled, the ITestAzureBlobFileSystemAuthorization tests that use ACLs were missing Assume.assumeTrue(this.getFileSystem().getIsNamespaceEnabled()) so I have added that. 3. I changed the default for fs.azure.enable.check.access back to false. We will wait for the next release of ADLS Gen2 REST API before changing this to true. My USWest2 account with namespace disabled: $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 63, Failures: 0, Errors: 0, Skipped: 0 Tests run: 432, Failures: 0, Errors: 0, Skipped: 244 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 My USWest2 account with namespace enabled: $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 63, Failures: 0, Errors: 0, Skipped: 0 Tests run: 432, Failures: 0, Errors: 0, Skipped: 41 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 My eastus2euap test account with namespace enabled: $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 63, Failures: 0, Errors: 0, Skipped: 0 Tests run: 432, Failures: 0, Errors: 0, Skipped: 41 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 My eastus2euap test account with namespace enabled and Delegation SAS tests enabled: $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 63, Failures: 0, Errors: 0, Skipped: 0 Tests run: 432, Failures: 0, Errors: 0, Skipped: 33 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 > ABFS: Delegation SAS generator for integration with Ranger > -- > > Key: HADOOP-16916 > URL: https://issues.apache.org/jira/browse/HADOOP-16916 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Minor > Attachments: HADOOP-16916.001.patch > > > HADOOP-16730 added support for Shared Access Signatures (SAS). Azure Data > Lake Storage Gen2 supports a new SAS type known as User Delegation SAS. This > Jira tracks an update to the ABFS driver that will include a Delegation SAS > generator and tests to validate that this SAS type is working correctly with > the driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16916) ABFS: Delegation SAS generator for integration with Ranger
[ https://issues.apache.org/jira/browse/HADOOP-16916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098624#comment-17098624 ] Thomas Marqardt commented on HADOOP-16916: -- I updated PR 1965 so that we can complete this sooner, the following updates have been made: 1) Added configuration "fs.azure.rest.version" with a default of 2018-11-09. To run ITestAzureBlobFileSystemDelegationSAS.testList you need to temporariliy set this to 2019-12-12 due to a server-side bug that will be fixed in the weeks to come. 2) AzureBlobFileSystem.getFileStatus is currently calling the GetAccessControl REST API. My previous PR fixed this to call GetStatus REST API, but the fix depends on a server-side bug that will not be available for a few weeks. Instead we'll postpone this until later, can continue calling GetAccessControl for now. With these changes, all tests are passing. Results for my production account in USWest2 without Delegation SAS tests: Tests run: 63, Failures: 0, Errors: 0, Skipped: 0 Tests run: 432, Failures: 0, Errors: 0, Skipped: 41 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 Results for my test account in eastus2euap with Delegation SAS tests: Tests run: 63, Failures: 0, Errors: 0, Skipped: 0 Tests run: 432, Failures: 0, Errors: 0, Skipped: 33 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 > ABFS: Delegation SAS generator for integration with Ranger > -- > > Key: HADOOP-16916 > URL: https://issues.apache.org/jira/browse/HADOOP-16916 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Minor > Attachments: HADOOP-16916.001.patch > > > HADOOP-16730 added support for Shared Access Signatures (SAS). Azure Data > Lake Storage Gen2 supports a new SAS type known as User Delegation SAS. This > Jira tracks an update to the ABFS driver that will include a Delegation SAS > generator and tests to validate that this SAS type is working correctly with > the driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16916) ABFS: Delegation SAS generator for integration with Ranger
[ https://issues.apache.org/jira/browse/HADOOP-16916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087060#comment-17087060 ] Thomas Marqardt commented on HADOOP-16916: -- I updated the PR with fixes for checkstyle and suppression for findbugs. Also note there is a description of the changes in the PR and that all tests are passing against my eastus2euap storage account: $ mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify [INFO] Tests run: 56, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 424, Failures: 0, Errors: 0, Skipped: 33 [WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 > ABFS: Delegation SAS generator for integration with Ranger > -- > > Key: HADOOP-16916 > URL: https://issues.apache.org/jira/browse/HADOOP-16916 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Minor > Attachments: HADOOP-16916.001.patch > > > HADOOP-16730 added support for Shared Access Signatures (SAS). Azure Data > Lake Storage Gen2 supports a new SAS type known as User Delegation SAS. This > Jira tracks an update to the ABFS driver that will include a Delegation SAS > generator and tests to validate that this SAS type is working correctly with > the driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16916) ABFS: Delegation SAS generator for integration with Ranger
[ https://issues.apache.org/jira/browse/HADOOP-16916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086779#comment-17086779 ] Thomas Marqardt commented on HADOOP-16916: -- Sorry for the long delay but I was waiting for bug fixes to be deployed to the ADLS Gen2 service. That has finally happened but the fixes have not deployed globally yet, so we will need to wait for that to complete. In the meantime you can review PR 1965, which I have carefully reviewed myself and validated against an account where the bug fixes have deployed. > ABFS: Delegation SAS generator for integration with Ranger > -- > > Key: HADOOP-16916 > URL: https://issues.apache.org/jira/browse/HADOOP-16916 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Minor > Attachments: HADOOP-16916.001.patch > > > HADOOP-16730 added support for Shared Access Signatures (SAS). Azure Data > Lake Storage Gen2 supports a new SAS type known as User Delegation SAS. This > Jira tracks an update to the ABFS driver that will include a Delegation SAS > generator and tests to validate that this SAS type is working correctly with > the driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15669) ABFS: Improve HTTPS Performance
[ https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070621#comment-17070621 ] Thomas Marqardt commented on HADOOP-15669: -- Ok, this is a performance feature so it is not critical for functional testing. > ABFS: Improve HTTPS Performance > --- > > Key: HADOOP-15669 > URL: https://issues.apache.org/jira/browse/HADOOP-15669 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marqardt >Assignee: Vishwajeet Dusane >Priority: Major > Fix For: 3.2.0 > > Attachments: ABFS - Improve HTTPS Performance Over Java Based > Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, > HADOOP-15669-HADOOP-15407-02.patch, HADOOP-15669-HADOOP-15407-03.patch, > HADOOP-15669-HADOOP-15407-04.patch > > > We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP. Lets > perform a detailed measurement and see what can be done to improve throughput. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15669) ABFS: Improve HTTPS Performance
[ https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070489#comment-17070489 ] Thomas Marqardt commented on HADOOP-15669: -- Please ensure that you have Open SSL installed and are using Java 8 or higher. > ABFS: Improve HTTPS Performance > --- > > Key: HADOOP-15669 > URL: https://issues.apache.org/jira/browse/HADOOP-15669 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marqardt >Assignee: Vishwajeet Dusane >Priority: Major > Fix For: 3.2.0 > > Attachments: ABFS - Improve HTTPS Performance Over Java Based > Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, > HADOOP-15669-HADOOP-15407-02.patch, HADOOP-15669-HADOOP-15407-03.patch, > HADOOP-15669-HADOOP-15407-04.patch > > > We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP. Lets > perform a detailed measurement and see what can be done to improve throughput. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16916) ABFS: Delegation SAS generator for integration with Ranger
[ https://issues.apache.org/jira/browse/HADOOP-16916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-16916: - Attachment: HADOOP-16916.001.patch Status: Patch Available (was: Open) Submitting patch HADOOP-16916.001.patch. This is the first draft and will be iterated on. This patch adds tests in ITestAzureBlobFileSystemDelegationSAS that have a dependency on new Delegation SAS features that are not yet available in ADLS Gen2. These tests are not run by default so all the pre-existing tests are still passing with this change. We may wait for the new ADLS Gen2 features to be available before committing this patch. This patch adds a DelegationSASGenerator which returns SAS with minimal permissions to the caller. This is for testing purposes to ensure that the ABFS driver operations succeed with minimal permission SAS. This patch adds a MockDelegationSASTokenProvider which calls the DelegationSASGenerator to provide SAS tokens. The MockDelegationSASTokenProvider relies on an Azure app registration and client credential grant flow to obtain a user delegation key for signing SAS tokens. This is not the way the SASTokenProvider should be used in production, since this test scenario allows the potentially low privilege user of ABFS to access the credentials used by the SASTokenProvider. In production, it is expected that a low privilege user would not have access to these credentials, for example the SASTokenProvider could use an endpoint which authenticates the low privilege user and returns SAS to the user based on authorization rules. All tests passing against my US West account: $ mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 Tests run: 420, Failures: 0, Errors: 0, Skipped: 41 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 > ABFS: Delegation SAS generator for integration with Ranger > -- > > Key: HADOOP-16916 > URL: https://issues.apache.org/jira/browse/HADOOP-16916 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Minor > Attachments: HADOOP-16916.001.patch > > > HADOOP-16730 added support for Shared Access Signatures (SAS). Azure Data > Lake Storage Gen2 supports a new SAS type known as User Delegation SAS. This > Jira tracks an update to the ABFS driver that will include a Delegation SAS > generator and tests to validate that this SAS type is working correctly with > the driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16916) ABFS: Delegation SAS generator for integration with Ranger
Thomas Marqardt created HADOOP-16916: Summary: ABFS: Delegation SAS generator for integration with Ranger Key: HADOOP-16916 URL: https://issues.apache.org/jira/browse/HADOOP-16916 Project: Hadoop Common Issue Type: New Feature Components: fs/azure Affects Versions: 3.2.1 Reporter: Thomas Marqardt HADOOP-16730 added support for Shared Access Signatures (SAS). Azure Data Lake Storage Gen2 supports a new SAS type known as User Delegation SAS. This Jira tracks an update to the ABFS driver that will include a Delegation SAS generator and tests to validate that this SAS type is working correctly with the driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16916) ABFS: Delegation SAS generator for integration with Ranger
[ https://issues.apache.org/jira/browse/HADOOP-16916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt reassigned HADOOP-16916: Assignee: Thomas Marqardt > ABFS: Delegation SAS generator for integration with Ranger > -- > > Key: HADOOP-16916 > URL: https://issues.apache.org/jira/browse/HADOOP-16916 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Thomas Marqardt >Assignee: Thomas Marqardt >Priority: Minor > > HADOOP-16730 added support for Shared Access Signatures (SAS). Azure Data > Lake Storage Gen2 supports a new SAS type known as User Delegation SAS. This > Jira tracks an update to the ABFS driver that will include a Delegation SAS > generator and tests to validate that this SAS type is working correctly with > the driver. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16730) ABFS: Support for Shared Access Signatures (SAS)
[ https://issues.apache.org/jira/browse/HADOOP-16730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt updated HADOOP-16730: - Fix Version/s: 3.3.1 Release Note: ABFS support for a SAS token provider plug-in. Resolution: Fixed Status: Resolved (was: Patch Available) commit 791270a2e5e31546ff5c1ef4fa8bad6852b906dc Author: Sneha Vijayarajan Date: Thu Feb 27 17:00:15 2020 + HADOOP-16730: ABFS: Support for Shared Access Signatures (SAS). Contributed by Sneha Vijayarajan. > ABFS: Support for Shared Access Signatures (SAS) > > > Key: HADOOP-16730 > URL: https://issues.apache.org/jira/browse/HADOOP-16730 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Thomas Marqardt >Assignee: Sneha Vijayarajan >Priority: Major > Fix For: 3.3.1 > > Original Estimate: 1,008h > Remaining Estimate: 1,008h > > ABFS supports OAuth and Shared Key but currently lacks support for [Shared > Access Signatures > (SAS)|[https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview]]. > SAS is a great way to constrain access to a low-privilege ABFS client. The > ABFS client does not need to possess persistent credentials for accessing > storage but instead can request temporary, constrained access tokens from a > trusted endpoint. This endpoint can authenticate the caller, make an > authorization decision and return a constrained SAS token. The token may > have an expiration, it may be scoped to a specific file or directory, and it > may grant an action or set of actions such as read, write, list, or delete. > Azure Storage also has a new identity based SAS scheme in preview named > Delegation SAS. These new Delegation SAS have these advantages over Service > SAS: > 1) Delegation SAS provide authentication as well as authorization. The user > identity associated with each request will appear in the logs when logging is > enabled for the account. > 2) Instead of using storage account keys to sign tokens, Delegation SAS > relies on keys assigned to each user. These keys are called user delegation > keys. If a storage account key is leaked, an attacker would have full access > to the storage account. If a user delegation key is leaked, an attacker > would only have access to resources that user has access to within the Blob > service–for example, the user might only have read access to a specific > container. > This feature will add support for the ABFS driver to authenticate against a > trusted endpoint. The endpoint will return a SAS which the ABFS driver will > use to access Azure storage. The SAS may be a container or directory SAS to > be used for all subsequent operations, and thus cached for the lifetime of > the filesystem. Or it may be a SAS to be used for the current filesystem > operation, in this case, the ABFS driver will request a SAS for each > operation. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16845) ITestAbfsClient.testContinuationTokenHavingEqualSign failing
[ https://issues.apache.org/jira/browse/HADOOP-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt resolved HADOOP-16845. -- Fix Version/s: 3.3.0 Resolution: Fixed commit 55f2421580678a6793c8cb6ad10fee3f4ec833aa Author: Sneha Vijayarajan Date: Thu Feb 6 18:41:06 2020 + HADOOP-16845: Disable ITestAbfsClient.testContinuationTokenHavingEqualSign due to ADLS Gen2 service bug. Contributed by Sneha Vijayarajan. > ITestAbfsClient.testContinuationTokenHavingEqualSign failing > > > Key: HADOOP-16845 > URL: https://issues.apache.org/jira/browse/HADOOP-16845 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, test >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > Fix For: 3.3.0 > > > Testcase testContinuationTokenHavingEqualSign is failing as request that was > expected to fail is passing. > There is change in the queryparam validation in ContinuationToken at server > end wihch has resulted in this behaviour. > Server request trace: > 2020-02-05 16:59:17,001 DEBUG [JUnit-testContinuationTokenHavingEqualSign]: > services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(263)) - > HttpRequest: > 200,,cid=87c3ebea-def7-4fdd-a21a-a56c63a59387,rid=0931c565-201f-004c-1317-dcdd9000,sent=0,recv=0,GET,[https://snvijayaabfsns.dfs.core.windows.net/abfs-testcontainer-85bb9523-fccd-45f3-ae6d-37622d8231e5?upn=false=filesystem=500=/=%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D=90=true] > > Disabling the test until the server fix is in and deployed on all regions. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16845) ITestAbfsClient.testContinuationTokenHavingEqualSign failing
[ https://issues.apache.org/jira/browse/HADOOP-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031899#comment-17031899 ] Thomas Marqardt commented on HADOOP-16845: -- All tests passing for my account in uswest2 with fs.azure.test.namespace.enabled set to true: Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 Tests run: 414, Failures: 0, Errors: 0, Skipped: 32 Tests run: 194, Failures: 0, Errors: 0, Skipped: 24 > ITestAbfsClient.testContinuationTokenHavingEqualSign failing > > > Key: HADOOP-16845 > URL: https://issues.apache.org/jira/browse/HADOOP-16845 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, test >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > > Testcase testContinuationTokenHavingEqualSign is failing as request that was > expected to fail is passing. > There is change in the queryparam validation in ContinuationToken at server > end wihch has resulted in this behaviour. > Server request trace: > 2020-02-05 16:59:17,001 DEBUG [JUnit-testContinuationTokenHavingEqualSign]: > services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(263)) - > HttpRequest: > 200,,cid=87c3ebea-def7-4fdd-a21a-a56c63a59387,rid=0931c565-201f-004c-1317-dcdd9000,sent=0,recv=0,GET,[https://snvijayaabfsns.dfs.core.windows.net/abfs-testcontainer-85bb9523-fccd-45f3-ae6d-37622d8231e5?upn=false=filesystem=500=/=%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D=90=true] > > Disabling the test until the server fix is in and deployed on all regions. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16825) ITestAzureBlobFileSystemCheckAccess failing
[ https://issues.apache.org/jira/browse/HADOOP-16825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marqardt resolved HADOOP-16825. -- Fix Version/s: 3.3.0 Resolution: Fixed commit 5944d28130925fe1452f545e96b5e44f064bc69e Author: bilaharith <52483117+bilahar...@users.noreply.github.com> Date: Thu Feb 6 18:48:00 2020 + HADOOP-16825: ITestAzureBlobFileSystemCheckAccess failing. Contributed by Bilahari T H. > ITestAzureBlobFileSystemCheckAccess failing > --- > > Key: HADOOP-16825 > URL: https://issues.apache.org/jira/browse/HADOOP-16825 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Bilahari T H >Priority: Major > Fix For: 3.3.0 > > > Tests added in HADOOP-16455 are failing. > java.lang.IllegalArgumentException: The value of property > fs.azure.account.oauth2.client.id must not be null > Looks to me like there are new configuration options which are undocumented > # these need documentation in testing markdown file > # tests MUST downgrade to skip if not set -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16825) ITestAzureBlobFileSystemCheckAccess failing
[ https://issues.apache.org/jira/browse/HADOOP-16825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031895#comment-17031895 ] Thomas Marqardt commented on HADOOP-16825: -- All tests passing for my account in uswest2 with fs.azure.test.namespace.enabled set to true: Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 Tests run: 414, Failures: 0, Errors: 0, Skipped: 32 Tests run: 194, Failures: 0, Errors: 0, Skipped: 24 > ITestAzureBlobFileSystemCheckAccess failing > --- > > Key: HADOOP-16825 > URL: https://issues.apache.org/jira/browse/HADOOP-16825 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Bilahari T H >Priority: Major > > Tests added in HADOOP-16455 are failing. > java.lang.IllegalArgumentException: The value of property > fs.azure.account.oauth2.client.id must not be null > Looks to me like there are new configuration options which are undocumented > # these need documentation in testing markdown file > # tests MUST downgrade to skip if not set -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16845) ITestAbfsClient.testContinuationTokenHavingEqualSign failing
[ https://issues.apache.org/jira/browse/HADOOP-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031126#comment-17031126 ] Thomas Marqardt commented on HADOOP-16845: -- Sneha, thank you, please attach your test results. > ITestAbfsClient.testContinuationTokenHavingEqualSign failing > > > Key: HADOOP-16845 > URL: https://issues.apache.org/jira/browse/HADOOP-16845 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, test >Affects Versions: 3.3.0 >Reporter: Sneha Vijayarajan >Assignee: Sneha Vijayarajan >Priority: Major > > Testcase testContinuationTokenHavingEqualSign is failing as request that was > expected to fail is passing. > There is change in the queryparam validation in ContinuationToken at server > end wihch has resulted in this behaviour. > Server request trace: > 2020-02-05 16:59:17,001 DEBUG [JUnit-testContinuationTokenHavingEqualSign]: > services.AbfsClient (AbfsRestOperation.java:executeHttpOperation(263)) - > HttpRequest: > 200,,cid=87c3ebea-def7-4fdd-a21a-a56c63a59387,rid=0931c565-201f-004c-1317-dcdd9000,sent=0,recv=0,GET,[https://snvijayaabfsns.dfs.core.windows.net/abfs-testcontainer-85bb9523-fccd-45f3-ae6d-37622d8231e5?upn=false=filesystem=500=/=%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D%3D=90=true] > > Disabling the test until the server fix is in and deployed on all regions. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16785) Improve wasb and abfs resilience on double close() calls
[ https://issues.apache.org/jira/browse/HADOOP-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011497#comment-17011497 ] Thomas Marqardt commented on HADOOP-16785: -- I’m not setup to build/test at the moment, so could you use two streams writing to the same file and verify whether or not there is actually another issue? I hope FilterOutputStream.close calls AbfsOutputStream.close and there is not an issue. > Improve wasb and abfs resilience on double close() calls > > > Key: HADOOP-16785 > URL: https://issues.apache.org/jira/browse/HADOOP-16785 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.2.2 > > > # if you call write() after the NativeAzureFsOutputStream is closed it throws > an NPE ... which isn't always caught by closeQuietly code. It needs to raise > an IOE > # abfs close ops can trigger failures in try-with-resources use -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16730) ABFS: Support for Shared Access Signatures (SAS)
Thomas Marqardt created HADOOP-16730: Summary: ABFS: Support for Shared Access Signatures (SAS) Key: HADOOP-16730 URL: https://issues.apache.org/jira/browse/HADOOP-16730 Project: Hadoop Common Issue Type: New Feature Components: fs/azure Affects Versions: 3.2.1 Reporter: Thomas Marqardt Assignee: Sneha Vijayarajan ABFS supports OAuth and Shared Key but currently lacks support for [Shared Access Signatures (SAS)|[https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview]]. SAS is a great way to constrain access to a low-privilege ABFS client. The ABFS client does not need to possess persistent credentials for accessing storage but instead can request temporary, constrained access tokens from a trusted endpoint. This endpoint can authenticate the caller, make an authorization decision and return a constrained SAS token. The token may have an expiration, it may be scoped to a specific file or directory, and it may grant an action or set of actions such as read, write, list, or delete. Azure Storage also has a new identity based SAS scheme in preview named Delegation SAS. These new Delegation SAS have these advantages over Service SAS: 1) Delegation SAS provide authentication as well as authorization. The user identity associated with each request will appear in the logs when logging is enabled for the account. 2) Instead of using storage account keys to sign tokens, Delegation SAS relies on keys assigned to each user. These keys are called user delegation keys. If a storage account key is leaked, an attacker would have full access to the storage account. If a user delegation key is leaked, an attacker would only have access to resources that user has access to within the Blob service–for example, the user might only have read access to a specific container. This feature will add support for the ABFS driver to authenticate against a trusted endpoint. The endpoint will return a SAS which the ABFS driver will use to access Azure storage. The SAS may be a container or directory SAS to be used for all subsequent operations, and thus cached for the lifetime of the filesystem. Or it may be a SAS to be used for the current filesystem operation, in this case, the ABFS driver will request a SAS for each operation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
