[
https://issues.apache.org/jira/browse/HADOOP-18656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17836067#comment-17836067
]
ASF GitHub Bot commented on HADOOP-18656:
-----------------------------------------
anujmodi2021 commented on PR #6718:
URL: https://github.com/apache/hadoop/pull/6718#issuecomment-2049229760
------------------------------
:::: AGGREGATED TEST RESULT ::::
============================================================
HNS-OAuth
============================================================
[ERROR]
testListPathWithValueGreaterThanServerMaximum(org.apache.hadoop.fs.azurebfs.ITestAbfsClient)
Time elapsed: 290.912 s <<< FAILURE!
[ERROR]
test_120_terasort(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort) Time
elapsed: 4.531 s <<< ERROR!
[WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 2
[ERROR] Tests run: 623, Failures: 1, Errors: 0, Skipped: 73
[ERROR] Tests run: 380, Failures: 0, Errors: 1, Skipped: 55
============================================================
HNS-SharedKey
============================================================
[ERROR]
testListPathWithValueGreaterThanServerMaximum(org.apache.hadoop.fs.azurebfs.ITestAbfsClient)
Time elapsed: 237.663 s <<< FAILURE!
[WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 3
[ERROR] Tests run: 623, Failures: 1, Errors: 0, Skipped: 28
[WARNING] Tests run: 380, Failures: 0, Errors: 0, Skipped: 41
============================================================
NonHNS-SharedKey
============================================================
[WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 607, Failures: 0, Errors: 0, Skipped: 269
[WARNING] Tests run: 380, Failures: 0, Errors: 0, Skipped: 44
============================================================
AppendBlob-HNS-OAuth
============================================================
[ERROR]
testCloseOfDataBlockOnAppendComplete(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemAppend)
Time elapsed: 9.219 s <<< FAILURE!
[ERROR]
testListPathWithValueGreaterThanServerMaximum(org.apache.hadoop.fs.azurebfs.ITestAbfsClient)
Time elapsed: 226.54 s <<< FAILURE!
[ERROR]
testAbfsStreamOps(org.apache.hadoop.fs.azurebfs.ITestAbfsStreamStatistics)
Time elapsed: 5.942 s <<< FAILURE!
[ERROR]
testExpect100ContinueFailureInAppend(org.apache.hadoop.fs.azurebfs.services.ITestAbfsOutputStream)
Time elapsed: 5.002 s <<< ERROR!
[ERROR]
testAppendWithChecksumAtDifferentOffsets(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemChecksum)
Time elapsed: 6.037 s <<< ERROR!
[ERROR]
testTwoWritersCreateAppendNoInfiniteLease(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemLease)
Time elapsed: 3.717 s <<< ERROR!
[ERROR]
test_120_terasort(org.apache.hadoop.fs.azurebfs.commit.ITestAbfsTerasort) Time
elapsed: 4.503 s <<< ERROR!
[WARNING] Tests run: 137, Failures: 0, Errors: 0, Skipped: 2
[ERROR] Tests run: 623, Failures: 2, Errors: 3, Skipped: 73
[ERROR] Tests run: 380, Failures: 1, Errors: 1, Skipped: 79
Time taken: 60 mins 25 secs.
> ABFS: Support for Pagination in Recursive Directory Delete
> -----------------------------------------------------------
>
> Key: HADOOP-18656
> URL: https://issues.apache.org/jira/browse/HADOOP-18656
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.3.5
> Reporter: Sree Bhattacharyya
> Assignee: Anuj Modi
> Priority: Minor
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> Today, when a recursive delete is issued for a large directory in ADLS Gen2
> (HNS) account, the directory deletion happens in O(1) but in backend ACL
> Checks are done recursively for each object inside that directory which in
> case of large directory could lead to request time out. Pagination is
> introduced in the Azure Storage Backend for these ACL checks.
> More information on how pagination works can be found on public documentation
> of [Azure Delete Path
> API|https://learn.microsoft.com/en-us/rest/api/storageservices/datalakestoragegen2/path/delete?view=rest-storageservices-datalakestoragegen2-2019-12-12].
> This PR contains changes to support this from client side. To trigger
> pagination, client needs to add a new query parameter "paginated" and set it
> to true along with recursive set to true. In return if the directory is
> large, server might return a continuation token back to the caller. If caller
> gets back a continuation token, it has to call the delete API again with
> continuation token along with recursive and pagination set to true. This is
> similar to directory delete of FNS account.
> Pagination is available only in versions "2023-08-03" onwards.
> PR also contains functional tests to verify driver works well with different
> combinations of recursive and pagination features for HNS.
> Full E2E testing of pagination requires large dataset to be created and hence
> not added as part of driver test suite. But extensive E2E testing has been
> performed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
