[
https://issues.apache.org/jira/browse/HADOOP-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17767372#comment-17767372
]
ASF GitHub Bot commented on HADOOP-18873:
-----------------------------------------
saxenapranav opened a new pull request, #6105:
URL: https://github.com/apache/hadoop/pull/6105
JIRA: https://issues.apache.org/jira/browse/HADOOP-18873
Trunk PR: https://github.com/apache/hadoop/pull/6010
Commit cherrypicked from trunk:
https://github.com/apache/hadoop/commit/f24b73e5f3ac640f491231f02d9d8afaf1855b5c
------------------------
:::: AGGREGATED TEST RESULT ::::
HNS-OAuth
========================
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:336 » TestTimedOut
test timed o...
[INFO]
[ERROR] Tests run: 588, Failures: 0, Errors: 1, Skipped: 54
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 41
HNS-SharedKey
========================
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] ITestAzureBlobFileSystemLease.testAcquireRetry:336 » TestTimedOut
test timed o...
[INFO]
[ERROR] Tests run: 588, Failures: 0, Errors: 1, Skipped: 54
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 41
NonHNS-SharedKey
========================
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 11
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
ITestAzureBlobFileSystemRandomRead.testSkipBounds:218->Assert.assertTrue:42->Assert.fail:89
There should not be any network I/O (elapsedTimeMs=25).
[ERROR] Errors:
[ERROR]
ITestAzureBlobFileSystemLease.testAcquireRetry:344->lambda$testAcquireRetry$6:345
» TestTimedOut
[INFO]
[ERROR] Tests run: 588, Failures: 1, Errors: 1, Skipped: 277
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 44
AppendBlob-HNS-OAuth
========================
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 141, Failures: 1, Errors: 0, Skipped: 5
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:269->Assert.assertTrue:42->Assert.fail:89
There should not be any network I/O (elapsedTimeMs=113).
[ERROR] Errors:
[ERROR]
ITestAzureBlobFileSystemLease.testAcquireRetry:344->lambda$testAcquireRetry$6:345
» TestTimedOut
[INFO]
[ERROR] Tests run: 588, Failures: 1, Errors: 1, Skipped: 54
[INFO] Results:
[INFO]
[WARNING] Tests run: 339, Failures: 0, Errors: 0, Skipped: 41
Time taken: 45 mins 16 secs.
azureuser@Hadoop-VM-EAST2:~/hadoop/hadoop-tools/hadoop-azure$ git log
commit 3f87cc52cac5d7dd57a6e65ca1b26a34aa57781d (HEAD ->
branch-3.3_HADOOP-18873, origin/branch-3.3_HADOOP-18873)
Author: Pranav Saxena <[email protected]>
Date: Wed Sep 20 01:54:36 2023 -0700
HADOOP-18873. ABFS: AbfsOutputStream doesnt close DataBlocks object.
(#6010)
AbfsOutputStream to close the dataBlock object created for the upload.
Contributed By: Pranav Saxena
> ABFS: AbfsOutputStream doesnt close DataBlocks object.
> ------------------------------------------------------
>
> Key: HADOOP-18873
> URL: https://issues.apache.org/jira/browse/HADOOP-18873
> Project: Hadoop Common
> Issue Type: Sub-task
> Affects Versions: 3.3.4
> Reporter: Pranav Saxena
> Assignee: Pranav Saxena
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.4
>
>
> AbfsOutputStream doesnt close the dataBlock object created for the upload.
> What is the implication of not doing that:
> DataBlocks has three implementations:
> # ByteArrayBlock
> ## This creates an object of DataBlockByteArrayOutputStream (child of
> ByteArrayOutputStream: wrapper arround byte-arrray for populating, reading
> the array.
> ## This gets GCed.
> # ByteBufferBlock:
> ## There is a defined *DirectBufferPool* from which it tries to request the
> directBuffer.
> ## If nothing in the pool, a new directBuffer is created.
> ## the `close` method on the this object has the responsiblity of returning
> back the buffer to pool so it can be reused.
> ## Since we are not calling the `close`:
> ### The pool is rendered of less use, since each request creates a new
> directBuffer from memory.
> ### All the object can be GCed and the direct-memory allocated may be
> returned on the GC. What if the process crashes, the memory never goes back
> and cause memory issue on the machine.
> # DiskBlock:
> ## This creates a file on disk on which the data-to-upload is written. This
> file gets deleted in startUpload().close().
>
> startUpload() gives an object of BlockUploadData which gives method of
> `toByteArray()` which is used in abfsOutputStream to get the byteArray in the
> dataBlock.
>
> Method which uses the DataBlock object:
> https://github.com/apache/hadoop/blob/fac7d26c5d7f791565cc3ab45d079e2cca725f95/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java#L298
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
