[
https://issues.apache.org/jira/browse/HADOOP-18876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17761750#comment-17761750
]
Anmol Asrani edited comment on HADOOP-18876 at 9/4/23 9:40 AM:
---------------------------------------------------------------
Hi [[email protected]] , this is a summary of our test efforts: -
# {*}Concurrency Management for FileSystem{*}:
** Each FileSystem instance utilizes a BlockingThreadPoolExecutorService to
handle concurrency.
** We adjust parameters like
{{abfsConfiguration.getWriteMaxConcurrentRequestCount}} and
{{abfsConfiguration.getMaxWriteRequestsToQueue}} to control concurrent tasks.
# {*}Default Settings{*}:
** In cases where we don't specify
{*}{{fs.azure.write.max.concurrent.requests}}{*}, it defaults to four times the
number of available processors.
** In our setup, which has 8 processors per VM, this results in a maximum of
around 32 concurrent requests.
** Similarly, if we leave *{{fs.azure.write.max.requests.to.queue}}*
unspecified, it defaults to twice the value of
{{{}abfsConfiguration.getWriteMaxConcurrentRequestCount{}}}, which is
approximately 64 in our configuration.
# {*}Parallel FileSystem Instances{*}:
** With about 20 open FileSystem instances in our jobs, we've noticed that the
cap at the JVM level becomes roughly 20 times.
# {*}OutputStream and Block Uploads{*}:
** The default setting for {{*fs.azure.block.upload.active.blocks*}} is 20,
meaning each OutputStream level can handle around 20 blocks for upload.
# {*}Hardware and Configuration{*}:
** Our setup includes 40 worker nodes, each equipped with around 64 GB of RAM.
** We conducted 10-15 Spark TPCDS workload runs with various file sizes
ranging from 1 TB to 100 TB.
** Importantly, we didn't encounter any Out-of-Memory (OOM) errors, and we
experienced significant improvements in latency.
**
# {*}When Disk Can Result in OOM{*}:
** Parameters: -
*** {{fs.azure.write.max.concurrent.requests}} (m): Controls concurrent write
requests.
*** {{fs.azure.write.max.requests.to.queue}} (n ): Sets the max queued
requests.
*** {{FileSystem instances on a JVM}} (k): Concurrent filesystem objects on a
JVM.
*** {{fs.azure.write.request.size}} (writeSize): Default is 8MB per request.
*** {{fs.azure.block.upload.active.blocks}} (z): Limits queued blocks per
OutputStream.
** OOM issues with disk can arise when we set
{{fs.azure.write.max.concurrent.requests}} (m) to a high value.
** Even with disk usage, a large number of concurrent requests (m) can lead to
memory consumption issues.
** The formula for calculating the maximum memory consumed by AbfsOutputStream
is *k * writeSize * m.*
** High values of m can potentially deplete available memory resources,
resulting in OOM errors when using disk storage.
was (Author: JIRAUSER281089):
Hi [[email protected]] , this is a summary of our test efforts: -
# {*}Concurrency Management for FileSystem{*}:
** Each FileSystem instance utilizes a BlockingThreadPoolExecutorService to
handle concurrency.
** We adjust parameters like
{{abfsConfiguration.getWriteMaxConcurrentRequestCount}} and
{{abfsConfiguration.getMaxWriteRequestsToQueue}} to control concurrent tasks.
# {*}Default Settings{*}:
** In cases where we don't specify
{*}{{fs.azure.write.max.concurrent.requests}}{*}, it defaults to four times the
number of available processors.
** In our setup, which has 8 processors per VM, this results in a maximum of
around 32 concurrent requests.
** Similarly, if we leave *{{fs.azure.write.max.requests.to.queue}}*
unspecified, it defaults to twice the value of
{{{}abfsConfiguration.getWriteMaxConcurrentRequestCount{}}}, which is
approximately 64 in our configuration.
# {*}Parallel FileSystem Instances{*}:
** With about 20 open FileSystem instances in our jobs, we've noticed that the
cap at the JVM level becomes roughly 20 times.
# {*}OutputStream and Block Uploads{*}:
** The default setting for {{*fs.azure.block.upload.active.blocks*}} is 20,
meaning each OutputStream level can handle around 20 blocks for upload.
# {*}Hardware and Configuration{*}:
** Our setup includes 40 worker nodes, each equipped with around 64 GB of RAM.
** We conducted 10-15 Spark TPCDS workload runs with various file sizes
ranging from 1 TB to 100 TB.
** Importantly, we didn't encounter any Out-of-Memory (OOM) errors, and we
experienced significant improvements in latency.
**
# {*}When Disk Can Result in OOM{*}:
** Parameters: -
*** {{fs.azure.write.max.concurrent.requests}} (m): Controls concurrent write
requests.
*** {{fs.azure.write.max.requests.to.queue}} (n): Sets the max queued requests.
*** {{FileSystem instances on a JVM}} (k): Concurrent filesystem objects on a
JVM.
*** {{fs.azure.write.request.size}} (writeSize): Default is 8MB per request.
*** {{fs.azure.block.upload.active.blocks}} (z): Limits queued blocks per
OutputStream.
** OOM issues with disk can arise when we set
{{fs.azure.write.max.concurrent.requests}} (m) to a high value.
** Even with disk usage, a large number of concurrent requests (m) can lead to
memory consumption issues.
** The formula for calculating the maximum memory consumed by AbfsOutputStream
is *k * writeSize * m.*
** High values of m can potentially deplete available memory resources,
resulting in OOM errors when using disk storage.
> ABFS: Change default from disk to bytebuffer for fs.azure.data.blocks.buffer
> ----------------------------------------------------------------------------
>
> Key: HADOOP-18876
> URL: https://issues.apache.org/jira/browse/HADOOP-18876
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
> Affects Versions: 3.3.6
> Reporter: Anmol Asrani
> Assignee: Anmol Asrani
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.6
>
>
> Change default from disk to bytebuffer for fs.azure.data.blocks.buffer.
> Gathered from multiple workload runs, the presented data underscores a
> noteworthy enhancement in performance. The adoption of ByteBuffer for
> *reading operations* exhibited a remarkable improvement of approximately
> *64.83%* when compared to traditional disk-based reading. Similarly, the
> implementation of ByteBuffer for *write operations* yielded a substantial
> efficiency gain of about {*}60.75%{*}. These findings underscore the
> consistent and substantial advantages of integrating ByteBuffer across a
> range of workload scenarios.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
