[
https://issues.apache.org/jira/browse/HADOOP-19737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18040328#comment-18040328
]
ASF GitHub Bot commented on HADOOP-19737:
-----------------------------------------
anmolanmol1234 commented on code in PR #8056:
URL: https://github.com/apache/hadoop/pull/8056#discussion_r2556297999
##########
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ReadBufferManagerV2.java:
##########
@@ -831,41 +861,79 @@ private boolean manualEviction(final ReadBuffer buf) {
*/
private void adjustThreadPool() {
int currentPoolSize = workerRefs.size();
- double cpuLoad = getCpuLoad();
+ double cpuLoad = getJvmCpuLoad();
+ if (cpuLoad > maxCpuUtilization) {
+ maxCpuUtilization = cpuLoad;
+ }
int requiredPoolSize = getRequiredThreadPoolSize();
int newThreadPoolSize;
printTraceLog(
"Current CPU load: {}, Current worker pool size: {}, Current queue
size: {}",
cpuLoad, currentPoolSize, requiredPoolSize);
if (currentPoolSize < requiredPoolSize && cpuLoad < cpuThreshold) {
+ lastScaleDirection = "I";
// Submit more background tasks.
newThreadPoolSize = Math.min(maxThreadPoolSize,
(int) Math.ceil(
(currentPoolSize * (HUNDRED_D + threadPoolUpscalePercentage))
/ HUNDRED_D));
+ if (newThreadPoolSize == maxThreadPoolSize) {
+ lastScaleDirection = "+F"; // Already full, cannot scale up
+ } else {
+ lastScaleDirection = "I"; // Normal scale-up
+ }
// Create new Worker Threads
- for (int i = currentPoolSize; i < newThreadPoolSize; i++) {
- ReadBufferWorker worker = new ReadBufferWorker(i, getBufferManager());
- workerRefs.add(worker);
- workerPool.submit(worker);
+ if ("I".equals(lastScaleDirection)) {
+ for (int i = currentPoolSize; i < newThreadPoolSize; i++) {
+ ReadBufferWorker worker = new ReadBufferWorker(i,
+ getBufferManager(abfsClient));
+ workerRefs.add(worker);
+ workerPool.submit(worker);
+ }
}
+ // Capture the latest thread pool statistics (pool size, CPU, memory,
etc.)
+ ReadThreadPoolStats stats = getCurrentStats(cpuLoad, maxCpuUtilization);
Review Comment:
these are needed only when we are scaling up or down not always hence added
in blocks where scale up or down is happening
> ABFS: Add metrics to identify improvements with read and write aggressiveness
> -----------------------------------------------------------------------------
>
> Key: HADOOP-19737
> URL: https://issues.apache.org/jira/browse/HADOOP-19737
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.5.0, 3.4.2
> Reporter: Anmol Asrani
> Assignee: Anmol Asrani
> Priority: Major
> Labels: pull-request-available
>
> Introduces new performance metrics in the ABFS driver to monitor and evaluate
> the effectiveness of read and write aggressiveness tuning. These metrics help
> in understanding how thread pool behavior, CPU utilization, and heap
> availability impact overall I/O throughput and latency. By capturing detailed
> statistics such as active thread count, pool size, and system resource
> utilization, this enhancement enables data-driven analysis of optimizations
> made to improve ABFS read and write performance under varying workloads.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
