[
https://issues.apache.org/jira/browse/HDFS-16107?focusedWorklogId=652574&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-652574
]
ASF GitHub Bot logged work on HDFS-16107:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Sep/21 08:23
Start Date: 18/Sep/21 08:23
Worklog Time Spent: 10m
Work Description: jianghuazhu commented on a change in pull request #3170:
URL: https://github.com/apache/hadoop/pull/3170#discussion_r711552586
##########
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
##########
@@ -3192,23 +3196,41 @@ protected Server(String bindAddress, int port,
if (queueSizePerHandler != -1) {
this.maxQueueSize = handlerCount * queueSizePerHandler;
} else {
- this.maxQueueSize = handlerCount * conf.getInt(
- CommonConfigurationKeys.IPC_SERVER_HANDLER_QUEUE_SIZE_KEY,
- CommonConfigurationKeys.IPC_SERVER_HANDLER_QUEUE_SIZE_DEFAULT);
+ this.maxQueueSize = conf.getInt(getQueueClassPrefix() + "." +
+ CommonConfigurationKeys.SERVER_HANDLER_QUEUE_SIZE_KEY, 0);
+ if (this.maxQueueSize < 1) {
+ this.maxQueueSize = handlerCount * conf.getInt(
+ CommonConfigurationKeys.IPC_SERVER_HANDLER_QUEUE_SIZE_KEY,
+ CommonConfigurationKeys.IPC_SERVER_HANDLER_QUEUE_SIZE_DEFAULT);
+ }
+ }
+ int tmpMaxRespSize = conf.getInt(getQueueClassPrefix() + "." +
+ CommonConfigurationKeys.SERVER_RPC_MAX_RESPONSE_SIZE_KEY, 0);
+ if (tmpMaxRespSize < 1) {
+ this.maxRespSize = conf.getInt(
+ CommonConfigurationKeys.IPC_SERVER_RPC_MAX_RESPONSE_SIZE_KEY,
+ CommonConfigurationKeys.IPC_SERVER_RPC_MAX_RESPONSE_SIZE_DEFAULT);
+ } else {
+ this.maxRespSize = tmpMaxRespSize;
}
- this.maxRespSize = conf.getInt(
- CommonConfigurationKeys.IPC_SERVER_RPC_MAX_RESPONSE_SIZE_KEY,
- CommonConfigurationKeys.IPC_SERVER_RPC_MAX_RESPONSE_SIZE_DEFAULT);
if (numReaders != -1) {
this.readThreads = numReaders;
} else {
- this.readThreads = conf.getInt(
- CommonConfigurationKeys.IPC_SERVER_RPC_READ_THREADS_KEY,
- CommonConfigurationKeys.IPC_SERVER_RPC_READ_THREADS_DEFAULT);
+ this.readThreads = conf.getInt(getQueueClassPrefix() + "." +
+ CommonConfigurationKeys.SERVER_RPC_READ_THREADS_KEY, 0);
Review comment:
Thanks @tomscut for the comment.
If there is an RPC service whose port is 8020, we hope that the number of
read thread pools allocated for this RPC (ipc.8020.server.read.threadpool.size)
is different from other RPCs.
The logic here is:
1. First judge whether ipc.8020.server.read.threadpool.size has been
allocated. If it is not allocated, the value we get should be 0, indicating
that we want to use the public unified configuration
(ipc.server.read.threadpool .size).
2. If it is not allocated, we should use the public unified configuration
ipc.server.read.threadpool.size.
At present, CommonConfigurationKeys.IPC_SERVER_RPC_READ_THREADS_DEFAULT is
set to 1. If we use this attribute, we can still get a valid value when
ipc.8020.server.read.threadpool.size is not set, which may cause a Ambiguity.
This is my idea, welcome to continue to communicate. @tomscut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 652574)
Time Spent: 3.5h (was: 3h 20m)
> Split RPC configuration to isolate RPC
> --------------------------------------
>
> Key: HDFS-16107
> URL: https://issues.apache.org/jira/browse/HDFS-16107
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: JiangHua Zhu
> Assignee: JiangHua Zhu
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> For RPC of different ports, there are some common configurations, such as:
> ipc.server.read.threadpool.size
> ipc.server.read.connection-queue.size
> ipc.server.handler.queue.size
> Once we configure these values, it will affect all requests (including client
> and requests within the cluster).
> It is necessary for us to split these configurations to adapt to different
> ports, such as:
> ipc.8020.server.read.threadpool.size
> ipc.8021.server.read.threadpool.size
> ipc.8020.server.read.connection-queue.size
> ipc.8021.server.read.connection-queue.size
> The advantage of this is to isolate the RPC to deal with the pressure of
> requests from all sides.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
