[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335203#comment-15335203
]
Hudson commented on HADOOP-13189:
-
SUCCESS: Integrated in Hadoop-trunk-Commit #9974 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9974/])
HADOOP-13189. FairCallQueue makes callQueue larger than the configured (shv:
rev a2a5cb60b09491cb672978ba9442f02373392c67)
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestFairCallQueue.java
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/FairCallQueue.java
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch, HADOOP-13189.002.patch,
> HADOOP-13189.003.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332608#comment-15332608
]
Ming Ma commented on HADOOP-13189:
--
Sounds good. [~shv] please go ahead and thank you.
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch, HADOOP-13189.002.patch,
> HADOOP-13189.003.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332540#comment-15332540
]
Konstantin Shvachko commented on HADOOP-13189:
--
What do you think?
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch, HADOOP-13189.002.patch,
> HADOOP-13189.003.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332537#comment-15332537
]
Konstantin Shvachko commented on HADOOP-13189:
--
_I think it is a bug fix._ Current meaning of call queue size for FairCallQueue
contradicts documentation.
The refreshCallQueue is a good example. You had a standard queue of size 1000.
Then you switch to FairCallQueue using refreshCallQueue command, and suddenly
increase the queue size to 4000. I don't think this is expected. It is
definitely not documented.
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch, HADOOP-13189.002.patch,
> HADOOP-13189.003.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330927#comment-15330927
]
Ming Ma commented on HADOOP-13189:
--
[~shv], the reason I asked is refreshCallQueue should switch to the new call
queue right away while draining existing calls. Just to confirm, this patch
does change the meaning of call queue size for FairCallQueue(it was meant for
sub queue, but now it is for the aggregated size). It really depends on how you
interpret it, it could be a backward compatibility issue, or just bug fix. What
is your take?
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch, HADOOP-13189.002.patch,
> HADOOP-13189.003.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330660#comment-15330660
]
Konstantin Shvachko commented on HADOOP-13189:
--
[~mingma] do you suggest anything or should I commit it?
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch, HADOOP-13189.002.patch,
> HADOOP-13189.003.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328677#comment-15328677
]
Konstantin Shvachko commented on HADOOP-13189:
--
We did not use refreshCallQueue. Just configured FairCallQueue via
{{callqueue.impl}}
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch, HADOOP-13189.002.patch,
> HADOOP-13189.003.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325511#comment-15325511
]
Ming Ma commented on HADOOP-13189:
--
Thanks Konstantin. Did you use
https://issues.apache.org/jira/browse/HADOOP-10285 to change call CallQueue?
bq. when all requests go only into the high priority queue one should probably
switch to the default queue as FairCallQueue functionality is not utilized.
This only happens at a given point of time, with FairCallQueue is useful in
other times on the same NN.
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch, HADOOP-13189.002.patch,
> HADOOP-13189.003.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325015#comment-15325015
]
Konstantin Shvachko commented on HADOOP-13189:
--
Hey [~mingma] good to hear from you. I should have given a bit more context in
the description. Doing it now.
So we are trying to solve two inter-related issues here:
# Inconsistency between the config value of {{ipc.server.handler.queue.size}},
which is defined as _"How many calls per handler are allowed in the queue."_,
and the actual queue size when {{FairCallQueue}} is used, which lets 4 times
more.
# A need to reconfigure the cluster when switching from one CallQueue
implementation to another.
The latter is important to prevent NN meltdown. Before we learned about this
issue we saw too many clients connecting to the NameNode and making it
unresponsive. This is related to read requests like {{listStatus}} and
{{getFileInfo}}. And it worked fine with the default queue, but not with
FairCallQueue. Looking deeper it was noticed that NN allowed more requests with
FairCallQueue than with the default. We had to reconfigure NN and adjust the
queue size, but now we have problem 1. Hope this makes sense.
With your example when all requests go only into the high priority queue one
should probably switch to the default queue as FairCallQueue functionality is
not utilized.
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch, HADOOP-13189.002.patch,
> HADOOP-13189.003.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322007#comment-15322007
]
Ming Ma commented on HADOOP-13189:
--
Thanks [~shv], [~redvine]. That seems reasonable. Although in actual setup, we
haven't run into any issue or we have accounted for the factor. Another thing,
if RPC backoff is enabled and all users fall under highest priority queue,
equal divide means smaller capacity for the highest priority queue and thus got
backoff sooner than the configured queue size.
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch, HADOOP-13189.002.patch,
> HADOOP-13189.003.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321780#comment-15321780
]
Hadoop QA commented on HADOOP-13189:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 65 unchanged - 1 fixed = 65 total (was 66) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 55s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12809031/HADOOP-13189.003.patch
|
| JIRA Issue | HADOOP-13189 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 95aa22aee728 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1500a0a |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9700/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9700/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9700/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
>
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321767#comment-15321767
]
Konstantin Shvachko commented on HADOOP-13189:
--
+1 on the patch.
I restarted jenkins build for this patch. Will commit once it is finished.
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch, HADOOP-13189.002.patch,
> HADOOP-13189.003.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321449#comment-15321449
]
Hadoop QA commented on HADOOP-13189:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
5s{color} | {color:red} Docker failed to build yetus/hadoop:2c91fd8. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12809031/HADOOP-13189.003.patch
|
| JIRA Issue | HADOOP-13189 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9694/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch, HADOOP-13189.002.patch,
> HADOOP-13189.003.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321439#comment-15321439
]
Vinitha Reddy Gankidi commented on HADOOP-13189:
Added another patch (003) that resolves the checkstyle warning.
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch, HADOOP-13189.002.patch,
> HADOOP-13189.003.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321193#comment-15321193
]
Konstantin Shvachko commented on HADOOP-13189:
--
Looks like you need to split the comment line in {{TestFairCallQueue.java}}
into two as per checkstyle warning.
Test failures do not seem to be related to the patch.
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch, HADOOP-13189.002.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320166#comment-15320166
]
Hadoop QA commented on HADOOP-13189:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
27s{color} | {color:red} hadoop-common-project/hadoop-common: The patch
generated 1 new + 65 unchanged - 1 fixed = 66 total (was 66) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 5s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
| | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808854/HADOOP-13189.002.patch
|
| JIRA Issue | HADOOP-13189 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 23759cb9f6f5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 76f0800 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9687/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9687/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9687/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9687/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FairCallQueue makes callQueue larger than the configured capacity.
>
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320012#comment-15320012
]
Vinitha Reddy Gankidi commented on HADOOP-13189:
[~xyao] Thanks for the review. The 2nd parameter of the FairCallQueue
constructor is the per queue capacity in the current implementation. However,
this patch makes it the total capacity of all subqueues. So, this change is
needed.
It would be better to keep the capacity allocation to the subqueues flexible.
Instead of validating the internal subqueue capacity allocation, I will add a
test that validates that the total capacity of all subqueues equals the
maxQueueSize.
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317957#comment-15317957
]
Xiaoyu Yao commented on HADOOP-13189:
-
Thanks for the heads up [~arpiagariu]. The code change looks good. I'm not
sure whether the unit test change is needed as the 2nd parameter of the
FairCallQueue constructor is already based on per queue capacity. I would
suggest we test with mock (e.g., mockito) to validate internal subqueue
capacity allocation given different {{ipc.server.handler.queue.size}}.
{code}
-fcq = new FairCallQueue(2, 5, "ns", conf);
+fcq = new FairCallQueue(2, 10, "ns", conf);
{code}
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317628#comment-15317628
]
Arpit Agarwal commented on HADOOP-13189:
cc some folks who may be interested in FairCallQueue as FYI [~chrilisf],
[~mingma], [~xyao].
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317611#comment-15317611
]
Hadoop QA commented on HADOOP-13189:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
5s{color} | {color:red} Docker failed to build yetus/hadoop:2c91fd8. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808426/HADOOP-13189.001.patch
|
| JIRA Issue | HADOOP-13189 |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9672/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316800#comment-15316800
]
Vinitha Reddy Gankidi commented on HADOOP-13189:
Attached a patch. Please review.
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>Assignee: Vinitha Reddy Gankidi
> Attachments: HADOOP-13189.001.patch
>
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13189) FairCallQueue makes callQueue larger than the configured capacity.
[
https://issues.apache.org/jira/browse/HADOOP-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294394#comment-15294394
]
Konstantin Shvachko commented on HADOOP-13189:
--
This is actually in IPC code base, so moved under Hadoop Common.
> FairCallQueue makes callQueue larger than the configured capacity.
> --
>
> Key: HADOOP-13189
> URL: https://issues.apache.org/jira/browse/HADOOP-13189
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
>Affects Versions: 2.6.0
>Reporter: Konstantin Shvachko
>
> {{FairCallQueue}} divides {{callQueue}} into multiple (4 by default)
> sub-queues, with each sub-queue corresponding to a different level of
> priority. The constructor for {{FairCallQueue}} takes the same parameter
> {{capacity}} as the default CallQueue implementation, and allocates all its
> sub-queues of size {{capacity}}. With 4 levels of priority (sub-queues) by
> default it results in the total callQueue size 4 times larger than it should
> be based on the configuration.
> {{capacity}} should be divided by the number of sub-queues at some place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
