[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13917020#comment-13917020 ] Hudson commented on HADOOP-10278: - SUCCESS: Integrated in Hadoop-Yarn-trunk #496 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/496/]) Update target version to 2.4.0 for HADOOP-10278 and HADOOP-10285 in CHANGES.txt (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1573070) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Fix For: 3.0.0, 2.4.0 Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13917062#comment-13917062 ] Hudson commented on HADOOP-10278: - FAILURE: Integrated in Hadoop-Hdfs-trunk #1688 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1688/]) Update target version to 2.4.0 for HADOOP-10278 and HADOOP-10285 in CHANGES.txt (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1573070) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Fix For: 3.0.0, 2.4.0 Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13917079#comment-13917079 ] Hudson commented on HADOOP-10278: - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1713 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1713/]) Update target version to 2.4.0 for HADOOP-10278 and HADOOP-10285 in CHANGES.txt (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1573070) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Fix For: 3.0.0, 2.4.0 Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13916374#comment-13916374 ] Hudson commented on HADOOP-10278: - SUCCESS: Integrated in Hadoop-trunk-Commit #5248 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5248/]) Update target version to 2.4.0 for HADOOP-10278 and HADOOP-10285 in CHANGES.txt (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1573070) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Fix For: 3.0.0, 2.4.0 Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13909334#comment-13909334 ] Hudson commented on HADOOP-10278: - SUCCESS: Integrated in Hadoop-Yarn-trunk #489 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/489/]) HADOOP-10278. Refactor to make CallQueue pluggable. (Contributed by Chris Li) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1570703) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Fix For: 3.0.0, 2.5.0 Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13909372#comment-13909372 ] Hudson commented on HADOOP-10278: - FAILURE: Integrated in Hadoop-Hdfs-trunk #1681 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1681/]) HADOOP-10278. Refactor to make CallQueue pluggable. (Contributed by Chris Li) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1570703) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Fix For: 3.0.0, 2.5.0 Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13909402#comment-13909402 ] Hudson commented on HADOOP-10278: - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1706 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1706/]) HADOOP-10278. Refactor to make CallQueue pluggable. (Contributed by Chris Li) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1570703) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Fix For: 3.0.0, 2.5.0 Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13908770#comment-13908770 ] Chris Li commented on HADOOP-10278: --- [~ikeda] I think that could be explored at a later time; for right now I think it's good enough that we don't degrade performance rather than seeking to improve it. [~arpitagarwal] let me know if there's anything else you need before committing. Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13908809#comment-13908809 ] Chris Li commented on HADOOP-10278: --- Awesome, thanks all! The next patch in the series is https://issues.apache.org/jira/browse/HADOOP-10285 It adds the admin interface to trigger a queue swap at runtime. Further discussion can take place there. Thanks Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Fix For: 3.0.0, 2.5.0 Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13908820#comment-13908820 ] Hudson commented on HADOOP-10278: - SUCCESS: Integrated in Hadoop-trunk-Commit #5208 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5208/]) HADOOP-10278. Refactor to make CallQueue pluggable. (Contributed by Chris Li) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1570703) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/CallQueueManager.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestCallQueueManager.java Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Fix For: 3.0.0, 2.5.0 Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13907323#comment-13907323 ] Arpit Agarwal commented on HADOOP-10278: The latest patch looks good for a first cut implementation. If there are no objections by tonight PST I will commit it. Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13905836#comment-13905836
]
Arpit Agarwal commented on HADOOP-10278:
Thanks Chris. Couple of comments and I apologize if these have been discussed
already and I missed it.
Overall the patch looks good. Would like to understand if there are any
alternative use cases.
# Is this the only use case (admin changes the call queue at runtime for a
specific port)? Would it be useful to have a default global setting to refresh
all ports without specifying the port number?
# Will it be made available for other daemons apart from NN?
Also in {{refreshCallQueue}} perhaps you can skip the call to {{swapQueue}} if
{{queueClassToUse}} has not changed.
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Assignee: Chris Li
Attachments: HADOOP-10278-atomicref-adapter.patch,
HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch,
HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch,
HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch,
HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch,
HADOOP-10278.patch, HADOOP-10278.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13905983#comment-13905983 ] Chris Li commented on HADOOP-10278: --- Thanks for checking out the patch, Arapit Actually, the admin can only refresh the callQueue on the namenode today. This is the only case I've targeted, since the namenode can't be easily restarted. Thinking of what daemons might benefit: * Namenode is what swapping was made for, since the NN can't be restarted to refresh the queue so easily * Datanodes, nodemanagers can be restarted at will, so it's not as useful * Resourcemanager maybe, but I'm not sure whether QoS on the RM solves a problem yet Also, we allow swaps between the same queue, since the user may adjust features of the queue (today that might be queue size, later on it would be qos parameters) at runtime. Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13905997#comment-13905997 ] Arpit Agarwal commented on HADOOP-10278: Thanks for the clarification. I am +1 for the patch. [~daryn] and [~jingzhao] were looking at this too so I will hold off on committing in case they have comments. Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13906423#comment-13906423 ] Hiroshi Ikeda commented on HADOOP-10278: [~chrilisf] What do you think about implementing my sketch previously I wrote and comparing results by way of experiment, if you have time? I believe using semaphores + read-write lock + a concurrent queue is as good as (or better than) using an atomic reference + a blocking queue, especially with multiple threads which have possibility to collide. bq. Also, in the tests, the atomic members aren't being written by multiple threads–they're atomic so that I can read them from another thread without thread caching. I understand. I think adding some comment would be helpful because it is open to misunderstand at first glance. [~daryn] bq. I'm not sure I understand. Context switches are inevitable if the queue is empty. Under normal conditions the put/take lock reduces contention. I think LinkedBlockingQueue has more chances to block threads than you except. If multiple threads call at once LinkedBlockingQueue.put()/offer(), only one thread wins the put-side lock and the others are blocked causing context switches. The same goes for LinkedBlockingQueue.take()/poll(). Also, when a thread puts a element into an empty queue, it finally should get its take-side lock in order to send a signal to take-side, but it may be blocked because another thread is just taking an element (with non-trivial logic) within the take-side lock. The same goes for taking an element from a full queue. A context switch is quite heavy overhead so that it is not entirely off the mark that you can create 100 objects at the same cost. Reducing critical sections is one of the most important things to improve performance in multi-thread contexts, and AtomicReference etc. are well used to create non-blocking logic. So, I feel using AtomicReference instead of ReadWriteLock but sticking to LinkedBlockingQueue is not consistent. It makes even less sense that it requires bothersome swapping logic. bq. AtomicRef is just a wrapper around a volatile (by nature can't be optimized) with CAS methods. I agree AtomicReference is a wrapper in the Oracle (Sun) implementation, which internally uses sun.misc.Unsafe, and I think it depends on venders. Also I think VM can optimize volatile variables, similarly to native synchronizations (for example, eliminating unnecessary synchronizations with escape analysis), though I'm not sure for our context. Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13906451#comment-13906451 ] Chris Li commented on HADOOP-10278: --- Hey [~ikeda], the semaphore solution looks similar to my initial implementation, which used two re-entrant locks to handle blocking for a concurrent linked queue safely through their condition vars. Performance was not impacted either (in fact, performance was slightly better than java's LBQ in the RPC call benchmark). At the end of the day it added too much complexity in the way of code, so we dropped it for Daryn's suggestion. I suspect the semaphore solution would be the same way: it would make the code too complex for comfort. I can implement and test it, but I'd like to get [~daryn]'s input before I go off to build another big feature, since it would be a waste if it still doesn't solve the problem of code complexity. Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13906508#comment-13906508 ] Hiroshi Ikeda commented on HADOOP-10278: [~chrilisf] Quite different. A reentrant lock blocks other threads, and executing non-trivial tasks, such as accessing a queue, within the lock increases chance of the blocks. That seems logically similar to BlockingLinkedQueue. I think my sketch is simple and concrete enough, but anyway it is possible that the performance would not impact because of other factors. Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13904363#comment-13904363 ] Daryn Sharp commented on HADOOP-10278: -- Up front, my requirement is I don't want a feature that will be rarely used to add additional lock contention during normal operation. I'm -1 to more locking. bq. LinkedBlockingQueue separates its lock into only two locks, put side and get side, so that congesion in each side causes context switches. Moreover the separation of the lock is ineffective when there are almost always empty in the queue, and collision between put and get causes context switches. I'm not sure I understand. Context switches are inevitable if the queue is empty. Under normal conditions the put/take lock reduces contention. bq. An issue arises when production consumption: I thought about that case with the original analysis but I didn't want the post to be too big. If the NN is under heavy load, the callq is full, and the user uses a new queue is of a much smaller size - I'd argue that's a case of user error. The draining thread will compete with new calls but should eventually complete. bq. The atomic reference is not needed for the queue and volatile is enough if you only set and get. Volatile variables may have much chance to be optimized by VM because volatile is within the language specification. AtomicRef is just a wrapper around a volatile (by nature can't be optimized) with CAS methods. I like the fact AtomicRef provides an obvious hint that the value is going to be changed, plus we should be using CAS to swap the queue ref to prevent races. --- So how about something like this rough idea: # CallQueueManager (or whatever it's called) maintains a putRef and takeRef, initially set to the same queue # for a swap, use CAS to update the putRef with the new callq so readers begin filling it - note it's not yet being serviced by handlers # now wait for handlers to drain all the calls in takeRef's old callq (1) - periodically check isEmpty()? # finally update takeRef to point to the new callq (1)There's a race where readers may drop in 1 more call after the callq swapped. Using isEmpty() is imperfect due to the race. I dislike time-based solutions, but if handlers are polling for a few seconds, that should be plenty of time to get straggler calls before the handlers switch to the new/current queue. Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13904494#comment-13904494 ] Chris Li commented on HADOOP-10278: --- bq. plus we should be using CAS to swap the queue ref to prevent races Just for clarity: currently CallQueueManager.swapQueue isn't threadsafe, since the Server's calling method is synchronized. Are we trying to make swapQueue threadsafe, or are we trying to be even safer to make sure puts don't get left behind? If it's the later only, we can continue to use set() instead of CAS. Making swapQueue concurrent sounds like more hassle than it's worth. Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13904802#comment-13904802
]
Hadoop QA commented on HADOOP-10278:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629661/HADOOP-10278-atomicref-adapter.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.ipc.TestCallQueueManager
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3588//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3588//console
This message is automatically generated.
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Attachments: HADOOP-10278-atomicref-adapter.patch,
HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch,
HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch,
HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch,
HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13904878#comment-13904878
]
Hadoop QA commented on HADOOP-10278:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629682/HADOOP-10278-atomicref-adapter.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3589//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3589//console
This message is automatically generated.
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Assignee: Chris Li
Attachments: HADOOP-10278-atomicref-adapter.patch,
HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch,
HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch,
HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch,
HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch,
HADOOP-10278.patch, HADOOP-10278.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13904950#comment-13904950
]
Arpit Agarwal commented on HADOOP-10278:
Hi Chris, Who calls {{refreshCallQueue}}?
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Assignee: Chris Li
Attachments: HADOOP-10278-atomicref-adapter.patch,
HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch,
HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch,
HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch,
HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch,
HADOOP-10278.patch, HADOOP-10278.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13904964#comment-13904964 ] Chris Li commented on HADOOP-10278: --- Hi [~arpitagarwal], check out HADOOP-10285 It's called from the command line: hadoop dfsadmin -refreshCallQueue Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Assignee: Chris Li Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13903740#comment-13903740 ] Chris Li commented on HADOOP-10278: --- [~ikeda] Thanks for taking a look. bq. Although ReadWriteLock is much more complex than simple locks and is not prefered to enclose trival logic, I suspect it is not severe overhead compared to LinkedBlockingQueue. One of the issues with RWLock I found was that we'd have to use offer(timeout) instead of put(), or else the readlock may not be released. This makes the queue swap lock for about a second, and causes a huge backup in the process. Otherwise the overhead of the lock wasn't much, it's mainly the offer(timeout). bq. By the way, about the patch, nobody can eliminate the possibility that a blocked thread at LinkedBlockingQueue.put() cannot wake up in 1 seconds when another thread drains. At least you should check the reference after put(), such as We could increase the timeout period on the queue swap transfer too. bq. (I also worry about the order of elements is not preserved in spite of the name queue.) Order doesn't matter to the client, since the client cannot expect to send two consecutive commands without receiving a response from the first, and expect order that they're executed in order, even today. This is why we can even do QoS by re-ordering rpc calls in the first place. As far as the server goes, I guess this will have to be something we're aware of. bq. Still, implementation of size() is invalid. Are you referring to CallQueueManager.size(), due to the possibility that the queue is being swapped? I'm assuming that size() isn't being used for anything besides metrics, so a small discrepancy during swapping would be okay. bq. The atomic reference is not needed for the queue and volatile is enough if you only set and get. Volatile variables may have much chance to be optimized by VM because volatile is within the language specification. On the other hand, increment operation (++) is not atomic for volatile variables, so some of the test classes should be changed. Agreed, though I think [~daryn] says he prefers atomicref for clarity. I could go either way. Also, in the tests, the atomic members aren't being written by multiple threads–they're atomic so that I can read them from another thread without thread caching. Even if they incremented atomically, there's still the issue of the put happening before the increment, so the tests only check after the operations complete (using Thread.sleeps) Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13902965#comment-13902965
]
Hiroshi Ikeda commented on HADOOP-10278:
Although ReadWriteLock is much more complex than simple locks and is not
prefered to enclose trival logic, I suspect it is not severe overhead compared
to LinkedBlockingQueue.
Anybody seems not to worry about using LinkedBlockingQueue, but
LinkedBlockingQueue separates its lock into only two locks, put side and get
side, so that congesion in each side causes context switches. Moreover the
separation of the lock is ineffective when there are almost always empty in the
queue, and collision between put and get causes context switches.
By the way, about the patch, nobody can eliminate the possibility that a
blocked thread at LinkedBlockingQueue.put() cannot wake up in 1 seconds when
another thread drains. At least you should check the reference after put(),
such as
{code}
void put(e) {
q = queue.get();
q.put(e);
while((q2 = queue.get()) != q) {
drain(q, q2);
q = q2;
}
}
{code}
(I also worry about the order of elements is not preserved in spite of the name
queue.)
Still, implementation of size() is invalid.
The atomic reference is not needed for the queue and volatile is enough if you
only set and get. Volatile variables may have much chance to be optimized by VM
because volatile is within the language specification. On the other hand,
increment operation (++) is not atomic for volatile variables, so some of the
test classes should be changed.
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Attachments: HADOOP-10278-atomicref-adapter.patch,
HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch,
HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch,
HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch,
HADOOP-10278.patch, HADOOP-10278.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13902326#comment-13902326
]
Hadoop QA commented on HADOOP-10278:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629198/HADOOP-10278-atomicref-adapter.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3583//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3583//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3583//console
This message is automatically generated.
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Attachments: HADOOP-10278-atomicref-adapter.patch,
HADOOP-10278-atomicref-adapter.patch, HADOOP-10278-atomicref-rwlock.patch,
HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch,
HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch,
HADOOP-10278.patch, HADOOP-10278.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13898423#comment-13898423 ] Chris Li commented on HADOOP-10278: --- Not sure why that is, I can re-run that test with different timeouts. Right now I'm exploring using a ReadWriteLock to protect against that race. As an aside, I went back to get benchmarking results for the FIFOCallQueue and found that it has no significant difference against trunk for throughput and client, but a 2% reduction in server cpu (p=.0055) Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13891192#comment-13891192
]
Jing Zhao commented on HADOOP-10278:
One nit: in the latest patch, do we still need the String and configuration
parameter?
{code}
+ private BlockingQueueCall createCallQueueInstance(Class theClass, int
maxLen,
+String ns, Configuration conf) {
{code}
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Attachments: HADOOP-10278-atomicref.patch, HADOOP-10278.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13891220#comment-13891220 ] Daryn Sharp commented on HADOOP-10278: -- I need to look at the latest patch, but comments on the comments: bq. The performance hit will probably increase with more handler threads too Aside: With my background cycles (which is about nil), HADOOP-10300 should hopefully be able to reduce the number of handlers yet improve performance. Fewer threads also mean less contention on the callq and hopefully even better performance. I've got a POC but haven't stressed or benched it yet. bq. Even though the server CPU time increased, the throughput wasn't really affected An impact to throughput would be a concern, but cpu utilization isn't (yet) much of a concern. I'm trying to get the NN to actually use more than a few cores under heavy load. bq. You can get a small performance gain using volatile instead of AtomicRef I'm curious if that's true. I'd expect a warmed up JVM to have effectively inlined the call. Personally I prefer AtomicReference to volatile if for no other reason than it's explicit to a dev that something about the ref is magical, but if the impact is measurable I would be swayed. Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13891222#comment-13891222 ] Chris Li commented on HADOOP-10278: --- [~daryn] I'm working on a new set of benchmarks to test atomicref vs volatile, should have results later today Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13891232#comment-13891232
]
Hadoop QA commented on HADOOP-10278:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12626977/HADOOP-10278-atomicref.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3532//console
This message is automatically generated.
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Attachments: HADOOP-10278-atomicref.patch,
HADOOP-10278-atomicref.patch, HADOOP-10278.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13891666#comment-13891666
]
Hadoop QA commented on HADOOP-10278:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12627050/HADOOP-10278-atomicref.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3537//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3537//console
This message is automatically generated.
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Attachments: HADOOP-10278-atomicref.patch,
HADOOP-10278-atomicref.patch, HADOOP-10278-atomicref.patch, HADOOP-10278.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13889757#comment-13889757
]
Daryn Sharp commented on HADOOP-10278:
--
(Just noticed I didn't resubmit this comment after jira was having issues last
week)
I see. I'm a bit uneasy about the maintenance and reduced flexibility from a
custom concrete queue impl. I haven't thought this through, but would it be
feasible to do a swapout by using an atomic ref for the callq and using offer
and poll with timeouts? Ex. a handler would replace {{call =
callQueue.take()}} with:
{code}
Call call;
do {
call = callQueue.get().poll(100L, TimeUnit.MILLISECONDS));
} while (call == null);
{code}
Refreshing the queue would create a new queue, swap it into the atomic ref,
then drain calls in the prior queue and add to the new queue. Handlers would
consume at most 1 call during the swap, and if they block on an empty queue the
above code will cause them to switch over. Just a thought.
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Attachments: HADOOP-10278.patch, subtask1.3.patch, subtask1.4.patch,
subtask1.5.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13889798#comment-13889798 ] Chris Li commented on HADOOP-10278: --- That's an interesting thought, I do like that it would work with any BlockingQueue. I'll give it a try see how it benchmarks too. Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: HADOOP-10278.patch, subtask1.3.patch, subtask1.4.patch, subtask1.5.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13889817#comment-13889817
]
Hadoop QA commented on HADOOP-10278:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12626710/HADOOP-10278.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3523//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3523//console
This message is automatically generated.
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Attachments: HADOOP-10278.patch, subtask1.3.patch, subtask1.4.patch,
subtask1.5.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13890056#comment-13890056 ] Chris Li commented on HADOOP-10278: --- [~daryn] Implemented and benchmarked it. All around less clutter and custom locking required. I do like this solution. Benchmarked performance: using RPCCallBenchmark -s 4 -c 10 Average server time increased by 1.5% (0.05 p value, 1-tailed/2-sample-equal-variance t test) Other results were insignificant If the performance hit is acceptable, I think this solution is easier, and I can re-submit this patch. - Raw results Group calls/s Server CPU Client CPU Poll32356.0 48018 47837 Poll31497.0 46129 44052 Poll32456.0 46400 43643 Poll31253.0 46343 44152 Poll31572.0 46241 44123 Take30831.0 46521 45315 Take31764.0 45549 43827 Take32534.0 45949 44028 Take32153.0 45699 44215 Take31455.0 45823 43926 T Tests: Throughput 0.419851632864817 Server CPU 0.0509472764018302 Client CPU 0.279864716054668 Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: HADOOP-10278.patch, subtask1.3.patch, subtask1.4.patch, subtask1.5.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13890095#comment-13890095 ] Daryn Sharp commented on HADOOP-10278: -- Given my current focus on performance, it saddens me that the get+poll would be noticeable. Did you try tinkering with the poll time? Perhaps some number of seconds is a reasonable time for handlers to switch over to a new queue since I would expect that queue swapping would be an infrequent event. Please go ahead and post the patch. It's really the combination of performance and simplicity/maintainability that should be the determining factors. Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: HADOOP-10278.patch, subtask1.3.patch, subtask1.4.patch, subtask1.5.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13890260#comment-13890260 ] Benoy Antony commented on HADOOP-10278: --- You can get a small performance gain using _volatile_ instead of _AtomicRef_ since this case does not require the guarantee of AtomicRef's compareAndSwap. Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: HADOOP-10278-atomicref.patch, HADOOP-10278.patch, subtask1.3.patch, subtask1.4.patch, subtask1.5.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13890319#comment-13890319
]
Hadoop QA commented on HADOOP-10278:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12626790/HADOOP-10278-atomicref.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3525//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3525//console
This message is automatically generated.
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Attachments: HADOOP-10278-atomicref.patch, HADOOP-10278.patch,
subtask1.3.patch, subtask1.4.patch, subtask1.5.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13888764#comment-13888764 ] Benoy Antony commented on HADOOP-10278: --- Could you please generate the patch with --no-prefix ? There are a few additional white spaces in the patch. Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: subtask1.3.patch, subtask1.4.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13888785#comment-13888785
]
Hadoop QA commented on HADOOP-10278:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12626501/subtask1.5.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3522//console
This message is automatically generated.
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Attachments: subtask1.3.patch, subtask1.4.patch, subtask1.5.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13888293#comment-13888293 ] Benoy Antony commented on HADOOP-10278: --- [~chrili] All the methods in _CallQueue_ are already in _BlockingQueue_. Do we really need to define a new interface -_CallQueue_ ? Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: subtask1.3.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13888305#comment-13888305 ] Chris Li commented on HADOOP-10278: --- I've done so for flexibility–the later patch https://issues.apache.org/jira/browse/HADOOP-10302 adds methods to CallQueue necessary for swapping a queue at runtime. It's also convenient to be able to check if a call queue is a custom call queue vs a LinkedBlockingQueue, without requiring the callqueue to extend BaseCallQueue Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: subtask1.3.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13888352#comment-13888352 ] Benoy Antony commented on HADOOP-10278: --- If it is a marker interface for now, then can we remove the redudant methods from CallQueue ? Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: subtask1.3.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13888374#comment-13888374 ] Chris Li commented on HADOOP-10278: --- Sounds good, I'll update the patch Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: subtask1.3.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13888390#comment-13888390
]
Hadoop QA commented on HADOOP-10278:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12626436/subtask1.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3516//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3516//console
This message is automatically generated.
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Attachments: subtask1.3.patch, subtask1.4.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13884494#comment-13884494 ] Chris Li commented on HADOOP-10278: --- Hi [~daryn], Could you take a look please? Thanks, Chris Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: subtask1.2.patch, subtask1.3.patch, subtask1.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13884698#comment-13884698 ] Daryn Sharp commented on HADOOP-10278: -- Upon quick glance it looks much cleaner. Questions: # What advantage will the custom fifo call queue offer over a standard java queue? # Instead of the new base class providing a concrete queue implementation (locking and all), is it possible for the custom queues to using a containing relationship with a standard java queue? Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: subtask1.2.patch, subtask1.3.patch, subtask1.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13884720#comment-13884720 ] Chris Li commented on HADOOP-10278: --- Thanks for checking it out. 1. In a later patch I'll be introducing the ability to swap the call queue at runtime, which is essential for performance tuning without restarting the namenode. The FIFOCallQueue responds to methods needed to accomplish this transparently to the server. 2. I would have preferred this myself (and earlier versions did this), but we will need control of the locks to do runtime queue swapping. In any case, the FairCallQueue (which should be coming in subtask5) will use this same locking code, so refactoring it into the base makes things cleaner. I will start uploading the other subtask patches Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: subtask1.2.patch, subtask1.3.patch, subtask1.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[ https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13884803#comment-13884803 ] Chris Li commented on HADOOP-10278: --- I've uploaded a patch for https://issues.apache.org/jira/browse/HADOOP-10302 which should make some of the design decisions in this patch more clear. Refactor to make CallQueue pluggable Key: HADOOP-10278 URL: https://issues.apache.org/jira/browse/HADOOP-10278 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Chris Li Attachments: subtask1.2.patch, subtask1.3.patch, subtask1.patch * Refactor CallQueue into an interface, base, and default implementation that matches today's behavior * Make the call queue impl configurable, keyed on port so that we minimize coupling -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13883545#comment-13883545
]
Hadoop QA commented on HADOOP-10278:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12625474/subtask1.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1555 javac
compiler warnings (more than the trunk's current 1546 warnings).
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3482//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3482//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3482//console
This message is automatically generated.
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Attachments: subtask1.2.patch, subtask1.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13883623#comment-13883623
]
Hadoop QA commented on HADOOP-10278:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12625491/subtask1.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3484//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3484//console
This message is automatically generated.
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Attachments: subtask1.2.patch, subtask1.3.patch, subtask1.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10278) Refactor to make CallQueue pluggable
[
https://issues.apache.org/jira/browse/HADOOP-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13881772#comment-13881772
]
Hadoop QA commented on HADOOP-10278:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12625168/subtask1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1559 javac
compiler warnings (more than the trunk's current 1546 warnings).
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3469//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3469//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3469//console
This message is automatically generated.
Refactor to make CallQueue pluggable
Key: HADOOP-10278
URL: https://issues.apache.org/jira/browse/HADOOP-10278
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Chris Li
Attachments: subtask1.patch
* Refactor CallQueue into an interface, base, and default implementation that
matches today's behavior
* Make the call queue impl configurable, keyed on port so that we minimize
coupling
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
