[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-11552: - Fix Version/s: 3.0.0-alpha2 > Allow handoff on the server side for RPC requests > - > > Key: HADOOP-11552 > URL: https://issues.apache.org/jira/browse/HADOOP-11552 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: HADOOP-11552.05.patch, HADOOP-11552.06.patch, > HADOOP-11552.07.patch, HADOOP-11552.08.patch, HADOOP-11552.1.wip.txt, > HADOOP-11552.2.txt, HADOOP-11552.3.txt, HADOOP-11552.3.txt, HADOOP-11552.4.txt > > > An RPC server handler thread is tied up for each incoming RPC request. This > isn't ideal, since this essentially implies that RPC operations should be > short lived, and most operations which could take time end up falling back to > a polling mechanism. > Some use cases where this is useful. > - YARN submitApplication - which currently submits, followed by a poll to > check if the application is accepted while the submit operation is written > out to storage. This can be collapsed into a single call. > - YARN allocate - requests and allocations use the same protocol. New > allocations are received via polling. > The allocate protocol could be split into a request/heartbeat along with a > 'awaitResponse'. The request/heartbeat is sent only when there's a request or > on a much longer heartbeat interval. awaitResponse is always left active with > the RM - and returns the moment something is available. > MapReduce/Tez task to AM communication is another example of this pattern. > The same pattern of splitting calls can be used for other protocols as well. > This should serve to improve latency, as well as reduce network traffic since > the keep-alive heartbeat can be sent less frequently. > I believe there's some cases in HDFS as well, where the DN gets told to > perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated HADOOP-11552: - Target Version/s: 2.9.0 (was: 2.8.0) Fix Version/s: (was: 2.8.0) 2.9.0 > Allow handoff on the server side for RPC requests > - > > Key: HADOOP-11552 > URL: https://issues.apache.org/jira/browse/HADOOP-11552 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.9.0 > > Attachments: HADOOP-11552.05.patch, HADOOP-11552.06.patch, > HADOOP-11552.07.patch, HADOOP-11552.08.patch, HADOOP-11552.1.wip.txt, > HADOOP-11552.2.txt, HADOOP-11552.3.txt, HADOOP-11552.3.txt, HADOOP-11552.4.txt > > > An RPC server handler thread is tied up for each incoming RPC request. This > isn't ideal, since this essentially implies that RPC operations should be > short lived, and most operations which could take time end up falling back to > a polling mechanism. > Some use cases where this is useful. > - YARN submitApplication - which currently submits, followed by a poll to > check if the application is accepted while the submit operation is written > out to storage. This can be collapsed into a single call. > - YARN allocate - requests and allocations use the same protocol. New > allocations are received via polling. > The allocate protocol could be split into a request/heartbeat along with a > 'awaitResponse'. The request/heartbeat is sent only when there's a request or > on a much longer heartbeat interval. awaitResponse is always left active with > the RM - and returns the moment something is available. > MapReduce/Tez task to AM communication is another example of this pattern. > The same pattern of splitting calls can be used for other protocols as well. > This should serve to improve latency, as well as reduce network traffic since > the keep-alive heartbeat can be sent less frequently. > I believe there's some cases in HDFS as well, where the DN gets told to > perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated HADOOP-11552: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Committed to trunk and branch-2, thanks [~sseth] ! Thanks [~asuresh] for reviewing the patch ! > Allow handoff on the server side for RPC requests > - > > Key: HADOOP-11552 > URL: https://issues.apache.org/jira/browse/HADOOP-11552 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.8.0 > > Attachments: HADOOP-11552.05.patch, HADOOP-11552.06.patch, > HADOOP-11552.07.patch, HADOOP-11552.08.patch, HADOOP-11552.1.wip.txt, > HADOOP-11552.2.txt, HADOOP-11552.3.txt, HADOOP-11552.3.txt, HADOOP-11552.4.txt > > > An RPC server handler thread is tied up for each incoming RPC request. This > isn't ideal, since this essentially implies that RPC operations should be > short lived, and most operations which could take time end up falling back to > a polling mechanism. > Some use cases where this is useful. > - YARN submitApplication - which currently submits, followed by a poll to > check if the application is accepted while the submit operation is written > out to storage. This can be collapsed into a single call. > - YARN allocate - requests and allocations use the same protocol. New > allocations are received via polling. > The allocate protocol could be split into a request/heartbeat along with a > 'awaitResponse'. The request/heartbeat is sent only when there's a request or > on a much longer heartbeat interval. awaitResponse is always left active with > the RM - and returns the moment something is available. > MapReduce/Tez task to AM communication is another example of this pattern. > The same pattern of splitting calls can be used for other protocols as well. > This should serve to improve latency, as well as reduce network traffic since > the keep-alive heartbeat can be sent less frequently. > I believe there's some cases in HDFS as well, where the DN gets told to > perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated HADOOP-11552: - Labels: (was: BB2015-05-TBR) > Allow handoff on the server side for RPC requests > - > > Key: HADOOP-11552 > URL: https://issues.apache.org/jira/browse/HADOOP-11552 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.8.0 > > Attachments: HADOOP-11552.05.patch, HADOOP-11552.06.patch, > HADOOP-11552.07.patch, HADOOP-11552.08.patch, HADOOP-11552.1.wip.txt, > HADOOP-11552.2.txt, HADOOP-11552.3.txt, HADOOP-11552.3.txt, HADOOP-11552.4.txt > > > An RPC server handler thread is tied up for each incoming RPC request. This > isn't ideal, since this essentially implies that RPC operations should be > short lived, and most operations which could take time end up falling back to > a polling mechanism. > Some use cases where this is useful. > - YARN submitApplication - which currently submits, followed by a poll to > check if the application is accepted while the submit operation is written > out to storage. This can be collapsed into a single call. > - YARN allocate - requests and allocations use the same protocol. New > allocations are received via polling. > The allocate protocol could be split into a request/heartbeat along with a > 'awaitResponse'. The request/heartbeat is sent only when there's a request or > on a much longer heartbeat interval. awaitResponse is always left active with > the RM - and returns the moment something is available. > MapReduce/Tez task to AM communication is another example of this pattern. > The same pattern of splitting calls can be used for other protocols as well. > This should serve to improve latency, as well as reduce network traffic since > the keep-alive heartbeat can be sent less frequently. > I believe there's some cases in HDFS as well, where the DN gets told to > perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated HADOOP-11552: - Status: Patch Available (was: Open) > Allow handoff on the server side for RPC requests > - > > Key: HADOOP-11552 > URL: https://issues.apache.org/jira/browse/HADOOP-11552 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Labels: BB2015-05-TBR > Attachments: HADOOP-11552.05.patch, HADOOP-11552.06.patch, > HADOOP-11552.07.patch, HADOOP-11552.08.patch, HADOOP-11552.1.wip.txt, > HADOOP-11552.2.txt, HADOOP-11552.3.txt, HADOOP-11552.3.txt, HADOOP-11552.4.txt > > > An RPC server handler thread is tied up for each incoming RPC request. This > isn't ideal, since this essentially implies that RPC operations should be > short lived, and most operations which could take time end up falling back to > a polling mechanism. > Some use cases where this is useful. > - YARN submitApplication - which currently submits, followed by a poll to > check if the application is accepted while the submit operation is written > out to storage. This can be collapsed into a single call. > - YARN allocate - requests and allocations use the same protocol. New > allocations are received via polling. > The allocate protocol could be split into a request/heartbeat along with a > 'awaitResponse'. The request/heartbeat is sent only when there's a request or > on a much longer heartbeat interval. awaitResponse is always left active with > the RM - and returns the moment something is available. > MapReduce/Tez task to AM communication is another example of this pattern. > The same pattern of splitting calls can be used for other protocols as well. > This should serve to improve latency, as well as reduce network traffic since > the keep-alive heartbeat can be sent less frequently. > I believe there's some cases in HDFS as well, where the DN gets told to > perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated HADOOP-11552: - Status: Open (was: Patch Available) > Allow handoff on the server side for RPC requests > - > > Key: HADOOP-11552 > URL: https://issues.apache.org/jira/browse/HADOOP-11552 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Labels: BB2015-05-TBR > Attachments: HADOOP-11552.05.patch, HADOOP-11552.06.patch, > HADOOP-11552.07.patch, HADOOP-11552.08.patch, HADOOP-11552.1.wip.txt, > HADOOP-11552.2.txt, HADOOP-11552.3.txt, HADOOP-11552.3.txt, HADOOP-11552.4.txt > > > An RPC server handler thread is tied up for each incoming RPC request. This > isn't ideal, since this essentially implies that RPC operations should be > short lived, and most operations which could take time end up falling back to > a polling mechanism. > Some use cases where this is useful. > - YARN submitApplication - which currently submits, followed by a poll to > check if the application is accepted while the submit operation is written > out to storage. This can be collapsed into a single call. > - YARN allocate - requests and allocations use the same protocol. New > allocations are received via polling. > The allocate protocol could be split into a request/heartbeat along with a > 'awaitResponse'. The request/heartbeat is sent only when there's a request or > on a much longer heartbeat interval. awaitResponse is always left active with > the RM - and returns the moment something is available. > MapReduce/Tez task to AM communication is another example of this pattern. > The same pattern of splitting calls can be used for other protocols as well. > This should serve to improve latency, as well as reduce network traffic since > the keep-alive heartbeat can be sent less frequently. > I believe there's some cases in HDFS as well, where the DN gets told to > perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated HADOOP-11552: - Attachment: HADOOP-11552.08.patch > Allow handoff on the server side for RPC requests > - > > Key: HADOOP-11552 > URL: https://issues.apache.org/jira/browse/HADOOP-11552 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Labels: BB2015-05-TBR > Attachments: HADOOP-11552.05.patch, HADOOP-11552.06.patch, > HADOOP-11552.07.patch, HADOOP-11552.08.patch, HADOOP-11552.1.wip.txt, > HADOOP-11552.2.txt, HADOOP-11552.3.txt, HADOOP-11552.3.txt, HADOOP-11552.4.txt > > > An RPC server handler thread is tied up for each incoming RPC request. This > isn't ideal, since this essentially implies that RPC operations should be > short lived, and most operations which could take time end up falling back to > a polling mechanism. > Some use cases where this is useful. > - YARN submitApplication - which currently submits, followed by a poll to > check if the application is accepted while the submit operation is written > out to storage. This can be collapsed into a single call. > - YARN allocate - requests and allocations use the same protocol. New > allocations are received via polling. > The allocate protocol could be split into a request/heartbeat along with a > 'awaitResponse'. The request/heartbeat is sent only when there's a request or > on a much longer heartbeat interval. awaitResponse is always left active with > the RM - and returns the moment something is available. > MapReduce/Tez task to AM communication is another example of this pattern. > The same pattern of splitting calls can be used for other protocols as well. > This should serve to improve latency, as well as reduce network traffic since > the keep-alive heartbeat can be sent less frequently. > I believe there's some cases in HDFS as well, where the DN gets told to > perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-11552: Attachment: HADOOP-11552.07.patch Thanks for taking a look, and pointing out the issue with setDeferredResponse. Modified to not attempt a sendResponse - that's what is done for non deferred calls as well (LOG and forget). > Allow handoff on the server side for RPC requests > - > > Key: HADOOP-11552 > URL: https://issues.apache.org/jira/browse/HADOOP-11552 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Labels: BB2015-05-TBR > Attachments: HADOOP-11552.05.patch, HADOOP-11552.06.patch, > HADOOP-11552.07.patch, HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt, > HADOOP-11552.3.txt, HADOOP-11552.3.txt, HADOOP-11552.4.txt > > > An RPC server handler thread is tied up for each incoming RPC request. This > isn't ideal, since this essentially implies that RPC operations should be > short lived, and most operations which could take time end up falling back to > a polling mechanism. > Some use cases where this is useful. > - YARN submitApplication - which currently submits, followed by a poll to > check if the application is accepted while the submit operation is written > out to storage. This can be collapsed into a single call. > - YARN allocate - requests and allocations use the same protocol. New > allocations are received via polling. > The allocate protocol could be split into a request/heartbeat along with a > 'awaitResponse'. The request/heartbeat is sent only when there's a request or > on a much longer heartbeat interval. awaitResponse is always left active with > the RM - and returns the moment something is available. > MapReduce/Tez task to AM communication is another example of this pattern. > The same pattern of splitting calls can be used for other protocols as well. > This should serve to improve latency, as well as reduce network traffic since > the keep-alive heartbeat can be sent less frequently. > I believe there's some cases in HDFS as well, where the DN gets told to > perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-11552: Attachment: HADOOP-11552.06.patch Revised to address review comments, and fix the findbugs warnings. > Allow handoff on the server side for RPC requests > - > > Key: HADOOP-11552 > URL: https://issues.apache.org/jira/browse/HADOOP-11552 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Labels: BB2015-05-TBR > Attachments: HADOOP-11552.05.patch, HADOOP-11552.06.patch, > HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt, HADOOP-11552.3.txt, > HADOOP-11552.3.txt, HADOOP-11552.4.txt > > > An RPC server handler thread is tied up for each incoming RPC request. This > isn't ideal, since this essentially implies that RPC operations should be > short lived, and most operations which could take time end up falling back to > a polling mechanism. > Some use cases where this is useful. > - YARN submitApplication - which currently submits, followed by a poll to > check if the application is accepted while the submit operation is written > out to storage. This can be collapsed into a single call. > - YARN allocate - requests and allocations use the same protocol. New > allocations are received via polling. > The allocate protocol could be split into a request/heartbeat along with a > 'awaitResponse'. The request/heartbeat is sent only when there's a request or > on a much longer heartbeat interval. awaitResponse is always left active with > the RM - and returns the moment something is available. > MapReduce/Tez task to AM communication is another example of this pattern. > The same pattern of splitting calls can be used for other protocols as well. > This should serve to improve latency, as well as reduce network traffic since > the keep-alive heartbeat can be sent less frequently. > I believe there's some cases in HDFS as well, where the DN gets told to > perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-11552: Attachment: HADOOP-11552.05.patch Rebased patch for trunk. This makes some changes for metrics as well. > Allow handoff on the server side for RPC requests > - > > Key: HADOOP-11552 > URL: https://issues.apache.org/jira/browse/HADOOP-11552 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Labels: BB2015-05-TBR > Attachments: HADOOP-11552.05.patch, HADOOP-11552.1.wip.txt, > HADOOP-11552.2.txt, HADOOP-11552.3.txt, HADOOP-11552.3.txt, HADOOP-11552.4.txt > > > An RPC server handler thread is tied up for each incoming RPC request. This > isn't ideal, since this essentially implies that RPC operations should be > short lived, and most operations which could take time end up falling back to > a polling mechanism. > Some use cases where this is useful. > - YARN submitApplication - which currently submits, followed by a poll to > check if the application is accepted while the submit operation is written > out to storage. This can be collapsed into a single call. > - YARN allocate - requests and allocations use the same protocol. New > allocations are received via polling. > The allocate protocol could be split into a request/heartbeat along with a > 'awaitResponse'. The request/heartbeat is sent only when there's a request or > on a much longer heartbeat interval. awaitResponse is always left active with > the RM - and returns the moment something is available. > MapReduce/Tez task to AM communication is another example of this pattern. > The same pattern of splitting calls can be used for other protocols as well. > This should serve to improve latency, as well as reduce network traffic since > the keep-alive heartbeat can be sent less frequently. > I believe there's some cases in HDFS as well, where the DN gets told to > perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11552: -- Labels: BB2015-05-TBR (was: ) Allow handoff on the server side for RPC requests - Key: HADOOP-11552 URL: https://issues.apache.org/jira/browse/HADOOP-11552 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Siddharth Seth Assignee: Siddharth Seth Labels: BB2015-05-TBR Attachments: HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt, HADOOP-11552.3.txt, HADOOP-11552.3.txt, HADOOP-11552.4.txt An RPC server handler thread is tied up for each incoming RPC request. This isn't ideal, since this essentially implies that RPC operations should be short lived, and most operations which could take time end up falling back to a polling mechanism. Some use cases where this is useful. - YARN submitApplication - which currently submits, followed by a poll to check if the application is accepted while the submit operation is written out to storage. This can be collapsed into a single call. - YARN allocate - requests and allocations use the same protocol. New allocations are received via polling. The allocate protocol could be split into a request/heartbeat along with a 'awaitResponse'. The request/heartbeat is sent only when there's a request or on a much longer heartbeat interval. awaitResponse is always left active with the RM - and returns the moment something is available. MapReduce/Tez task to AM communication is another example of this pattern. The same pattern of splitting calls can be used for other protocols as well. This should serve to improve latency, as well as reduce network traffic since the keep-alive heartbeat can be sent less frequently. I believe there's some cases in HDFS as well, where the DN gets told to perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-11552: Target Version/s: 2.8.0 (was: 2.7.0) Allow handoff on the server side for RPC requests - Key: HADOOP-11552 URL: https://issues.apache.org/jira/browse/HADOOP-11552 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt, HADOOP-11552.3.txt, HADOOP-11552.3.txt, HADOOP-11552.4.txt An RPC server handler thread is tied up for each incoming RPC request. This isn't ideal, since this essentially implies that RPC operations should be short lived, and most operations which could take time end up falling back to a polling mechanism. Some use cases where this is useful. - YARN submitApplication - which currently submits, followed by a poll to check if the application is accepted while the submit operation is written out to storage. This can be collapsed into a single call. - YARN allocate - requests and allocations use the same protocol. New allocations are received via polling. The allocate protocol could be split into a request/heartbeat along with a 'awaitResponse'. The request/heartbeat is sent only when there's a request or on a much longer heartbeat interval. awaitResponse is always left active with the RM - and returns the moment something is available. MapReduce/Tez task to AM communication is another example of this pattern. The same pattern of splitting calls can be used for other protocols as well. This should serve to improve latency, as well as reduce network traffic since the keep-alive heartbeat can be sent less frequently. I believe there's some cases in HDFS as well, where the DN gets told to perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-11552: Attachment: HADOOP-11552.3.txt Updated patch to fix the rat check and javac warning. I don't think the unit test failure is related. bq. I would like to see some code actually using this before we add it, to make sure we are getting the APIs right. [~cmccabe] - the APIs are being added as unstable for now. There's at least one follow up jira to make use a common thread pool to handle requests and send responses - or at least have the RPC layer provide a pool for responses. At the mement apps would have to setup threads to send responses in parallel. The test does show how this would be used in an application - where the specific method on a protocol would need to indicate intent to send a delayed response, and relinquish control. I think it'll be useful to get this in - so that it's possible to start trying out this mechanism. Allow handoff on the server side for RPC requests - Key: HADOOP-11552 URL: https://issues.apache.org/jira/browse/HADOOP-11552 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt, HADOOP-11552.3.txt, HADOOP-11552.3.txt An RPC server handler thread is tied up for each incoming RPC request. This isn't ideal, since this essentially implies that RPC operations should be short lived, and most operations which could take time end up falling back to a polling mechanism. Some use cases where this is useful. - YARN submitApplication - which currently submits, followed by a poll to check if the application is accepted while the submit operation is written out to storage. This can be collapsed into a single call. - YARN allocate - requests and allocations use the same protocol. New allocations are received via polling. The allocate protocol could be split into a request/heartbeat along with a 'awaitResponse'. The request/heartbeat is sent only when there's a request or on a much longer heartbeat interval. awaitResponse is always left active with the RM - and returns the moment something is available. MapReduce/Tez task to AM communication is another example of this pattern. The same pattern of splitting calls can be used for other protocols as well. This should serve to improve latency, as well as reduce network traffic since the keep-alive heartbeat can be sent less frequently. I believe there's some cases in HDFS as well, where the DN gets told to perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-11552: Status: Open (was: Patch Available) Allow handoff on the server side for RPC requests - Key: HADOOP-11552 URL: https://issues.apache.org/jira/browse/HADOOP-11552 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt, HADOOP-11552.3.txt, HADOOP-11552.3.txt, HADOOP-11552.4.txt An RPC server handler thread is tied up for each incoming RPC request. This isn't ideal, since this essentially implies that RPC operations should be short lived, and most operations which could take time end up falling back to a polling mechanism. Some use cases where this is useful. - YARN submitApplication - which currently submits, followed by a poll to check if the application is accepted while the submit operation is written out to storage. This can be collapsed into a single call. - YARN allocate - requests and allocations use the same protocol. New allocations are received via polling. The allocate protocol could be split into a request/heartbeat along with a 'awaitResponse'. The request/heartbeat is sent only when there's a request or on a much longer heartbeat interval. awaitResponse is always left active with the RM - and returns the moment something is available. MapReduce/Tez task to AM communication is another example of this pattern. The same pattern of splitting calls can be used for other protocols as well. This should serve to improve latency, as well as reduce network traffic since the keep-alive heartbeat can be sent less frequently. I believe there's some cases in HDFS as well, where the DN gets told to perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-11552: Status: Patch Available (was: Open) Allow handoff on the server side for RPC requests - Key: HADOOP-11552 URL: https://issues.apache.org/jira/browse/HADOOP-11552 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt, HADOOP-11552.3.txt, HADOOP-11552.3.txt, HADOOP-11552.4.txt An RPC server handler thread is tied up for each incoming RPC request. This isn't ideal, since this essentially implies that RPC operations should be short lived, and most operations which could take time end up falling back to a polling mechanism. Some use cases where this is useful. - YARN submitApplication - which currently submits, followed by a poll to check if the application is accepted while the submit operation is written out to storage. This can be collapsed into a single call. - YARN allocate - requests and allocations use the same protocol. New allocations are received via polling. The allocate protocol could be split into a request/heartbeat along with a 'awaitResponse'. The request/heartbeat is sent only when there's a request or on a much longer heartbeat interval. awaitResponse is always left active with the RM - and returns the moment something is available. MapReduce/Tez task to AM communication is another example of this pattern. The same pattern of splitting calls can be used for other protocols as well. This should serve to improve latency, as well as reduce network traffic since the keep-alive heartbeat can be sent less frequently. I believe there's some cases in HDFS as well, where the DN gets told to perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-11552: Attachment: HADOOP-11552.4.txt Allow handoff on the server side for RPC requests - Key: HADOOP-11552 URL: https://issues.apache.org/jira/browse/HADOOP-11552 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt, HADOOP-11552.3.txt, HADOOP-11552.3.txt, HADOOP-11552.4.txt An RPC server handler thread is tied up for each incoming RPC request. This isn't ideal, since this essentially implies that RPC operations should be short lived, and most operations which could take time end up falling back to a polling mechanism. Some use cases where this is useful. - YARN submitApplication - which currently submits, followed by a poll to check if the application is accepted while the submit operation is written out to storage. This can be collapsed into a single call. - YARN allocate - requests and allocations use the same protocol. New allocations are received via polling. The allocate protocol could be split into a request/heartbeat along with a 'awaitResponse'. The request/heartbeat is sent only when there's a request or on a much longer heartbeat interval. awaitResponse is always left active with the RM - and returns the moment something is available. MapReduce/Tez task to AM communication is another example of this pattern. The same pattern of splitting calls can be used for other protocols as well. This should serve to improve latency, as well as reduce network traffic since the keep-alive heartbeat can be sent less frequently. I believe there's some cases in HDFS as well, where the DN gets told to perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-11552: Status: Patch Available (was: Open) Allow handoff on the server side for RPC requests - Key: HADOOP-11552 URL: https://issues.apache.org/jira/browse/HADOOP-11552 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt An RPC server handler thread is tied up for each incoming RPC request. This isn't ideal, since this essentially implies that RPC operations should be short lived, and most operations which could take time end up falling back to a polling mechanism. Some use cases where this is useful. - YARN submitApplication - which currently submits, followed by a poll to check if the application is accepted while the submit operation is written out to storage. This can be collapsed into a single call. - YARN allocate - requests and allocations use the same protocol. New allocations are received via polling. The allocate protocol could be split into a request/heartbeat along with a 'awaitResponse'. The request/heartbeat is sent only when there's a request or on a much longer heartbeat interval. awaitResponse is always left active with the RM - and returns the moment something is available. MapReduce/Tez task to AM communication is another example of this pattern. The same pattern of splitting calls can be used for other protocols as well. This should serve to improve latency, as well as reduce network traffic since the keep-alive heartbeat can be sent less frequently. I believe there's some cases in HDFS as well, where the DN gets told to perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-11552: Attachment: HADOOP-11552.2.txt Updated patch with another couple of tests, and error handling. [~sanjay.radia], [~daryn] - please review. Would appreciate it if you could pay additional attention to ensuring this doesn't break the regular flow. I've tried to keep the changes to a minimal for the regular flow. Allow handoff on the server side for RPC requests - Key: HADOOP-11552 URL: https://issues.apache.org/jira/browse/HADOOP-11552 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt An RPC server handler thread is tied up for each incoming RPC request. This isn't ideal, since this essentially implies that RPC operations should be short lived, and most operations which could take time end up falling back to a polling mechanism. Some use cases where this is useful. - YARN submitApplication - which currently submits, followed by a poll to check if the application is accepted while the submit operation is written out to storage. This can be collapsed into a single call. - YARN allocate - requests and allocations use the same protocol. New allocations are received via polling. The allocate protocol could be split into a request/heartbeat along with a 'awaitResponse'. The request/heartbeat is sent only when there's a request or on a much longer heartbeat interval. awaitResponse is always left active with the RM - and returns the moment something is available. MapReduce/Tez task to AM communication is another example of this pattern. The same pattern of splitting calls can be used for other protocols as well. This should serve to improve latency, as well as reduce network traffic since the keep-alive heartbeat can be sent less frequently. I believe there's some cases in HDFS as well, where the DN gets told to perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-11552: Status: Open (was: Patch Available) Allow handoff on the server side for RPC requests - Key: HADOOP-11552 URL: https://issues.apache.org/jira/browse/HADOOP-11552 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt, HADOOP-11552.3.txt An RPC server handler thread is tied up for each incoming RPC request. This isn't ideal, since this essentially implies that RPC operations should be short lived, and most operations which could take time end up falling back to a polling mechanism. Some use cases where this is useful. - YARN submitApplication - which currently submits, followed by a poll to check if the application is accepted while the submit operation is written out to storage. This can be collapsed into a single call. - YARN allocate - requests and allocations use the same protocol. New allocations are received via polling. The allocate protocol could be split into a request/heartbeat along with a 'awaitResponse'. The request/heartbeat is sent only when there's a request or on a much longer heartbeat interval. awaitResponse is always left active with the RM - and returns the moment something is available. MapReduce/Tez task to AM communication is another example of this pattern. The same pattern of splitting calls can be used for other protocols as well. This should serve to improve latency, as well as reduce network traffic since the keep-alive heartbeat can be sent less frequently. I believe there's some cases in HDFS as well, where the DN gets told to perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-11552: Attachment: HADOOP-11552.3.txt Updated patch to apply cleanly to trunk. Allow handoff on the server side for RPC requests - Key: HADOOP-11552 URL: https://issues.apache.org/jira/browse/HADOOP-11552 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt, HADOOP-11552.3.txt An RPC server handler thread is tied up for each incoming RPC request. This isn't ideal, since this essentially implies that RPC operations should be short lived, and most operations which could take time end up falling back to a polling mechanism. Some use cases where this is useful. - YARN submitApplication - which currently submits, followed by a poll to check if the application is accepted while the submit operation is written out to storage. This can be collapsed into a single call. - YARN allocate - requests and allocations use the same protocol. New allocations are received via polling. The allocate protocol could be split into a request/heartbeat along with a 'awaitResponse'. The request/heartbeat is sent only when there's a request or on a much longer heartbeat interval. awaitResponse is always left active with the RM - and returns the moment something is available. MapReduce/Tez task to AM communication is another example of this pattern. The same pattern of splitting calls can be used for other protocols as well. This should serve to improve latency, as well as reduce network traffic since the keep-alive heartbeat can be sent less frequently. I believe there's some cases in HDFS as well, where the DN gets told to perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-11552: Status: Patch Available (was: Open) Allow handoff on the server side for RPC requests - Key: HADOOP-11552 URL: https://issues.apache.org/jira/browse/HADOOP-11552 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Siddharth Seth Assignee: Siddharth Seth Attachments: HADOOP-11552.1.wip.txt, HADOOP-11552.2.txt, HADOOP-11552.3.txt An RPC server handler thread is tied up for each incoming RPC request. This isn't ideal, since this essentially implies that RPC operations should be short lived, and most operations which could take time end up falling back to a polling mechanism. Some use cases where this is useful. - YARN submitApplication - which currently submits, followed by a poll to check if the application is accepted while the submit operation is written out to storage. This can be collapsed into a single call. - YARN allocate - requests and allocations use the same protocol. New allocations are received via polling. The allocate protocol could be split into a request/heartbeat along with a 'awaitResponse'. The request/heartbeat is sent only when there's a request or on a much longer heartbeat interval. awaitResponse is always left active with the RM - and returns the moment something is available. MapReduce/Tez task to AM communication is another example of this pattern. The same pattern of splitting calls can be used for other protocols as well. This should serve to improve latency, as well as reduce network traffic since the keep-alive heartbeat can be sent less frequently. I believe there's some cases in HDFS as well, where the DN gets told to perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11552) Allow handoff on the server side for RPC requests
[ https://issues.apache.org/jira/browse/HADOOP-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HADOOP-11552: Attachment: HADOOP-11552.1.wip.txt Work in progress patch to get input on the approach. Trying to make this as unobtrusive as possible so that APIs do not break. Ofcourse, changing YARN, HDFS, etc to use something like this will require a fair amount of attention to ensure compatibility. Individual methods to decide whether they want to perform the operation within the handler thread, or send the response back at a later point. Borrows elements from HADOOP-10300. There's several bits pending - like error handling. Also, this ends up sending the response back on the application thread which invokes the reply - which may not work for models like YARN where the central dispatcher would end up blocking. Ideally, we should be able to use the handler threads to send these responses. Alternately, with the current patch - a threadpool for sending responses could be setup. Additional considerations - number of outstanding requests, and whether they need to be limited at the RPC layer. Allow handoff on the server side for RPC requests - Key: HADOOP-11552 URL: https://issues.apache.org/jira/browse/HADOOP-11552 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Siddharth Seth Attachments: HADOOP-11552.1.wip.txt An RPC server handler thread is tied up for each incoming RPC request. This isn't ideal, since this essentially implies that RPC operations should be short lived, and most operations which could take time end up falling back to a polling mechanism. Some use cases where this is useful. - YARN submitApplication - which currently submits, followed by a poll to check if the application is accepted while the submit operation is written out to storage. This can be collapsed into a single call. - YARN allocate - requests and allocations use the same protocol. New allocations are received via polling. The allocate protocol could be split into a request/heartbeat along with a 'awaitResponse'. The request/heartbeat is sent only when there's a request or on a much longer heartbeat interval. awaitResponse is always left active with the RM - and returns the moment something is available. MapReduce/Tez task to AM communication is another example of this pattern. The same pattern of splitting calls can be used for other protocols as well. This should serve to improve latency, as well as reduce network traffic since the keep-alive heartbeat can be sent less frequently. I believe there's some cases in HDFS as well, where the DN gets told to perform some operations when they heartbeat into the NN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
