[ 
https://issues.apache.org/jira/browse/HDFS-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16728537#comment-16728537
 ] 

Konstantin Shvachko edited comment on HDFS-14162 at 12/24/18 10:08 PM:
-----------------------------------------------------------------------

Looks like [~xkrogen]'s and [~csun]'s approaches are similar in creating 
ClientProtocol(s) along with another one requested.
 I think a cleaner way would be to allow proxies on multiple protocols. I don't 
like the common pattern now in the code that different services instantiate a 
separate proxy for each protocol they need. Like Balancer uses two protocols 
{{ClientProtocol}} and {{NameNodeProtocol}} and creates a proxy for each of 
them. It would be good to be able to declare
{code:java}
public interface BalancerProtocols extends NamenodeProtocol, ClientProtocol {}
{code}
And then call {{createProxy(conf, nameNodeUri, BalancerProtocols.class)}} to 
get a combined proxy. We can do it now by defining 
{{BalancerProtocols*TranslatorPB}} classes. But the translations are already 
defined so they should just be combined into something like 
{{CombinerProtocolTranslator}}, which would use individual translators for each 
protocol that {{BalancerProtocols}} extends. In this case 
{{ClientNamenodeProtocolTranslatorPB}} and {{NamenodeProtocolTranslatorPB}}.
 Then the translator protocols, in this case {{ClientNamenodeProtocolPB}} and 
{{NamenodeProtocolPB}} could be combined into a single proxy by
{code:java}
Proxy.newProxyInstance(protocol.getClassLoader(),
    new Class[]{ClientNamenodeProtocolPB.class, NamenodeProtocolPB.class}, 
invoker);
{code}
Currently all {{getProxy()}} methods in {{RpcEngine}} classes use only one 
protocol class:
{code:java}
Proxy.newProxyInstance(protocol.getClassLoader(), new Class[]{protocol}, 
invoker)
{code}
which should be generalized for multiple protocols.
 Hope this makes sense. It is a general improvement for Hadoop RPC. If we can 
implement it we will be able to use it anywhere where proxies with multiple 
protocols are needed. Recent examples include Balancer, Fsck, 
NNThroughputBenchmark, and I've seen more cases.


was (Author: shv):
Looks like [~xkrogen]'s and [~csun]'s approaches are similar in creating 
ClientProtocol(s) along with another one requested.
 I think a cleaner way would be to allow proxies on multiple protocols. I don't 
like the common pattern now in the code that different services instantiate a 
separate proxy for each protocol they need. Like Balancer uses two protocols 
{{ClientProtocol}} and {{NameNodeProtocol}} and creates a proxy for each of 
them. It would be good to be able to declare
{code:java}
public interface BalancerProtocols extends NamenodeProtocol, ClientProtocol {}
{code}
And then call {{createProxy(conf, nameNodeUri, BalancerProtocols.class)}} to 
get a combined proxy. We can do it now by defining 
{{BalancerProtocols*TranslatorPB}} classes. But the translations are already 
defined so they should just be combined into something like 
{{CombinerProtocolTranslator}}, which would use individual translators for each 
protocol that {{BalancerProtocols}} extends.
 Then the translator protocols, in this case {{ClientNamenodeProtocolPB}} and 
{{NamenodeProtocolPB}} could be combined into a single proxy by
{code:java}
Proxy.newProxyInstance(protocol.getClassLoader(),
    new Class[]{ClientNamenodeProtocolPB.class, NamenodeProtocolPB.class}, 
invoker);
{code}
Currently all {{getProxy()}} methods in {{RpcEngine}} classes use only one 
protocol class:
{code:java}
Proxy.newProxyInstance(protocol.getClassLoader(), new Class[]{protocol}, 
invoker)
{code}
which should be generalized for multiple protocols.
 Hope this makes sense. It is a general improvement for Hadoop RPC. If we can 
implement it we will be able to use it anywhere where proxies with multiple 
protocols are needed. Recent examples include Balancer, Fsck, 
NNThroughputBenchmark, and I've seen more cases.

> Balancer should work with ObserverNode
> --------------------------------------
>
>                 Key: HDFS-14162
>                 URL: https://issues.apache.org/jira/browse/HDFS-14162
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Konstantin Shvachko
>            Priority: Major
>         Attachments: HDFS-14162-HDFS-12943.wip0.patch, 
> testBalancerWithObserver.patch
>
>
> Balancer provides a substantial RPC load on NameNode. It would be good to 
> divert Balancer RPCs {{getBlocks()}}, etc. to ObserverNode. The main problem 
> is that Balancer uses {{NamenodeProtocol}}, while ORPP currently supports 
> only {{ClientProtocol}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to