[
https://issues.apache.org/jira/browse/HDFS-14162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16728537#comment-16728537
]
Konstantin Shvachko edited comment on HDFS-14162 at 12/24/18 10:08 PM:
-----------------------------------------------------------------------
Looks like [~xkrogen]'s and [~csun]'s approaches are similar in creating
ClientProtocol(s) along with another one requested.
I think a cleaner way would be to allow proxies on multiple protocols. I don't
like the common pattern now in the code that different services instantiate a
separate proxy for each protocol they need. Like Balancer uses two protocols
{{ClientProtocol}} and {{NameNodeProtocol}} and creates a proxy for each of
them. It would be good to be able to declare
{code:java}
public interface BalancerProtocols extends NamenodeProtocol, ClientProtocol {}
{code}
And then call {{createProxy(conf, nameNodeUri, BalancerProtocols.class)}} to
get a combined proxy. We can do it now by defining
{{BalancerProtocols*TranslatorPB}} classes. But the translations are already
defined so they should just be combined into something like
{{CombinerProtocolTranslator}}, which would use individual translators for each
protocol that {{BalancerProtocols}} extends. In this case
{{ClientNamenodeProtocolTranslatorPB}} and {{NamenodeProtocolTranslatorPB}}.
Then the translator protocols, in this case {{ClientNamenodeProtocolPB}} and
{{NamenodeProtocolPB}} could be combined into a single proxy by
{code:java}
Proxy.newProxyInstance(protocol.getClassLoader(),
new Class[]{ClientNamenodeProtocolPB.class, NamenodeProtocolPB.class},
invoker);
{code}
Currently all {{getProxy()}} methods in {{RpcEngine}} classes use only one
protocol class:
{code:java}
Proxy.newProxyInstance(protocol.getClassLoader(), new Class[]{protocol},
invoker)
{code}
which should be generalized for multiple protocols.
Hope this makes sense. It is a general improvement for Hadoop RPC. If we can
implement it we will be able to use it anywhere where proxies with multiple
protocols are needed. Recent examples include Balancer, Fsck,
NNThroughputBenchmark, and I've seen more cases.
was (Author: shv):
Looks like [~xkrogen]'s and [~csun]'s approaches are similar in creating
ClientProtocol(s) along with another one requested.
I think a cleaner way would be to allow proxies on multiple protocols. I don't
like the common pattern now in the code that different services instantiate a
separate proxy for each protocol they need. Like Balancer uses two protocols
{{ClientProtocol}} and {{NameNodeProtocol}} and creates a proxy for each of
them. It would be good to be able to declare
{code:java}
public interface BalancerProtocols extends NamenodeProtocol, ClientProtocol {}
{code}
And then call {{createProxy(conf, nameNodeUri, BalancerProtocols.class)}} to
get a combined proxy. We can do it now by defining
{{BalancerProtocols*TranslatorPB}} classes. But the translations are already
defined so they should just be combined into something like
{{CombinerProtocolTranslator}}, which would use individual translators for each
protocol that {{BalancerProtocols}} extends.
Then the translator protocols, in this case {{ClientNamenodeProtocolPB}} and
{{NamenodeProtocolPB}} could be combined into a single proxy by
{code:java}
Proxy.newProxyInstance(protocol.getClassLoader(),
new Class[]{ClientNamenodeProtocolPB.class, NamenodeProtocolPB.class},
invoker);
{code}
Currently all {{getProxy()}} methods in {{RpcEngine}} classes use only one
protocol class:
{code:java}
Proxy.newProxyInstance(protocol.getClassLoader(), new Class[]{protocol},
invoker)
{code}
which should be generalized for multiple protocols.
Hope this makes sense. It is a general improvement for Hadoop RPC. If we can
implement it we will be able to use it anywhere where proxies with multiple
protocols are needed. Recent examples include Balancer, Fsck,
NNThroughputBenchmark, and I've seen more cases.
> Balancer should work with ObserverNode
> --------------------------------------
>
> Key: HDFS-14162
> URL: https://issues.apache.org/jira/browse/HDFS-14162
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Konstantin Shvachko
> Priority: Major
> Attachments: HDFS-14162-HDFS-12943.wip0.patch,
> testBalancerWithObserver.patch
>
>
> Balancer provides a substantial RPC load on NameNode. It would be good to
> divert Balancer RPCs {{getBlocks()}}, etc. to ObserverNode. The main problem
> is that Balancer uses {{NamenodeProtocol}}, while ORPP currently supports
> only {{ClientProtocol}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
