Michael Dagaev wrote:
Thank you for the answer. Some more questions ...
Do you know why Hadoop RPC does not allow more than one connection per JVM?
We could make the hadoop RCP client non-blocking. That'd help. Might
need to also chunk send/receive when sizes are large so the send/receive
of big data blocks so access to socket was more fair.
Would suggest you try as is and get some numbers to identify first how
much current implementation is hampering throughput.
Can we increase this limit without changing the code?
No. Would take some work.
Can we use another transport (Thrift, REST) which does not have such a
limitation?
Would be easier doing above suggested modifications but yes, could also
do the above, though would take some work. Currently thrift and REST
implementations are servers that host an instance of the hbase client
and use it to do the client-server communication so would have the same
bottleneck. Would have to make thrift into first-class rpc client of
hbase. You are not the first to make this suggestion. See the issues
database for current suggestions in this regard.
St.Ack
