Thanx Hexiaoqiao for putting this up. As already discussed at the JIRA The Approach A sounds best to me.
-Ayush > On 11-Apr-2019, at 11:50 PM, Giovanni Matteo Fumarola > <giovanni.fumar...@gmail.com> wrote: > > +1 on Approach A. > >> On Thu, Apr 11, 2019 at 10:30 AM Iñigo Goiri <elgo...@gmail.com> wrote: >> >> Thanks Hexiaoqiao for starting the vote. >> As I said in the JIRA, I prefer Approach A. >> >> I wanted to bring a broader audience as this has changes in RBF, HDFS and >> Commons. >> I think adding a new optional field to the RPC header should be lightweight >> enough. >> The idea of passing a proxied client is already available in places like >> UGI but not to this level. >> I haven't been able to figure other uses but maybe other applications could >> take advantage of this new field. >> >> Please, raise any concerns regarding any of the 3 approaches proposed. >> >> On Wed, Apr 10, 2019 at 11:53 PM Akira Ajisaka <aajis...@apache.org> >> wrote: >> >>> The Approach A looks good to me. >>> >>> Thanks, >>> Akira >>> >>>> On Thu, Apr 11, 2019 at 2:30 PM Xiaoqiao He <xq.he2...@gmail.com> wrote: >>>> >>>> Hi forks, >>>> >>>> The current implementation of RBF is not sensitive about data locality, >>>> since NameNode could not get real client hostname by invoke >>>> Server#getRemoteAddress when RPC request forward by Router to NameNode. >>>> Therefore, it will lead to several challenges, for instance, >>>> >>>> - a. Client could have to go for remote read instead of local read, >>>> Short-Circuit could not be used in most cases. >>>> - b. Block placement policy could not run as except based on defined >>>> rack aware. Thus it will loss local node write. >>>> >>>> There are some different solutions to solve data locality issue after >>>> discussion, some of them will change RPC protocol, so we look forward >> to >>>> furthermore suggestions and votes. HDFS-13248 is tracking the issue. >>>> >>>> - Approach A: Changing IPC/RPC layer protocol >>> (IpcConnectionContextProto >>>> or RpcHeader#RpcRequestHeaderProto) and add extra field about client >>>> hostname. Of course the new field is optional, only input by Router >>> and >>>> parse by Namenode in generally. This approach is compatibility and >>> Client >>>> should do nothing after changing. >>>> - Approach B: Changing ClientProtocol and add extra interface >>>> create/append/getBlockLocations with additional parameter about >> client >>>> hostname. As approach A, it is input by Router and parse by >> Namenode, >>> and >>>> also is compatibility. >>>> - Approach C: Solve write and read locality separately based on >>> current >>>> interface and no changes, for write, hack client hostname as one of >>> favor >>>> nodes for addBlocks, for read, reorder targets at Router after >>> Namenode >>>> returns result to Router. >>>> >>>> As discussion and evaluation in HDFS-13248, we prefer to change IPC/RPC >>>> layer protocol to support RPC data locality. We welcome more >> suggestions, >>>> votes or just give us feedback to push forward this feature. Thanks. >>>> >>>> Best Regards, >>>> Hexiaoqiao >>>> >>>> reference >>>> [1] https://issues.apache.org/jira/browse/HDFS-13248 >>>> [2] https://issues.apache.org/jira/browse/HDFS-10467 >>>> >>>> [3] https://issues.apache.org/jira/browse/HDFS-12615 >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org >>> >>> >> --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
