+1 on Approach A. On Thu, Apr 11, 2019 at 10:30 AM Iñigo Goiri <elgo...@gmail.com> wrote:
> Thanks Hexiaoqiao for starting the vote. > As I said in the JIRA, I prefer Approach A. > > I wanted to bring a broader audience as this has changes in RBF, HDFS and > Commons. > I think adding a new optional field to the RPC header should be lightweight > enough. > The idea of passing a proxied client is already available in places like > UGI but not to this level. > I haven't been able to figure other uses but maybe other applications could > take advantage of this new field. > > Please, raise any concerns regarding any of the 3 approaches proposed. > > On Wed, Apr 10, 2019 at 11:53 PM Akira Ajisaka <aajis...@apache.org> > wrote: > > > The Approach A looks good to me. > > > > Thanks, > > Akira > > > > On Thu, Apr 11, 2019 at 2:30 PM Xiaoqiao He <xq.he2...@gmail.com> wrote: > > > > > > Hi forks, > > > > > > The current implementation of RBF is not sensitive about data locality, > > > since NameNode could not get real client hostname by invoke > > > Server#getRemoteAddress when RPC request forward by Router to NameNode. > > > Therefore, it will lead to several challenges, for instance, > > > > > > - a. Client could have to go for remote read instead of local read, > > > Short-Circuit could not be used in most cases. > > > - b. Block placement policy could not run as except based on defined > > > rack aware. Thus it will loss local node write. > > > > > > There are some different solutions to solve data locality issue after > > > discussion, some of them will change RPC protocol, so we look forward > to > > > furthermore suggestions and votes. HDFS-13248 is tracking the issue. > > > > > > - Approach A: Changing IPC/RPC layer protocol > > (IpcConnectionContextProto > > > or RpcHeader#RpcRequestHeaderProto) and add extra field about client > > > hostname. Of course the new field is optional, only input by Router > > and > > > parse by Namenode in generally. This approach is compatibility and > > Client > > > should do nothing after changing. > > > - Approach B: Changing ClientProtocol and add extra interface > > > create/append/getBlockLocations with additional parameter about > client > > > hostname. As approach A, it is input by Router and parse by > Namenode, > > and > > > also is compatibility. > > > - Approach C: Solve write and read locality separately based on > > current > > > interface and no changes, for write, hack client hostname as one of > > favor > > > nodes for addBlocks, for read, reorder targets at Router after > > Namenode > > > returns result to Router. > > > > > > As discussion and evaluation in HDFS-13248, we prefer to change IPC/RPC > > > layer protocol to support RPC data locality. We welcome more > suggestions, > > > votes or just give us feedback to push forward this feature. Thanks. > > > > > > Best Regards, > > > Hexiaoqiao > > > > > > reference > > > [1] https://issues.apache.org/jira/browse/HDFS-13248 > > > [2] https://issues.apache.org/jira/browse/HDFS-10467 > > > > > > [3] https://issues.apache.org/jira/browse/HDFS-12615 > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > >
