[ https://issues.apache.org/jira/browse/HDFS-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300317#comment-16300317 ]
Andrew Schiefelbein commented on HDFS-12957: -------------------------------------------- Someone just pointed this out to me: https://community.hortonworks.com/articles/103176/hdfs-settings-for-better-hadoop-performance.html The short circuit reads will work for my local processes, but for distributed ones having the ability to have name nodes behind a load balancer would be a wonderful addition. > Multi master name node > ---------------------- > > Key: HDFS-12957 > URL: https://issues.apache.org/jira/browse/HDFS-12957 > Project: Hadoop HDFS > Issue Type: Wish > Components: namenode > Affects Versions: 3.0.0 > Environment: Multi node high availability with high throughput > Reporter: Andrew Schiefelbein > Priority: Minor > > While I do appreciate that the HA and federated setups do take care of some > of the issues running in a multi node distributed fault tolerant way I > believe that a singe name node is a bit of a pain point for me. > My wish, and your mileage may vary, is that here could be a name node on each > data node system so that each system could act as a gateway and worker for > the cluster. > In my tests right now I can push nearly 2 million records / second through a > single node instance, but when I bring up 4 more nodes it doesn't go 4x > faster as it has to retrieve from the name node and go through it as a > gateway to the cluster. For speed and resiliency if this could be spread out > among n number of nodes and put behind a load balancer it would be the ideal > solution for distributed resilient and high throughput shared storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org