[
https://issues.apache.org/jira/browse/HDFS-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300592#comment-16300592
]
Andrew Schiefelbein edited comment on HDFS-12957 at 12/21/17 10:02 PM:
-----------------------------------------------------------------------
Thank you for the comments. I do understand that the ask is not a trivial one
and I'm doing my best to try to optimize the current setup I have for
throughput.
The choices for my particular use case is to buy an expensive san / nas, use a
glusterfs style setup or to hope that I can get enough throughput on the name
node. The scale of reads I need to do I'm not sure I can accomplish the speed
I need in a system that has to go through a single node, what I'm trying to do
may be an edge case and not fit within the HDFS model.
While I can get the HA setup with active / passive to work and I can have it
behind a load balancer with a health check, because the fail over should be
transparent to the down stream processes, I would much prefer to have an active
/ active setup instead. Even if it's 2 name nodes, both of them active at the
same time to me is a better solution than having one in a passive role. If
we're going to spend the time and expense to setup servers to do a role they
should take load or it's a waste of electricity to sit in a passive mode for
the "what if" days that may never come. It's an opinion and I know what that
means, but if it could be done I think it would be a wonderful addition to the
tool set.
was (Author: aschiefe):
Thank you for the comments. I do understand that the ask is not a trivial one
and I'm doing my best to try to optimize the current setup I have for
throughput.
The choices for my particular use case is to buy an expensive san / nas, use a
glusterfs style setup or to hope that I can get enough throughput on the name
node. The scale of reads I need to do I'm not sure I can accomplish the speed
I need in a system that has to go through a single node, what I'm trying to do
may be an edge case and not fit within the HDFS model.
While I can get the HA setup with active / passive to work and I can have it
behind a load balancer with a health check, because the fail over should be
transparent to the down stream processes, I would much prefer not to have an
active / active setup instead. Even if it's 2 name nodes, both of them active
at the same time to me is a better solution than having one in a passive role.
If we're going to spend the time and expense to setup servers to do a role they
should take load or it's a waste of electricity to sit in a passive mode for
the "what if" days that may never come. It's an opinion and I know what that
means, but if it could be done I think it would be a wonderful addition to the
tool set.
> Multi master name node
> ----------------------
>
> Key: HDFS-12957
> URL: https://issues.apache.org/jira/browse/HDFS-12957
> Project: Hadoop HDFS
> Issue Type: Wish
> Components: namenode
> Affects Versions: 3.0.0
> Environment: Multi node high availability with high throughput
> Reporter: Andrew Schiefelbein
> Priority: Minor
>
> While I do appreciate that the HA and federated setups do take care of some
> of the issues running in a multi node distributed fault tolerant way I
> believe that a singe name node is a bit of a pain point for me.
> My wish, and your mileage may vary, is that here could be a name node on each
> data node system so that each system could act as a gateway and worker for
> the cluster.
> In my tests right now I can push nearly 2 million records / second through a
> single node instance, but when I bring up 4 more nodes it doesn't go 4x
> faster as it has to retrieve from the name node and go through it as a
> gateway to the cluster. For speed and resiliency if this could be spread out
> among n number of nodes and put behind a load balancer it would be the ideal
> solution for distributed resilient and high throughput shared storage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
