[
https://issues.apache.org/jira/browse/HBASE-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651360#comment-13651360
]
Devaraj Das edited comment on HBASE-7932 at 5/7/13 9:59 PM:
------------------------------------------------------------
Ok this patch should address most of Stack's previous comments except for the
unit tests one. I have a unit test that does end-to-end verification of this
whole thing but not at the level of racks and so on. Since a significant part
of the 'math' is based on the rack configuration, etc., I'll try to write some
unit tests that fake racks and so on but I wanted to get a review on the patch
as it stands now.
On the specific questions:
bq. I was thinking that when balance is called, your new balancer should have
nothing to do right?
I have put in a comment in the balanceCluster cluster call in
FavoredNodeLoadBalancer. The work on the balancer core is going to be done in a
follow up jira.
bq. Does access to rackToRegionServerMap need to be synchronized
I have changed the patch so that all these data structures are in a separate
class (FavoredNodesAssignmentHelper). An instance of this class is created per
a call to the balancer methods (via the assignmanager's methods), and this
class is seeded with the servers by the balancer. My previous patch had this
bug where it would not distinguish between servers based on what the
AssignmentManager chooses to use for a particular assignment.
bq. return serverSize >= FAVORED_NODES_NUM;
Done.
bq. On assign, you are just trying to spread the regions. I suppose it would
make some sense to consider how loaded a regionserver already is before
choosing it? How many regions it is carrying?
The hosts are decided on by the AssignmentMananger and passed to the
FavoredNodesAssignmentHelper via the FavoredNodeLoadBalancer.. The AM would
choose the right hosts and give it to us.. The
FavoredNodeLoadBalancer.balanceCluster call would look at the attributes of the
regions/nodes but that is not part of this patch now.
bq. So what if it is cluster startup and there 10k regions to assign.
Yeah.. I reverted this change and send the hosts to the RegionServers as part
of the openRegion request.
bq. If .META. already has been marked with favored nodes, we do not seem to be
taking notice only overwriting whatever is there on each assign
Fixed that (when a cluster starts up it looks at the favored nodes in
AssignmentManager.assignAllUserRegions)
bq. I do not see how it is added to updateRegionFavoredNodesMapping.... am I
reading it wrong?
Removed this in the new patch since the nodes are sent in the RPC as opposed to
meta scan.
bq. favoredNodes in .META. need to be pb'd
Done
bq. Why InetSocketAddress saving favored nodes
The new HDFS create API works with InetSocketAddress and hence I thought we
could start with that itself. (HBASE-7942 would start using the new HDFS API.)
bq. I think this approach much cleaner; better packaging of all that is going
on.
Cool.
was (Author: devaraj):
Ok this patch should address most of Stack's previous comments except for
the unit tests one. I have a unit test that does end-to-end verification of
this whole thing but not at the level of racks and so on. Since a significant
part of the 'math' is based on the rack configuration, etc., I'll try to write
some unit tests that fake racks and so on but I wanted to get a review on the
patch as it stands now.
On the specific questions:
bq. I was thinking that when balance is called, your new balancer should have
nothing to do right?
I have put in a comment in the balanceCluster cluster call in
FavoredNodeLoadBalancer. The work on the balancer core is going to be done in a
follow up jira.
bq. Does access to rackToRegionServerMap need to be synchronized
I have changed the patch so that all these data structures are in a separate
class (FavoredNodesAssignmentHelper). An instance of this class is created per
a call to the balancer methods (via the assignmanager's methods), and this
class is seeded with the servers by the balancer. My previous patch had this
bug where it would not distinguish between servers based on what the
AssignmentManager chooses to use for a particular assignment.
bq. return serverSize >= FAVORED_NODES_NUM;
Done.
bq. On assign, you are just trying to spread the regions. I suppose it would
make some sense to consider how loaded a regionserver already is before
choosing it? How many regions it is carrying?
The hosts are decided on by the AssignmentMananger and passed to the
FavoredNodesAssignmentHelper via the FavoredNodeLoadBalancer.. The AM would
choose the right hosts and give it to us.. The
FavoredNodeLoadBalancer.balanceCluster call would look at the attributes of the
regions/nodes but that is not part of this patch now.
bq. So what if it is cluster startup and there 10k regions to assign.
Yeah.. I reverted this change and send the hosts to the RegionServers as part
of the openRegion request.
bq. If .META. already has been marked with favored nodes, we do not seem to be
taking notice only overwriting whatever is there on each assign
Fixed that (when a cluster starts up it looks at the favored nodes in
AssignmentManager.assignAllUserRegions)
bq. I do not see how it is added to updateRegionFavoredNodesMapping.... am I
reading it wrong?
Removed this in the new patch since the nodes are sent in the RPC as opposed to
meta scan.
bq. Why InetSocketAddress saving favored nodes
The new HDFS create API works with InetSocketAddress and hence I thought we
could start with that itself. (HBASE-7942 would start using the new HDFS API.)
bq. I think this approach much cleaner; better packaging of all that is going
on.
Cool.
> Do the necessary plumbing for the region locations in META table and send the
> info to the RegionServers
> -------------------------------------------------------------------------------------------------------
>
> Key: HBASE-7932
> URL: https://issues.apache.org/jira/browse/HBASE-7932
> Project: HBase
> Issue Type: Sub-task
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Priority: Critical
> Fix For: 0.95.1
>
> Attachments: 7932-1.patch, 7932-3.patch, 7932-8.1.patch,
> 7932-8.patch, 7932-simple-2.patch, 7932-simple-with-meta-pb.1.txt,
> 7932-wip-2.patch, 7932-wip-3.patch, 7932-wip-4.patch, 7932-wip.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
