[
https://issues.apache.org/jira/browse/HBASE-1357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nitay Joffe updated HBASE-1357:
-------------------------------
Description:
(2:11:20 PM) posix4e: so i want to run a back master on each node
(2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
(2:14:59 PM) posix4e: each master only gets the local regionserver connecting
(2:15:08 PM) posix4e: as it must be using that variable to know what to connect
to
(2:15:32 PM) nitay: the RS don't use hbase.master* anymore
(2:15:36 PM) nitay: ohhh i think i know th eproblem
(2:15:44 PM) nitay: so the RS use ZK to get the master address
(2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
(2:15:58 PM) nitay: b/c they write whatever was in their conf
(2:16:20 PM) posix4e: yea
(2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my
thinking?
(2:16:55 PM) posix4e: yea
(2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
(2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS
(allowing 0.0.0.0) by having master actually write RS's address to ZK when it
gets contacted
(2:18:21 PM) nitay: so now we need to find a way to find out the _actual_
address the master has bound to
(2:19:47 PM) posix4e: is their a way to do that?
(2:20:16 PM) nitay: i dont know, good question
(2:20:18 PM) posix4e: or does it require code changes i.e. regionserver
checking zk
(2:20:27 PM) nitay: did u verify the master address?
(2:20:48 PM) posix4e: one sec
(2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what
address we're using to talk to it
(2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs.
HBase
(2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c
the RS knows nothing about the master until the master address appears in ZK
(2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
(2:22:40 PM) nitay: yep that's the magic
(2:22:45 PM) nitay: k thx for verifying
(2:22:54 PM) nitay: u want to open up a JIRA?
(2:22:57 PM) posix4e: but if i could tell hbase.site to just use my
hostname:port it would work ok
(2:22:58 PM) posix4e: yea
(2:23:09 PM) posix4e: can i quote this conversation?
(2:23:18 PM) nitay: yes please do
(2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need
to actually set hbase.master* to the ip/host u're using
(2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
(2:24:02 PM) nitay: i know, its a royal PITA
(2:24:59 PM) posix4e: yea
(2:25:03 PM) posix4e: no problem
(2:25:20 PM) nitay: but that should work till we find a better solution
(2:25:21 PM) posix4e: I am trying to think how a patch would work
(2:25:25 PM) posix4e: have a masters file?
(2:25:44 PM) nitay: yeah if u have any ideas please offer them
(2:25:46 PM) nitay: hmm interesting idea
(2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking
against masters file?
(2:26:26 PM) posix4e: yea
(2:28:23 PM) nitay: one thing to note is we've talked about eventually getting
to a place where any RS can be master
(2:28:30 PM) nitay: but i like your idea
(2:28:37 PM) nitay: post it on the JIRA
(2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful,
its great to hear from people actually using this stuff
(2:32:56 PM) posix4e: yep
I also solved this by manually setting the hbase.master on each host to point
to the local hostname, which sucks.
was:
(2:11:20 PM) posix4e: so i want to run a back master on each node
(2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
(2:14:59 PM) posix4e: each master only gets the local regionserver connecting
(2:15:08 PM) posix4e: as it must be using that variable to know what to connect
to
(2:15:32 PM) nitay: the RS don't use hbase.master* anymore
(2:15:36 PM) nitay: ohhh i think i know th eproblem
(2:15:44 PM) nitay: so the RS use ZK to get the master address
(2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
(2:15:58 PM) nitay: b/c they write whatever was in their conf
(2:16:20 PM) posix4e: yea
(2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my
thinking?
(2:16:55 PM) posix4e: yea
(2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the defaults
(2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS
(allowing 0.0.0.0) by having master actually write RS's address to ZK when it
gets contacted
(2:18:21 PM) nitay: so now we need to find a way to find out the _actual_
address the master has bound to
(2:19:47 PM) posix4e: is their a way to do that?
(2:20:16 PM) nitay: i dont know, good question
(2:20:18 PM) posix4e: or does it require code changes i.e. regionserver
checking zk
(2:20:27 PM) nitay: did u verify the master address?
(2:20:48 PM) posix4e: one sec
(2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what
address we're using to talk to it
(2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs.
HBase
(2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell b/c
the RS knows nothing about the master until the master address appears in ZK
(2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
(2:22:40 PM) nitay: yep that's the magic
(2:22:45 PM) nitay: k thx for verifying
(2:22:54 PM) nitay: u want to open up a JIRA?
(2:22:57 PM) posix4e: but if i could tell hbase.site to just use my
hostname:port it would work ok
(2:22:58 PM) posix4e: yea
(2:23:09 PM) posix4e: can i quote this conversation?
(2:23:18 PM) nitay: yes please do
(2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially need
to actually set hbase.master* to the ip/host u're using
(2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
(2:24:02 PM) nitay: i know, its a royal PITA
(2:24:59 PM) posix4e: yea
(2:25:03 PM) posix4e: no problem
(2:25:20 PM) nitay: but that should work till we find a better solution
(2:25:21 PM) posix4e: I am trying to think how a patch would work
(2:25:25 PM) posix4e: have a masters file?
(2:25:44 PM) nitay: yeah if u have any ideas please offer them
(2:25:46 PM) nitay: hmm interesting idea
(2:26:16 PM) nitay: and then do some local gethostbyname() type thing checking
against masters file?
(2:26:26 PM) posix4e: yea
(2:28:23 PM) nitay: one thing to note is we've talked about eventually getting
to a place where any RS can be master
(2:28:30 PM) nitay: but i like your idea
(2:28:37 PM) nitay: post it on the JIRA
(2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful,
its great to hear from people actually using this stuff
(2:32:56 PM) posix4e: yep
I also solved this by manually setting the hbase.master on each host to point
to the local hostname, which sucks.
Fix Version/s: (was: 0.20.1)
(was: 0.20.0)
Moving out of 0.20 for now. In meeting we discussed:
- Getting rid of hbase.master.hostname property completely. Detect master
address using an inet address detection scheme similar to what Hadoop does.
- The above breaks the default ZooKeeper case of a single server running on the
master node. RegionServers currently find this ZooKeeper using the master
address property. This default case should be replaced by having the zoo.cfg
turn into a template file which can be created and rsync'ed to RegionServer
nodes by e.g. start-hbase.sh script.
- Add a masters file, similar to regionservers, with pool of servers for master
election.
> If one sets the hbase.master to 0.0.0.0 non local regionservers can't find
> the master
> -------------------------------------------------------------------------------------
>
> Key: HBASE-1357
> URL: https://issues.apache.org/jira/browse/HBASE-1357
> Project: Hadoop HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: All
> Reporter: Alex Newman
> Fix For: 0.21.0
>
>
> (2:11:20 PM) posix4e: so i want to run a back master on each node
> (2:11:29 PM) posix4e: and i have my hbase.master set to 0.0.0.0
> (2:14:59 PM) posix4e: each master only gets the local regionserver connecting
> (2:15:08 PM) posix4e: as it must be using that variable to know what to
> connect to
> (2:15:32 PM) nitay: the RS don't use hbase.master* anymore
> (2:15:36 PM) nitay: ohhh i think i know th eproblem
> (2:15:44 PM) nitay: so the RS use ZK to get the master address
> (2:15:49 PM) nitay: but the masters are writing 0.0.0.0 to it
> (2:15:58 PM) nitay: b/c they write whatever was in their conf
> (2:16:20 PM) posix4e: yea
> (2:16:42 PM) nitay: can u do a zookeeper dump of that node to verify my
> thinking?
> (2:16:55 PM) posix4e: yea
> (2:17:12 PM) nitay: it should be /hbase/master, unless u've changed the
> defaults
> (2:17:59 PM) nitay: hmm s o ye this is a problem, we solved this in RS
> (allowing 0.0.0.0) by having master actually write RS's address to ZK when it
> gets contacted
> (2:18:21 PM) nitay: so now we need to find a way to find out the _actual_
> address the master has bound to
> (2:19:47 PM) posix4e: is their a way to do that?
> (2:20:16 PM) nitay: i dont know, good question
> (2:20:18 PM) posix4e: or does it require code changes i.e. regionserver
> checking zk
> (2:20:27 PM) nitay: did u verify the master address?
> (2:20:48 PM) posix4e: one sec
> (2:21:03 PM) nitay: its almost like we want ZK to be able to tell us what
> address we're using to talk to it
> (2:21:20 PM) nitay: that assumes u dont have different NICs to talk to ZK vs.
> HBase
> (2:21:59 PM) nitay: posix4e, u can't really use the RS as far as i can tell
> b/c the RS knows nothing about the master until the master address appears in
> ZK
> (2:22:25 PM) posix4e: 0:0:0:0:0:0:0:0:60000
> (2:22:40 PM) nitay: yep that's the magic
> (2:22:45 PM) nitay: k thx for verifying
> (2:22:54 PM) nitay: u want to open up a JIRA?
> (2:22:57 PM) posix4e: but if i could tell hbase.site to just use my
> hostname:port it would work ok
> (2:22:58 PM) posix4e: yea
> (2:23:09 PM) posix4e: can i quote this conversation?
> (2:23:18 PM) nitay: yes please do
> (2:23:45 PM) nitay: also, to fix this here and now for u, u'd essentially
> need to actually set hbase.master* to the ip/host u're using
> (2:23:55 PM) nitay: and change it on each backup master to that guy's host/ip
> (2:24:02 PM) nitay: i know, its a royal PITA
> (2:24:59 PM) posix4e: yea
> (2:25:03 PM) posix4e: no problem
> (2:25:20 PM) nitay: but that should work till we find a better solution
> (2:25:21 PM) posix4e: I am trying to think how a patch would work
> (2:25:25 PM) posix4e: have a masters file?
> (2:25:44 PM) nitay: yeah if u have any ideas please offer them
> (2:25:46 PM) nitay: hmm interesting idea
> (2:26:16 PM) nitay: and then do some local gethostbyname() type thing
> checking against masters file?
> (2:26:26 PM) posix4e: yea
> (2:28:23 PM) nitay: one thing to note is we've talked about eventually
> getting to a place where any RS can be master
> (2:28:30 PM) nitay: but i like your idea
> (2:28:37 PM) nitay: post it on the JIRA
> (2:30:24 PM) nitay: i gotta run, thanks for the info posix4e - very helpful,
> its great to hear from people actually using this stuff
> (2:32:56 PM) posix4e: yep
> I also solved this by manually setting the hbase.master on each host to
> point to the local hostname, which sucks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
