That error message shdn't be a problem. Can you try upgrading to 0.9.3.1 and see if that fixes anything? Theres been many bugfixes since 0.9.2.8 and though I can't think of one that explains this scenario, it would be easier to debug/reproduce with the latest & greatest code.
So Hyperspace is up and running but the RangeServer can't connect to it? Can you do a clean run and (cap cleandb; kill all dangling HT procs manually; cap dist; cap start) and tar ball all the logs, Capfile and config file here<http://groups.google.com/group/hypertable-user/files?pli=1>? Btw I'm not a capistrano expert, but I wonder if the way you've defined the roles is correct. If you look at the examples in the conf directory it looks like it shd be: *role :localhost, "master"* Also I hope you have two servers named "master" & "slave" and you're running the cap commands from "master". Btw do both RangeServers fail to come up or just the one "slave" ? If its just the one on "slave" can you try to connect to Hyperspace from "slave" using the command line: *<$INSTALL_DIR>/bin/ht hyperspace --conf <$HT_CONFIG_FILE>--exec "open /; readdir /;"* on "slave". If not I suspect you might have some firewall blocking access to the Hyperspace port. -Sanjit On Sat, May 29, 2010 at 2:08 AM, Harshada Chavan <[email protected]>wrote: > > > > $ps -ef | grep Hyperspace > > harshada 15228 1 0 14:20 ? 00:00:00 > /opt/hypertable/hypertable-0.9.2.8-alpha/0.9.2.8/bin/Hyperspace.Master--pidfile > /opt/hypertable/hypertable-0.9.2.8-alpha/ > 0.9.2.8/run/Hyperspace.pid --verbose > --config=/opt/hypertable/hypertable-0.9.2.8-alpha/ > 0.9.2.8/conf/hypertable.cfg > > ------------------------------------------------------------------------------------------------------ > > Is Hyperspace running? Can you try something like >> /opt/hypertable/hypertable-0.9.2.8-alpha/bin/ht hyperspace --exec "open /; >> readdir /;" >> > > When I ran above command on master, I got following output.. > > > > ------------------------------------------------------------------------------- > $ ./ht hyperspace --exec "open /; readdir /;" > SESSION CALLBACK: Safe > > Welcome to the hyperspace command interpreter. > For information about Hypertable, visit http://www.hypertable.org/ > > Type 'help' for a list of commands, or 'help shell' for a > list of shell meta commands. > > (dir) hyperspace > (dir) hypertable > > --------------------------------------------------------------------------------- > > seems it's running. But then why RangeServer log at slave says, > > -------------------- > > "1275077211 ERROR Hypertable.RangeServer : > (/opt/hypertable/hypertable-0.9.2.8-alpha/src/cc/Hypertable/RangeServer/main.cc:86)Unable > to connect to hyperspace, exiting..." > -------------------- > > Also, in log/Hyperspace.log at master I see following error: > > ------------------------------ > 1275123032 ERROR Hyperspace.Master : run > (/opt/hypertable/hypertable-0.9.2.8-alpha/src/cc/Hyperspace/RequestHandlerOpen.cc:60): > Hypertable::Exception: node: '/hypertable/master' parent node:* * > '/hypertable' - HYPERSPACE file not found ** > > at void Hyperspace::Master::open(Hyperspace::ResponseCallbackOpen*, > uint64_t, const char*, uint32_t, uint32_t, > std::vector<Hyperspace::Attribute, std::allocator<Hyperspace::Attribute> >&) > (/opt/hypertable/hypertable-0.9.2.8-alpha/src/cc/Hyperspace/Master.cc:877) > -------------------- > > >> -Sanjit >> >> On Fri, May 28, 2010 at 2:32 PM, Harshada <[email protected]>wrote: >> >>> >>> >>> On May 29, 2:10 am, Sanjit Jhala <[email protected]> wrote: >>> > Hi Harshada, >>> > >>> > I'm confused about what state your setup is in. >>> >>> I have followed procedure given at >>> http://code.google.com/p/hypertable/wiki/UpAndRunningWithHadoop >>> and have been successful till "cap dist". >>> >>> >Can you try the following. >>> > 1. cap cleandb >>> > 2. Run ps and make sure all Hypertable related procs (Hyperspace, >>> > RangeServer, Master, DFSBroker, ThroftBroker, any dangling start/stop >>> > scripts) have been stopped. If not kill them manually. >>> >>> Killed everything. >>> >>> > 3. just to be sure run cap cleandb again >>> >>> Executed this command again. It's output is exactly similar to what I >>> have posted in the reply to your earlier reply (my prev to prev mail). >>> >>> > 4. cap start >>> >>> Again same output as posted earlier in the same mail. >>> >>> > >>> > Also are you running on a single node? If not is your capfile updated >>> with >>> > all the servers in your cluster? >>> >>> my Capfile: >>> >>> set :source_machine, "master" >>> set :install_dir, "/opt/hypertable/hypertable-0.9.2.8-alpha" >>> set :hypertable_version, "0.9.2.8" >>> set :default_dfs, "hadoop" >>> set :default_config, "/opt/hypertable/hypertable.cfg" >>> >>> role :master, "master" >>> role :hyperspace, "master" >>> role :slave, "master", "slave" >>> role :localhost, "127.0.0.1" >>> >>> and hypertable.cfg: >>> >>> # >>> # hypertable.cfg >>> # >>> >>> # HDFS Broker >>> HdfsBroker.Port=38030 >>> HdfsBroker.fs.default.name=hdfs://master:54310 >>> HdfsBroker.Workers=20 >>> >>> # Ceph Broker >>> CephBroker.Port=38030 >>> CephBroker.Workers=20 >>> CephBroker.MonAddr=10.0.1.245:6789 >>> >>> # Local Broker >>> DfsBroker.Local.Port=38030 >>> DfsBroker.Local.Root=fs/local >>> >>> # DFS Broker - for clients >>> DfsBroker.Host=localhost >>> DfsBroker.Port=38030 >>> >>> # Hyperspace >>> Hyperspace.Replica.Host=localhost >>> Hyperspace.Replica.Port=38040 >>> Hyperspace.Replica.Dir=hyperspace >>> Hyperspace.Replica.Workers=20 >>> >>> # Hypertable.Master >>> Hypertable.Master.Host=localhost >>> Hypertable.Master.Port=38050 >>> Hypertable.Master.Workers=20 >>> >>> # Hypertable.RangeServer >>> Hypertable.RangeServer.Port=38060 >>> >>> Hyperspace.KeepAlive.Interval=30000 >>> Hyperspace.Lease.Interval=1000000 >>> Hyperspace.GracePeriod=200000 >>> >>> # ThriftBroker >>> ThriftBroker.Port=38080 >>> >>> **NOTE: There's no property like: Hyperspace.Master.Host. Looking at >>> the log files, it takes "localhost" as the value. >>> >>> After reading log files (as posted in my prev mail), I think the >>> problem is with Hyperspace not being up at master. Because the log for >>> RangServer at Slave machine says: >>> >>> "1275082230 ERROR Hypertable.RangeServer : (/opt/hypertable/ >>> hypertable-0.9.2.8-alpha/src/cc/Hypertable/RangeServer/main.cc:86) >>> Unable to connect to hyperspace, exiting..." >>> >>> > >>> > -Sanjit >>> > >>> > On Fri, May 28, 2010 at 1:43 PM, Harshada <[email protected]> >>> wrote: >>> > > Looks like this issue is related to the Hyperspace @master. >>> > >>> > > When I checked RangServer's log at slave, I found this >>> > >>> > > "1275077211 ERROR Hypertable.RangeServer : (/opt/hypertable/ >>> > > hypertable-0.9.2.8-alpha/src/cc/Hypertable/RangeServer/main.cc:86) >>> > > Unable to connect to hyperspace, exiting..." >>> > >>> > > I checked Hyperspace's log at master for the error: >>> > >>> > > 1275076989 ERROR Hyperspace.Master : run (/opt/hypertable/ >>> > > hypertable-0.9.2.8-alpha/src/cc/Hyperspace/RequestHandlerOpen.cc:60): >>> > > Hypertable::Exception: node: '/hypertable/master' parent node: '/ >>> > > hypertable' - HYPERSPACE file not found >>> > > at void >>> > > Hyperspace::Master::open(Hyperspace::ResponseCallbackOpen*, uint64_t, >>> > > const char*, uint32_t, uint32_t, std::vector<Hyperspace::Attribute, >>> > > std::allocator<Hyperspace::Attribute> >&) (/opt/hypertable/ >>> > > hypertable-0.9.2.8-alpha/src/cc/Hyperspace/Master.cc:877) >>> > >>> > > What does HYPERSPACE file not found mean? >>> > >>> > > There exists a directory /hypertable at HDFS at the master with 777 >>> > > permission. >>> > >>> > > Please help. >>> > >>> > > On May 29, 12:00 am, Harshada <[email protected]> wrote: >>> > > > Thanks for your reply Sanjit. >>> > >>> > > > 'cap cleandb' gives following output: (with one error at the end) >>> > >>> > > > * executing `stop' >>> > > > ** transaction: start >>> > > > * executing `stop_slaves' >>> > > > * executing "/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/bin/ >>> > > > stop-servers.sh --no-hyperspace" >>> > > > servers: ["master", "slave"] >>> > > > [master] executing command >>> > > > ** [out :: master] Sending shutdown command >>> > > > ** [out :: master] Unable to establish connection to range server >>> > > > [slave] executing command >>> > > > ** [out :: slave] Sending shutdown command >>> > > > ** [out :: slave] Unable to establish connection to range server >>> > > > ** [out :: master] Shutdown range server complete >>> > > > ** [out :: slave] Shutdown range server complete >>> > > > ** [out :: master] Shutdown DFS broker complete >>> > > > ** [out :: master] Shutdown thrift broker complete >>> > > > ** [out :: master] Shutdown hypertable master complete >>> > > > ** [out :: slave] Shutdown thrift broker complete >>> > > > ** [out :: slave] Shutdown DFS broker complete >>> > > > ** [out :: slave] Shutdown hypertable master complete >>> > > > command finished >>> > > > * executing `stop_master' >>> > > > * executing "/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/bin/ >>> > > > stop-servers.sh --no-hyperspace" >>> > > > servers: ["master"] >>> > > > [master] executing command >>> > > > ** [out :: master] Sending shutdown command >>> > > > ** [out :: master] Unable to establish connection to range server >>> > > > ** [out :: master] Shutdown range server complete >>> > > > ** [out :: master] Shutdown DFS broker complete >>> > > > ** [out :: master] Shutdown thrift broker complete >>> > > > ** [out :: master] Shutdown hypertable master complete >>> > > > command finished >>> > > > * executing `stop_hyperspace' >>> > > > * executing "/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/bin/ >>> > > > stop-hyperspace.sh" >>> > > > servers: ["master"] >>> > > > [master] executing command >>> > > > ** [out :: master] Killing Hyperspace.pid 14096 >>> > > > *** [err :: master] /opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/ >>> > > > bin/ht-env.sh: line 68: kill: (14096) - No such process >>> > > > ** [out :: master] Shutdown hyperspace complete >>> > > > command finished >>> > > > ** transaction: commit >>> > > > harsh...@erts-server:/opt/hypertable/hypertable-0.9.2.8-alpha/conf$ >>> / >>> > > > var/lib/gems/1.8/bin/cap cleandb >>> > > > * executing `cleandb' >>> > > > ** transaction: start >>> > > > * executing `clean_ranges' >>> > > > * executing "/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/bin/ >>> > > > start-dfsbroker.sh hadoop --config=/opt/hypertable/ >>> > > > hypertable-0.9.2.8-alpha/0.9.2.8/conf/hypertable.cfg && /opt/ >>> > > > hypertable/hypertable-0.9.2.8-alpha/0.9.2.8/bin/clean-database.sh >>> ;" >>> > > > servers: ["master", "slave"] >>> > > > [master] executing command >>> > > > [slave] executing command >>> > > > ** [out :: master] DFS broker: available file descriptors: 1024 >>> > > > ** [out :: slave] DFS broker: available file descriptors: 1024 >>> > > > ** [out :: slave] Started DFS Broker (hadoop) >>> > > > ** [out :: slave] Removed /hypertable/servers in DFS >>> > > > ** [out :: slave] Removed /hypertable/tables in DFS >>> > > > ** [out :: slave] Cleared hyperspace >>> > > > ** [out :: slave] Killing DfsBroker.hadoop.pid 24587 >>> > > > ** [out :: slave] Shutdown hyperspace complete >>> > > > ** [out :: slave] Shutdown range server complete >>> > > > ** [out :: slave] Shutdown thrift broker complete >>> > > > ** [out :: slave] Shutdown hypertable master complete >>> > > > ** [out :: slave] Shutdown DFS broker complete >>> > > > ** [out :: master] Waiting for DFS Broker (hadoop) to come up... >>> > > > ** [out :: master] Waiting for DFS Broker (hadoop) to come up... >>> > > > ** [out :: master] Waiting for DFS Broker (hadoop) to come up... >>> > > > ** [out :: master] Waiting for DFS Broker (hadoop) to come up... >>> > > > ** [out :: master] Waiting for DFS Broker (hadoop) to come up... >>> > > > ** [out :: master] Waiting for DFS Broker (hadoop) to come up... >>> > > > ** [out :: master] Waiting for DFS Broker (hadoop) to come up... >>> > > > ** [out :: master] Waiting for DFS Broker (hadoop) to come up... >>> > > > ** [out :: master] ERROR: DFS Broker (hadoop) did not come up >>> > > > ** [out :: master] DfsBroker.hadoop appears to be running (19475): >>> > > > ** [out :: master] harshada 19475 19394 0 May28 ? 00:00:00 java - >>> > > > classpath /opt/hypertable/hypertable-0.9.2.8-alpha/0.9.2.8:/opt/ >>> > > > hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/lib/*.jar:/opt/hypertable/ >>> > > > hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/lib/java/commons-logging-1.0.4.jar:/ >>> > > > opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/lib/java/hadoop-0.20.1- >>> > > > core.jar:/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/lib/java/ >>> > > > hypertable-0.9.2.8.jar:/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> > > > >>> 0.9.2.8/lib/java/junit-4.3.1.jar:/opt/hypertable/hypertable-0.9.2.8- >>> > > > alpha/0.9.2.8/lib/java/libthrift-0.2.0.jar:/opt/hypertable/ >>> > > > hypertable-0.9.2.8-alpha/0.9.2.8/lib/java/log4j-1.2.13.jar:/opt/ >>> > > > hypertable/hypertable-0.9.2.8-alpha/0.9.2.8/lib/java/slf4j- >>> > > > api-1.5.8.jar:/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/lib/ >>> > > > >>> java/slf4j-log4j12-1.5.8.jar:/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> > > > 0.9.2.8/lib/jetty-ext/*.jar org.hypertable.DfsBroker.hadoop.main >>> -- >>> > > > verbose --config=/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/conf/ >>> > > > hypertable.cfg >>> > > > command finished >>> > > > failed: "sh -c '/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/bin/ >>> > > > start-dfsbroker.sh hadoop --config=/opt/hypertable/ >>> > > > hypertable-0.9.2.8-alpha/0.9.2.8/conf/hypertable.cfg && /opt/ >>> > > > hypertable/hypertable-0.9.2.8-alpha/0.9.2.8/bin/clean-database.sh; >>> '" >>> > > > on master >>> > >>> > > > But going ahead when I run 'cap start', everything else starts up >>> > > > except slaves's RangeServer. Here's the output: >>> > >>> > > > * executing `start' >>> > > > ** transaction: start >>> > > > * executing `start_hyperspace' >>> > > > * executing "/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/bin/ >>> > > > start-hyperspace.sh >>> --config=/opt/hypertable/hypertable-0.9.2.8- >>> > > > alpha/0.9.2.8/conf/hypertable.cfg" >>> > > > servers: ["master"] >>> > > > [master] executing command >>> > > > ** [out :: master] Started Hyperspace >>> > > > command finished >>> > > > * executing `start_master' >>> > > > * executing "/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/bin/ >>> > > > start-dfsbroker.sh hadoop --config=/opt/hypertable/ >>> > > > hypertable-0.9.2.8-alpha/0.9.2.8/conf/hypertable.cfg &&\\\n >>> /opt/ >>> > > > hypertable/hypertable-0.9.2.8-alpha/0.9.2.8/bin/start-master.sh -- >>> > > > config=/opt/hypertable/hypertable-0.9.2.8-alpha/0.9.2.8/conf/ >>> > > > hypertable.cfg" >>> > > > servers: ["master"] >>> > > > [master] executing command >>> > > > ** [out :: master] DFS broker: available file descriptors: 1024 >>> > > > ** [out :: master] DfsBroker.hadoop appears to be running (19475): >>> > > > ** [out :: master] harshada 19475 1 0 May28 ? 00:00:01 java - >>> > > > classpath /opt/hypertable/hypertable-0.9.2.8-alpha/0.9.2.8:/opt/ >>> > > > hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/lib/*.jar:/opt/hypertable/ >>> > > > hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/lib/java/commons-logging-1.0.4.jar:/ >>> > > > opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/lib/java/hadoop-0.20.1- >>> > > > core.jar:/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/lib/java/ >>> > > > hypertable-0.9.2.8.jar:/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> > > > >>> 0.9.2.8/lib/java/junit-4.3.1.jar:/opt/hypertable/hypertable-0.9.2.8- >>> > > > alpha/0.9.2.8/lib/java/libthrift-0.2.0.jar:/opt/hypertable/ >>> > > > hypertable-0.9.2.8-alpha/0.9.2.8/lib/java/log4j-1.2.13.jar:/opt/ >>> > > > hypertable/hypertable-0.9.2.8-alpha/0.9.2.8/lib/java/slf4j- >>> > > > api-1.5.8.jar:/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/lib/ >>> > > > >>> java/slf4j-log4j12-1.5.8.jar:/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> > > > 0.9.2.8/lib/jetty-ext/*.jar org.hypertable.DfsBroker.hadoop.main >>> -- >>> > > > verbose --config=/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/conf/ >>> > > > hypertable.cfg >>> > > > ** [out :: master] Started Hypertable.Master >>> > > > command finished >>> > > > * executing `start_slaves' >>> > > > * executing "/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> 0.9.2.8/bin/ >>> > > > random-wait.sh 5 &&\\\n /opt/hypertable/hypertable-0.9.2.8-alpha/ >>> > > > 0.9.2.8/bin/start-dfsbroker.sh hadoop >>> --config=/opt/hypertable/ >>> > > > hypertable-0.9.2.8-alpha/0.9.2.8/conf/hypertable.cfg &&\\\n >>> /opt/ >>> > > > hypertable/hypertable-0.9.2.8-alpha/0.9.2.8/bin/start- >>> > > > rangeserver.sh >>> --config=/opt/hypertable/hypertable-0.9.2.8-alpha/ >>> > > > 0.9.2.8/conf/hypertable.cfg &&\\\n >>> > >>> > ... >>> > >>> > read more ยป >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Hypertable Development" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]<hypertable-dev%[email protected]> >>> . >>> For more options, visit this group at >>> http://groups.google.com/group/hypertable-dev?hl=en. >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Hypertable Development" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]<hypertable-dev%[email protected]> >> . >> For more options, visit this group at >> http://groups.google.com/group/hypertable-dev?hl=en. >> > > > > -- > Regards, > Harshada Chavan > [MTech3, CSE, IITB] > > -- > You received this message because you are subscribed to the Google Groups > "Hypertable Development" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<hypertable-dev%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/hypertable-dev?hl=en. > -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
