StarGate HTTP ERROR: 404
Hi there, Anything wrrong with my stargate server? Thanks HBase 20.2 Starting restServer 2010-10-11 14:05:22.365::INFO: Logging to STDERR via org.mortbay.log.StdErrLog 2010-10-11 14:05:22.430::INFO: jetty-6.1.14 2010-10-11 14:05:22.892::INFO: Started socketconnec...@0.0.0.0:60050 [d...@f12p4cc11 bin]$ curl http://localhost:60050/version html head meta http-equiv=Content-Type content=text/html; charset=ISO-8859-1/ titleError 404 NOT_FOUND/title /head bodyh2HTTP ERROR: 404/h2preNOT_FOUND/pre Fleming Chiu(邱宏明) Ext: 707-2260 Be Veg, Go Green, Save the Planet! --- TSMC PROPERTY This email communication (and any attachments) is proprietary information for the sole use of its intended recipient. Any unauthorized review, use or distribution by anyone other than the intended recipient is strictly prohibited. If you are not the intended recipient, please notify the sender by replying to this email, and then delete this email and any copies of it immediately. Thank you. ---
Re: StarGate HTTP ERROR: 404
Actually, this is not Stargate, but older REST service that was deprecated. To activate Stargate, copy $HBASE_HOME/contrib/stargate/* and $HBASE_HOME/contrib/stargate/lib/* to hbase lib directory ($HBASE_HOME/lib) and start it with: $HBASE_HOME/bin/hbase org.apache.hadoop.hbase.stargate.Main Now, you should get Stargate on 8080 port, as was described here: http://wiki.apache.org/hadoop/Hbase/Stargate 2010/10/11 hmch...@tsmc.com: Hi there, Anything wrrong with my stargate server? Thanks HBase 20.2 Starting restServer 2010-10-11 14:05:22.365::INFO: Logging to STDERR via org.mortbay.log.StdErrLog 2010-10-11 14:05:22.430::INFO: jetty-6.1.14 2010-10-11 14:05:22.892::INFO: Started socketconnec...@0.0.0.0:60050 [d...@f12p4cc11 bin]$ curl http://localhost:60050/version html head meta http-equiv=Content-Type content=text/html; charset=ISO-8859-1/ titleError 404 NOT_FOUND/title /head bodyh2HTTP ERROR: 404/h2preNOT_FOUND/pre Fleming Chiu(邱宏明) Ext: 707-2260 Be Veg, Go Green, Save the Planet! --- TSMC PROPERTY This email communication (and any attachments) is proprietary information for the sole use of its intended recipient. Any unauthorized review, use or distribution by anyone other than the intended recipient is strictly prohibited. If you are not the intended recipient, please notify the sender by replying to this email, and then delete this email and any copies of it immediately. Thank you. ---
Re: StarGate HTTP ERROR: 404
Hi Fleming, First, Sanel is correct, whatever you are attempting to use is not Stargate. Kindly follow the rest of the advice. HBase 20.2 You should be using HBase 0.20.6. We can't help muchwith problems with 0.20.2 any more -- in just about all cases the first advice will be to upgrade to 0.20.6 -- given there have been several releases made since then. Best regards, - Andy --- On Mon, 10/11/10, Sanel Zukan san...@gmail.com wrote: From: Sanel Zukan san...@gmail.com Subject: Re: StarGate HTTP ERROR: 404 To: user@hbase.apache.org Date: Monday, October 11, 2010, 2:03 AM Actually, this is not Stargate, but older REST service that was deprecated. To activate Stargate, copy $HBASE_HOME/contrib/stargate/* and $HBASE_HOME/contrib/stargate/lib/* to hbase lib directory ($HBASE_HOME/lib) and start it with: $HBASE_HOME/bin/hbase org.apache.hadoop.hbase.stargate.Main Now, you should get Stargate on 8080 port, as was described here: http://wiki.apache.org/hadoop/Hbase/Stargate 2010/10/11 hmch...@tsmc.com: Hi there, Anything wrrong with my stargate server? Thanks HBase 20.2 Starting restServer 2010-10-11 14:05:22.365::INFO: Logging to STDERR via org.mortbay.log.StdErrLog 2010-10-11 14:05:22.430::INFO: jetty-6.1.14 2010-10-11 14:05:22.892::INFO: Started socketconnec...@0.0.0.0:60050 [d...@f12p4cc11 bin]$ curl http://localhost:60050/version html head meta http-equiv=Content-Type content=text/html; charset=ISO-8859-1/ titleError 404 NOT_FOUND/title /head bodyh2HTTP ERROR: 404/h2preNOT_FOUND/pre Fleming Chiu(邱宏明) Ext: 707-2260 Be Veg, Go Green, Save the Planet! --- TSMC PROPERTY This email communication (and any attachments) is proprietary information for the sole use of its intended recipient. Any unauthorized review, use or distribution by anyone other than the intended recipient is strictly prohibited. If you are not the intended recipient, please notify the sender by replying to this email, and then delete this email and any copies of it immediately. Thank you. ---
Re: Hbase internally row location mechanism
Section 5.1 of the Bigtable paper gives a pretty good explanation: http://labs.google.com/papers/bigtable.html In HBase, Chubby is replaced by ZooKeeper, root tablet by the -ROOT- table, and METADATA tablets by the .META. table. J-D On Sun, Oct 10, 2010 at 10:54 PM, William Kang weliam.cl...@gmail.com wrote: Hi, Can somebody explain briefly how Hbase locate a row internally by using Get g = new Get(Bytes.toBytes(RowID)); table.get(g); What type of searching algorithm Hbase use to locate the rows ordering lexicographically? Many thanks! William
Re: Number of column families vs Number of column family qualifiers
Yes. I agree. OOME unlikely. I misinterpreted my current problem. I found, that this (gc timeout) on my 0.89-stumpbleupon hbase occurs only if writeToWAL=false. My RS eats all available memory (5GB), but don't get OOME. I try ti figure out what is going on. Long GC pauses happens for many different reasons, first make sure that your IO, CPU, and RAM aren't over committed and that there's no swap. Hm.. How I can flush family from client side? I don't see any api in 0.20.x. Is it 0.89 api changes? (don't dig into 0.89 yet). You can't, I was talking about a possible fix in the code. Sorry for wrong information. No problem :) J-D
hbase.client.retries.number
HBase was seamless for first couple of weeks..now all kinds of issues in production :) fun fun.. Curious ..does this property have to match up on hbase client side region server side.. I've this number set to 0 on region server side default on client side.. I can't do any put (new) thanks venkatesh
Re: hbase.client.retries.number
BTW..get this exception while trying a new put.. Also, get this exception on gets on some region servers org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server Some server, retryOnlyOne=true, index=0, islastrow=true, tries=9, numtries=10, i=0, listsize=1, region=user_activity,1286789413060_atanackovics_30306_4a3e0812,1286789581757 for region user_activity,1286789413060_30306_4a3e0812,1286789581757, row '1286823659253_v6_1_df34b22f', but failed after 10 attempts. Exceptions: org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1149) org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1230) org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:666) org.apache.hadoop.hbase.client.HTable.close(HTable.java:682) com.aol.mail.antispam.Profiler.notifyEmailSendActivity.processGetRequest(notifyEmailSendActivity.java:363) com.aol.mail.antispam.Profiler.notifyEmailSendActivity.doGet(notifyEmailSendActivity.java:450) javax.servlet.http.HttpServlet.service(HttpServlet.java:617) javax.servlet.http.HttpServlet.service(HttpServlet.java:717) -Original Message- From: Venkatesh vramanatha...@aol.com To: user@hbase.apache.org Sent: Mon, Oct 11, 2010 2:35 pm Subject: hbase.client.retries.number HBase was seamless for first couple of weeks..now all kinds of issues in production :) fun fun.. Curious ..does this property have to match up on hbase client side region server side.. I've this number set to 0 on region server side default on client side.. I can't do any put (new) thanks venkatesh
HBase 0.89.20100726 with unmanaged zookeeper fails to start
We're using a pre-existing zookeeper cluster (HBASE_MANAGES_ZK=false),
and trying to port some code from 0.20 to 0.89, but hbase fails
to start with
Couldnt start ZK at requested address of 2181 [..blah..] 2182
(from ./src/main/java/org/apache/hadoop/hbase/master/HMaster.java)
Because port 2181 is already running our separate zk instance.
I commented out (and used if(false)) to work around the issue
and now hbase shell appears to work (see below).
Is anyone else seeing this problem? or does anyone know if it's repaired
in trunk or being worked on, before I file a ticket?
Thanks,
/charles
PS. Around line HMaster.java:1291 (0.89.20100726)
if (cmd.getArgList().contains(start)) {
try {
// Print out vm stats before starting up.
RuntimeMXBean runtime = ManagementFactory.getRuntimeMXBean();
if (runtime != null) {
LOG.info(vmName= + runtime.getVmName() + , vmVendor= +
runtime.getVmVendor() + , vmVersion= + runtime.getVmVersion());
LOG.info(vmInputArguments= + runtime.getInputArguments());
}
// If 'local', defer to LocalHBaseCluster instance. Starts master
// and regionserver both in the one JVM.
if (LocalHBaseCluster.isLocal(conf)) {
if (false) {
final MiniZooKeeperCluster zooKeeperCluster =
new MiniZooKeeperCluster();
File zkDataPath = new
File(conf.get(hbase.zookeeper.property.dataDir));
int zkClientPort =
conf.getInt(hbase.zookeeper.property.clientPort, 0);
if (zkClientPort == 0) {
throw new IOException(No config value for
hbase.zookeeper.property.clientPort);
}
zooKeeperCluster.setTickTime(conf.getInt(hbase.zookeeper.property.tickTime,
3000));
zooKeeperCluster.setClientPort(zkClientPort);
int clientPort = zooKeeperCluster.startup(zkDataPath);
if (clientPort != zkClientPort) {
String errorMsg = Couldnt start ZK at requested address of +
zkClientPort + , instead got: + clientPort + . Aborting.
Why? +
Because clients (eg shell) wont be able to find this ZK
quorum;
System.err.println(errorMsg);
throw new IOException(errorMsg);
}
conf.set(hbase.zookeeper.property.clientPort,
Integer.toString(clientPort));
}
// Need to have the zk cluster shutdown when master is shutdown.
// Run a subclass that does the zk cluster shutdown on its way out.
LocalHBaseCluster cluster = new LocalHBaseCluster(conf, 1,
LocalHMaster.class, HRegionServer.class);
//
((LocalHMaster)cluster.getMaster()).setZKCluster(zooKeeperCluster);
cluster.startup();
} else {
HMaster master = constructMaster(masterClass, conf);
if (master.shutdownRequested.get()) {
LOG.info(Won't bring the Master up as a shutdown is requested);
return;
}
master.start();
}
} catch (Throwable t) {
LOG.error(Failed to start master, t);
System.exit(-1);
}
Re: Hbase rollback..
Thanks for clarifying this. But on the other hand, wow... that means that even I like the consistency enhancement in 0.90, I can not enjoy it if I have started using HBase 0.20 on a production? On Thu, Sep 16, 2010 at 10:49 PM, Stack st...@duboce.net wrote: On Thu, Sep 16, 2010 at 10:22 PM, Todd Lipcon t...@cloudera.com wrote: On Thu, Sep 16, 2010 at 8:51 PM, Stack st...@duboce.net wrote: What about region ID naming changes? I don't think the new region IDs would work in 0.20. True. So no going back from a 0.89+ to a 0.20 because format of region names in filesystem has changed. St.Ack
Re: Hbase rollback..
That is correct. But we are confident with the new durability changes and other things 0.90 will be safer and faster than 0.20.6. On Oct 11, 2010 4:51 PM, Sean Bigdatafun sean.bigdata...@gmail.com wrote: Thanks for clarifying this. But on the other hand, wow... that means that even I like the consistency enhancement in 0.90, I can not enjoy it if I have started using HBase 0.20 on a production? On Thu, Sep 16, 2010 at 10:49 PM, Stack st...@duboce.net wrote: On Thu, Sep 16, 2010 at 10:22 PM, Todd Lipcon t...@cloudera.com wrote: On Thu, Sep 16, 2010 at 8:51 PM, Stack st...@duboce.net wrote: What about region ID naming changes? I don't think the new region IDs would work in 0.20. True. So no going back from a 0.89+ to a 0.20 because format of region names in filesystem has changed. St.Ack
HLog and durability question --0.90 and 0.20
Can someone give me a detailed look at the HLog mechanism for 0.90 durablity? I recall that HBase committers claim that data will be truly durable in 0.90 after the client gets 'ok' acknowledgement from server, while it was not true in 0.20 (i.e., HBase may have the chance to lose the data even it says 'ok, I got it' to clients). Can someone give me some detailed info? First, HLog can't seem to be flushed onto disk for each Put operation for performance reason (i.e., there exists time window when the log of a certain amout of hlog reside in the RegionServer's memory). If this understanding is correct, how are we going to assure total durablity? Secondly, would anyone share the change related to this between 0.90 and 0.20? Sean On Mon, Oct 11, 2010 at 1:58 PM, Ryan Rawson ryano...@gmail.com wrote: That is correct. But we are confident with the new durability changes and other things 0.90 will be safer and faster than 0.20.6. On Oct 11, 2010 4:51 PM, Sean Bigdatafun sean.bigdata...@gmail.com wrote: Thanks for clarifying this. But on the other hand, wow... that means that even I like the consistency enhancement in 0.90, I can not enjoy it if I have started using HBase 0.20 on a production? On Thu, Sep 16, 2010 at 10:49 PM, Stack st...@duboce.net wrote: On Thu, Sep 16, 2010 at 10:22 PM, Todd Lipcon t...@cloudera.com wrote: On Thu, Sep 16, 2010 at 8:51 PM, Stack st...@duboce.net wrote: What about region ID naming changes? I don't think the new region IDs would work in 0.20. True. So no going back from a 0.89+ to a 0.20 because format of region names in filesystem has changed. St.Ack
Re: HBase cluster with heterogeneous resources
On Sun, Oct 10, 2010 at 12:28 PM, Abhijit Pol a...@rocketfuel.com wrote: Thanks Stack. I think we have GC under control. We have CMS tunned to start early and don't see slept x longer y in logs anymore. We also have higher zk timeout (150 seconds), guess can bump that up a bit. I was able to point to swap on couple of RSs. Will disable the swap and see how that helps suicides. We observed RSs on machines with swap disabled doing very good so far. Disabling SWAP at the OS level (i.e., resize /swap to zero)? If we control the sum of each JVM's heasize to be under the total physical memory size, does the disabling-or-not make any difference? Thanks. e.g., RS(6GB) + DN (2GB) + TaskTracker (1GB) +... 16GB seems to be quite possible, does not it? Lots of people mentioned that we should give a lot of physical memory to RegionServer machine. But I'd like to ask a more detailed memory allocation breakdown because in practice, RS runs along with other service like Znode, Datanode, TaskTracker and etc. BTW, for on bully machines, how much heapsize have you allocated to RegionServer? Also, as you suggested we will take odd man out. We don't have to have it in. Our master is already low key machine. --Abhi On Sat, Oct 9, 2010 at 11:12 PM, Stack st...@duboce.net wrote: On Sat, Oct 9, 2010 at 1:15 PM, Abhijit Pol a...@rocketfuel.com wrote: We are testing with 4 nodes HBase cluster out of which 3 machines are identical with 64GB RAM and 6x1TB disks. and 4th machine has only 16GB RAM and 2x1TB disks We observe (from server side metrics) frequent latency spikes and RS suicide ~ every 8hrs from our 4th machine. How much heap have you given your servers? You could up your zk timeout of play with GC tunings -- if full GC the reason RSs are committing hari-kari. We do have overall heap size configured based on total RAM available but all other configs are same across RSs Is there a way to hint master to distribute regions based on available resources? No. Not currently. We are using 0.89.20100924 branch. We have flop at default 0.3 and roughly equal number of regions across all RSs. I'd suggest taking the odd-man-out out of your cluster or repurposing it as a master node. Usually clusters are homogeneous and much of the software assumes each node equivalent. We've not had a chance to work on clusters made of differently spec'd machines. St.Ack
Re: Bulk import tools for HBase
Another potential problem of incremental bulk loader is that the number of reducers (for the bulk loading process) needs to be equal to the existing regions -- this seems to be unfeasible for very large table, say with 2000 regions. Any comment on this? Thanks. Sean On Fri, Oct 8, 2010 at 9:03 PM, Todd Lipcon t...@cloudera.com wrote: What version are you building from? These tools are new as of this past june. -Todd On Fri, Oct 8, 2010 at 4:52 PM, Leo Alekseyev dnqu...@gmail.com wrote: We want to investigate HBase bulk imports, as described on http://hbase.apache.org/docs/r0.89.20100726/bulk-loads.html and and/or JIRA HBASE-48. I can't seem to run either the importtsv tool or the completebulkload tool using the hadoop jar /path/to/hbase-VERSION.jar command. In fact, the ImportTsv class is not part of that jar file. Am I looking in the wrong place for this class, or do I need to somehow customize the build process to include it?.. Our HBase was built from source using the default procedure. Thanks for any insight, --Leo -- Todd Lipcon Software Engineer, Cloudera
Question regarding data location in hdfs after hbase restarts
hi, all I set hdfs replica=1 when running hbase. And DN and RS co-exists on each slave node. So the data in the regions managed by RS will be stored on its local data node, rite? But when I restart hbase and hbase client does gets on RS, datanode will read data from remote data nodes. Does that mean when RS restart, the regions are re-arranged? If so, will hbase is clever enough to re-adjust the regions? I'm not clear about the behind mechanism so anyone can give me some explanations? Thanks.
Increase region server throughput
I would like to tune region server to increase throughput..On a 10 node cluster, I'm getting 5 sec per put. (this is unbatched/unbuffered). Other than region server handler count property is there anything else I can tune to increase throughput? ( this operation i can't use buffered write without code change) thx venkatesh
Re: Number of column families vs Number of column family qualifiers
Yes this is spot on. When hbase scans we read a block, iterate through the keys in the block then goes to the next block. We try to be as efficient as possible, but the inescapable fact remains we must read all the intervening data. We can do tricks (in 0.90) to use the block index to skip some blocks, but it is not always possible. On Oct 11, 2010 5:01 PM, Sean Bigdatafun sean.bigdata...@gmail.com wrote: I think this is a good suggestion too. HBase linearly scans through the 64KB that is bring to memory. If big data payload (yet unused in a query/scan) is mixed with small data payload, it will be rather ineffective, I think? On Mon, Oct 11, 2010 at 9:43 AM, Ryan Rawson ryano...@gmail.com wrote: The reason I talk about value size is one area where multiple families are good is when you have really large values in one column and smaller values in different columns. So if you want to just read the small values without scanning through the big values you can use separate column families. -ryan On Mon, Oct 11, 2010 at 9:32 AM, Jean-Daniel Cryans jdcry...@apache.org wrote: Yes. I agree. OOME unlikely. I misinterpreted my current problem. I found, that this (gc timeout) on my 0.89-stumpbleupon hbase occurs only if writeToWAL=false. My RS eats all available memory (5GB), but don't get OOME. I try ti figure out what is going on. Long GC pauses happens for many different reasons, first make sure that your IO, CPU, and RAM aren't over committed and that there's no swap. Hm.. How I can flush family from client side? I don't see any api in 0.20.x. Is it 0.89 api changes? (don't dig into 0.89 yet). You can't, I was talking about a possible fix in the code. Sorry for wrong information. No problem :) J-D
Re: Question regarding data location in hdfs after hbase restarts
We don't attempt to optimize region placement with hdfs locations yet. A reason why is because on a long lived cluster compactions create the locality you are looking for. Furthermore, in the old master such an optimization was really hard to do. The new master should make it easier to write such 1 off hacks. On Oct 11, 2010 9:43 PM, Tao Xie xietao.mail...@gmail.com wrote: hi, all I set hdfs replica=1 when running hbase. And DN and RS co-exists on each slave node. So the data in the regions managed by RS will be stored on its local data node, rite? But when I restart hbase and hbase client does gets on RS, datanode will read data from remote data nodes. Does that mean when RS restart, the regions are re-arranged? If so, will hbase is clever enough to re-adjust the regions? I'm not clear about the behind mechanism so anyone can give me some explanations? Thanks.
