If a server crashes, its regions will be moved to other servers. So, if a
query fails because the region server does not respond, the hbase client
will retry as well. The principle is to make the failure transparent to the
application using the hbase client.
Le 3 déc. 2012 03:53, Jean-Marc
He could have meant that the region was unavailable for a longer duration.
My take is that you could have limited bandwidth, and a very large region on
the move.
While I don't imagine that it could take that long to move a region, you never
know how things are tuned and you could in theory hit
Thanks all for your replies.
So, to reply to all in one.
I'm not using CD3. I'm using Hadoop 1.0.3 and HBase 0.94.2 directly
from the JARs.
Here are all the places where I have put the lib:
/home/hadoop/hadoop-1.0.3/lib/native/Linux-amd64-64/libsnappy.so
Ok
This: http://code.google.com/p/hadoop-snappy/issues/detail?id=2 helped
me and my test program is now working. I'm able to load both
libraries. Fine.
But the CompressionTest is still not working.
What is very strange is that:
12/12/03 08:44:24 WARN snappy.LoadSnappy: Snappy native library
Hey Mike,
A move is logical in terms of a region and region servers (i.e. only
an ID and a command is passed around), but yes the location of blocks
could impact its performance a bit (until a further compaction), but
would not prevent availability.
On Mon, Dec 3, 2012 at 3:53 PM, Michel Segel
There is a compression test JIRA right now. What are you seeing?
On Mon, Dec 3, 2012 at 8:47 AM, Jean-Marc Spaggiari jean-m...@spaggiari.org
wrote:
Ok
This: http://code.google.com/p/hadoop-snappy/issues/detail?id=2 helped
me and my test program is now working. I'm able to load both
Hey JM,
Sorry for the quick message earlier. I tracked down the JIRA I was
referring to: https://issues.apache.org/jira/browse/HBASE-7080
Does this look like what you are seeing in Compression test?
On Mon, Dec 3, 2012 at 9:09 AM, Kevin O'dell kevin.od...@cloudera.comwrote:
There is a
Hi Kevin,
Thanks for the clarification.
No, it's not what I'm seeing.
Here is what I'm getting:
12/12/03 09:40:42 WARN snappy.LoadSnappy: Snappy native library is available
12/12/03 09:40:42 WARN snappy.LoadSnappy: Snappy native library not loaded
Exception in thread main
Hi,
I'm wondering who is taking care of the data locality. Is it hadoop? Or hbase?
Let's say I have disabled the load balancer and I'm manually moving a
region to a specific server. Who is going to take care that the data
is going to be on the same datanode as the regionserver I moved the
region
JM,
If you have disabled the balancer and are manually moving regions, you
will need to run a compaction on those regions. That is the only(logical)
way of bringing the data local. HDFS does not have a concept of HBase
locality. HBase locality is all managed through major and minor
Ok, I got it
I had to copy the hadoop native libs into the hbase native libs directory!
Now I get a SUCCESS when I'm doint the CompressionTest...
I'm not 100% sure that it's the only think which was missing because I
have done so many modifications in the last 3 days...
So I will start
Ok. So each time I move a region manually, I'm better to run a minor
compaction to make sure data is moved locally too.
Got it.
Thanks,
JM
2012/12/3, Kevin O'dell kevin.od...@cloudera.com:
JM,
If you have disabled the balancer and are manually moving regions, you
will need to run a
Never say die!
On Mon, Dec 3, 2012 at 10:15 AM, Jean-Marc Spaggiari
jean-m...@spaggiari.org wrote:
Ok, I got it
I had to copy the hadoop native libs into the hbase native libs directory!
Now I get a SUCCESS when I'm doint the CompressionTest...
I'm not 100% sure that it's the only
I would argue to running a major compact on the individual region. Should
be quick seeing as how you are on .94.
On Mon, Dec 3, 2012 at 10:16 AM, Jean-Marc Spaggiari
jean-m...@spaggiari.org wrote:
Ok. So each time I move a region manually, I'm better to run a minor
compaction to make sure
Hi there-
This is also discussed in the Regions section in the RefGuide:
http://hbase.apache.org/book.html#regions.arch
9.7.3. Region-RegionServer Locality
On 12/3/12 10:08 AM, Kevin O'dell kevin.od...@cloudera.com wrote:
JM,
If you have disabled the balancer and are manually moving
Ok. I retried and on brand new 0.94.3 installation and the only 2
things required are the libsnappy.so and the libhadoop.so files... And
it'S taking 5 minutes to install ;)
I will now deploy that all over the cluster and give snappy+0.94.3RC a try...
2012/12/3, Kevin O'dell
Ok. I will try the major compaction then ;)
Doug, thanks for pointing to the doc! I now totally understand why
it's moved locally when the compaction occurs!
Thanks all! I will give that a try very shortly.
JM
2012/12/3, Doug Meil doug.m...@explorysmedical.com:
Hi there-
This is also
Thanks all for your comments.
I was talking about the 3 minutes period because by default, ZK will
have a 3 minute timeout. So if a server crash, it can take up to 3
minutes to Zookeeper to detect that. And in this meantime, the client
might not be able to access the impacted regions.
So what I
I got stuck like you before trying to setup snappy on hbase. Here what I
recall I did from memory:
I think you need to copy the snappy libs inside the hadoop lib/native, then
point hbase to it.
So in hbase/conf/hbase.env , you should have something similar to this:
export
Hi,
Something more about my workaround last time:
I used the following steps to test my workaround:
1) cd $HBASE_HOME
./bin/hbase org.apache.hadoop.hbase.util.CompressionTest file:///tmp/testfile
lzo
2) hbase shell
create 't1', {NAME = 'cf1', COMPRESSION = 'LZO'}
You could modify above
Hi JM,
I had experienced similar error when I was installing LZO compression to
RegionServers:
Below is from my record about installing LZO:
Issue:
java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
...
12/11/23 19:03:14 ERROR lzo.LzoCodec: Cannot load native-lzo without
Any chance of an update to
http://hbase.apache.org/book.html#snappy.compression ? If someone writes
it up, I'll stitch it in. Thanks,
St.Ack
On Mon, Dec 3, 2012 at 6:29 AM, a...@hsk.hk a...@hsk.hk wrote:
Hi,
Something more about my workaround last time:
I used the following steps to test
Sure I will.
JM
2012/12/3, Stack st...@duboce.net:
Any chance of an update to
http://hbase.apache.org/book.html#snappy.compression ? If someone writes
it up, I'll stitch it in. Thanks,
St.Ack
On Mon, Dec 3, 2012 at 6:29 AM, a...@hsk.hk a...@hsk.hk wrote:
Hi,
Something more about my
Done. HBASE-7264 created. I have added few more details about the
steps to follow to install Snappy on HBase 0.94.x.
I don't know how to build the documentation locally, so I'm not 100%
sure about thhe XML structure. I did cutpast so it should be good...
JM
2012/12/3, Jean-Marc Spaggiari
Hi,
I have a schema where the rows are 8 bytes long and the columns are 12
bytes long (roughly 1000 columns per row). The value is 0 bytes. Is this
going to be space inefficient in terms of HFile size (large index + blocks)
? The total key size, as far as i know, would be 8 + 12 + 8 (timestamp) =
Thank you JM. Let me fold it in
St.Ack
On Mon, Dec 3, 2012 at 11:48 AM, Jean-Marc Spaggiari
jean-m...@spaggiari.org wrote:
Done. HBASE-7264 created. I have added few more details about the
steps to follow to install Snappy on HBase 0.94.x.
I don't know how to build the documentation
Regards, Varun.
I think that you can see the Bernoit Sigoure (@tsuna)´s talk called
Lessons learned from OpenTSDB in the last
HBaseCon . [1]
He explained in great detail how to design your schema to obtain the
best performance from HBase.
Other recommended talks are: HBase Internals from
Hi Marcos,
Thanks for the links. We have gone through these and thought about the
schema. My question is about whether using PrefixDeltaEncoding makes sense
in our situation...
Varun
On Mon, Dec 3, 2012 at 12:36 PM, Marcos Ortiz mlor...@uci.cu wrote:
Regards, Varun.
I think that you can see
Hi,
Has anybody tried to run multiple RegionServers on a single physical
node? Are there deep technical issues or minor impediments that would
hinder this?
We are trying to do this because we are facing a lot of GC pauses on the
large heap sizes (~70G) that we are using, which leads to a lot of
The HBase Team is pleased to announce the release of HBase 0.94.3.
Download it from your favorite Apache mirror [1].
HBase 0.94.3 is a bug fix release and has 51 issues resolved against it,
including some performance improvements.
0.94.3 is the current stable release of HBase.
All previous
+1.
Thank you for the hard work guys.
Regards,
Mohammad Tariq
On Tue, Dec 4, 2012 at 3:12 AM, lars hofhansl lhofha...@yahoo.com wrote:
The HBase Team is pleased to announce the release of HBase 0.94.3.
Download it from your favorite Apache mirror [1].
HBase 0.94.3 is a bug fix release
Regards, Ishan.
On 12/03/2012 04:39 PM, Ishan Chhabra wrote:
Hi,
Has anybody tried to run multiple RegionServers on a single physical
node? Are there deep technical issues or minor impediments that would
hinder this?
Can you provide more information about your setup?
- Network
- Disk schema
-
Hi there,
Not tried multi-RS on a single node, but have you looked at the off-heap
cache? It's a part of 0.92.x. From what I understand that feature was
designed with this case in mind (I.e., trying to do a lot of caching, but
don't want to introduce GC issues in RS).
Hi,
I'm wondering, what's the best way to know which RegionServer a region
is hosted on.
I looked at the table.jsp but it's using an HMaster from the session
and I did not find how it's initialized.
From the HBaseAdmin I can get the regions for one table, but I don't
have the RegionServer
I think all is clear now.. Just to conclude, the data locality is feature
provided by HDFS. When DFS client writes some data, hadoop will try to maintain
the data locality. HBase region server writes and reads data via the DFS client
which is in the same process as that of the RS. When the
Hi
Actually you can try using HTable.getRegionLocations(). But you need to
pass the table name for it.
Regards
Ram
On Tue, Dec 4, 2012 at 6:34 AM, Jean-Marc Spaggiari jean-m...@spaggiari.org
wrote:
Hi,
I'm wondering, what's the best way to know which RegionServer a region
is hosted on.
Hi,
I am using hbase 0.94.0 and am using the HTablePool - Reusable typre with a
pool size of 50. I have a lot of threads using the htable pool concurrently
(~ 3500) - The client side timeout is 5 seconds and the threads start okay
producing good QPS to the hbase cluster, finally the QPS drops
Hi Varun
It looks to be very clear that you need to use some sort of
encoding scheme. PrefixDeltaEncoding would be fine may be.. You can see the
other algos also like the FastDiff... and see how much space it can save in
your case. Also suggest you can use the encoding for
Okay - this was contention issue -
https://issues.apache.org/jira/browse/HBASE-2939 solves the issue - upping
the IPC pool size. Thanks !
On Mon, Dec 3, 2012 at 8:04 PM, Varun Sharma va...@pinterest.com wrote:
Hi,
I am using hbase 0.94.0 and am using the HTablePool - Reusable typre with
a
Hi Anoop,
I agree - I am not so concerned about the savings on disk - rather I am
thinking about the savings inside the block cache. I am not sure how stable
PrefixDeltaEncoding is and who else uses it. If not, are there people using
FastDiff encoding - it seems like any form of encoding scheme
Hi All,
We would like to integrate HBase with our corporate Active Directory for
authentication and authorization of HBase users. I have gone through the
following links in HBase:
http://blog.cloudera.com/blog/2012/09/understanding-user-authentication-and-authorization-in-apache-hbase/
good ! never surrender !
you also could have done like I posted you yesterday which is equivalent:
in hbase-env.sh:
export HBASE_LIBRARY_PATH=/pathtoyourhadoop/lib/native/Linux-amd64-64
in /pathtoyourhadoop/lib/native/Linux-amd64-64 I have added:
libsnappy.a
libsnappy.so
libsnappy.so.1
42 matches
Mail list logo
