Hi Ted,
We currently use this tool in the scenario where data is consumed by
MapReduce jobs, so we haven't tested the performance of pure distributed
scan (i.e. N scans instead of 1) a lot. I expect it to be close to simple
scan performance, or may be sometimes even faster depending on your data
On Sat, Feb 12, 2011 at 7:13 AM, Jason Rutherglen
jason.rutherg...@gmail.com wrote:
solr/katta/elasticsearch
These don't have a distributed solution for realtime search [yet].
Sorry if this is a naive question but can you explain why you consider
that ElasticSearch isn't a distributed solution
Hi everybody,
I am desiging a two-tiers comment system like facebook: the system
inlcuded main comments and each main comment has a various number of
sub-comments. My schema is as follow:
table comments
family data:
data:content - content of main comment
data:uid - uid of
Hurry up! Register now, Get Two full days of training prior to the conference
May 23-24 at a special conference rate only on Lucene Revolution 2011
http://us.ootoweb.com/luceneregistration
--
View this message in context:
http://old.nabble.com/Lucene-Revolution-2011-tp31439963p31439963.html
Alex:
If you read this, you would know why I asked:
https://issues.apache.org/jira/browse/HBASE-3679
I need to deal with normal scan and distributed scan at server side.
Basically bucketsCount may not equal number of regions for the underlying
table.
Cheers
On Tue, Apr 19, 2011 at 11:11 PM,
That's some old email :)
I think what Jason is doing is not so much about trying to get (N)RT search
(which already exists in raw Lucene, in ES, in Zoie, Sensei, and eventually
will
be in Solr), but trying to get full-text search via Lucene tightly integrated
with data storage via HBase.
My problem is as the following:
http://stackoverflow.com/questions/4790029/hbase-column-wide-scanning-and-fetching
Let's say i've created a table
rowkey (attrId+attr_value) //compound key
column = doc:doc1, doc:doc2, ...[the qualifier is variable, which depends
on the value]
when use scan
Sorry to bump this, but we could really use a hand here. Right now, we have
a very hard time seeing repeatable read/write consistency. Any suggestions
are welcome.
-GS
On Tue, Apr 19, 2011 at 3:08 PM, George P. Stathis gstat...@traackr.comwrote:
Hi all,
In this chapter of our 0.89 to 0.90
Yep. In all benchmarks response times for tiny data start at about 1-2ms but
not in our new setup. Which is why I am at loss where to start looking.
Seems like a network congestion but it can't be. Its a barebone setup and
admins tell me they have tested it for performance.
apologies for brevity.
I am having similar results but hadn't done enough testing yet.
Sent from my BlackBerry® wireless device
-Original Message-
From: Dmitriy Lyubimov dlie...@gmail.com
Date: Wed, 20 Apr 2011 08:09:29
To: user@hbase.apache.org
Reply-To: user@hbase.apache.org
Subject: Re: 0.90 latency
I have seen this before.
HTable isn't thread-safe.
Please describe your usage.
Thanks
On Wed, Apr 20, 2011 at 6:03 AM, Venkatesh vramanatha...@aol.com wrote:
Using hbase-0.90.2..(sigh..) Any tip? thanks
java.lang.IndexOutOfBoundsException: Index: 4, Size: 3
at
Are you sharing a single HTable between multiple threads that do puts?
J-D
On Wed, Apr 20, 2011 at 6:03 AM, Venkatesh vramanatha...@aol.com wrote:
Using hbase-0.90.2..(sigh..) Any tip? thanks
java.lang.IndexOutOfBoundsException: Index: 4, Size: 3
at
I think HConnectionManager can catch IndexOutOfBoundsException and translate
into a more user-friendly message, informing user about thread-safety.
On Wed, Apr 20, 2011 at 9:11 AM, Ted Yu yuzhih...@gmail.com wrote:
I have seen this before.
HTable isn't thread-safe.
Please describe your
Hey George,
Sorry for the late answer, there's nothing that comes to mind when
reading your email.
HBASE_SLAVE_SLEEP is only used by the bash scripts, like when you do
hbase-daemons.sh it will wait that sleep time between each machine.
Would you be able to come up with a test that shows the
It looks like you keep up a second index table ordered by last post
(row key will have a reversed timestamp of the post's date).
What are the queries you are going to make against the table?
St.Ack
On Tue, Apr 19, 2011 at 11:53 PM, Bui Ngoc Son gemmountain...@gmail.com wrote:
Hi everybody,
I
I guess George's case has something to do with pseudo-clustered mode.
On Wed, Apr 20, 2011 at 9:27 AM, Jean-Daniel Cryans jdcry...@apache.orgwrote:
Hey George,
Sorry for the late answer, there's nothing that comes to mind when
reading your email.
HBASE_SLAVE_SLEEP is only used by the bash
Ok. Let me ask a question.
When scan is performed and it obviously covers several regions, are
scan performance calls done in sinchronous succession or they are done
in parallel?
Assuming scan is returning 40 results but for some weird reason it
goes to 6 regions and caching is set to 100 (so it
On Wed, Apr 20, 2011 at 12:27 PM, Jean-Daniel Cryans jdcry...@apache.orgwrote:
Hey George,
Sorry for the late answer, there's nothing that comes to mind when
reading your email.
HBASE_SLAVE_SLEEP is only used by the bash scripts, like when you do
hbase-daemons.sh it will wait that sleep
Hi,
I would like to test the new Coprocessor feature and overwrite the prePut
function of the the RegionOberserver.
Unfortunately the function is never executed... I am using the current snapshot
from the repository.
Can anyone point me into the right direction? Or is there any way to check
On Tue, Apr 19, 2011 at 12:08 PM, George P. Stathis
gstat...@traackr.com wrote:
We have several unit tests that have started mysteriously failing in random
ways as soon as we migrated our EC2 CI build to the new 0.90 CDH3. Those
tests used to run against 0.89 and never failed before. They also
Ted, what makes you say that? Have you seen similar issues in
pseudo-clustered mode? We have been running in that mode on our dev
environment for a year now, we haven't had any issues like this before. At
any rate, I'll set it to standalone just in case to see if it makes a
difference.
On Wed,
On Wed, Apr 20, 2011 at 9:49 AM, Dmitriy Lyubimov dlie...@gmail.com wrote:
Ok. Let me ask a question.
When scan is performed and it obviously covers several regions, are
scan performance calls done in sinchronous succession or they are done
in parallel?
The former.
Assuming scan is
On Tue, Apr 19, 2011 at 4:46 PM, Dmitriy Lyubimov dlyubi...@apache.org wrote:
Right now i am shooting scans returning between 3 and 40 rows and
regardless of data size, approximately 500-400 QPS. The data tables
are almost empty and in-memory, so they surely should fit in those 40%
heap
Hi,
I just bounced a region server. How do I start just this one region server
and make sure it rejoins the cluster?
Thanks
-Pete
Yeah you J-D both hit it..
I knew it's bad..I was trying anything everything to solve the incredibly
long latency
with hbase puts on 0.90.2..
I get ok/better response with batch put.. this was quick dirty way to
accumulate puts by sharing
same HTable instance
Thanks for letting me
Got it. This must be the reason. Cause it is a laugh check, and i do
see 6 regions for 40 rows so it can span them, although i can't
confirm it for sure. It may be due to how table was set up or due to
some time running them and rotating some data there. The uniformly
distributed hashes are used
If the HTable is not thread safe how would one use multiple threads to
read/write to a table?
Thanks
-Pete
When using HTablePool, try not to define maxSize yourself - use the default.
On Wed, Apr 20, 2011 at 10:04 AM, Venkatesh vramanatha...@aol.com wrote:
Yeah you J-D both hit it..
I knew it's bad..I was trying anything everything to solve the incredibly
long latency
with hbase puts on
Each thread should maintain its own HTable instance.
On Wed, Apr 20, 2011 at 10:08 AM, Peter Haidinyak phaidin...@local.comwrote:
If the HTable is not thread safe how would one use multiple threads to
read/write to a table?
Thanks
-Pete
Sorry, my bad, I assumed each thread would have its own Table instance not
using a shared instance.
Thanks
-Pete
-Original Message-
From: Ted Yu [mailto:yuzhih...@gmail.com]
Sent: Wednesday, April 20, 2011 10:10 AM
To: user@hbase.apache.org
Subject: Re: HTable not thread safe
Each
If I use default ..i can't share/pass my HBaseConfiguration object..atleast i
don't see a constructor/setter..
that would go against previous suggestion
-Original Message-
From: Ted Yu yuzhih...@gmail.com
To: user@hbase.apache.org
Sent: Wed, Apr 20, 2011 1:08 pm
Subject:
On Wed, Apr 20, 2011 at 10:06 AM, Dmitriy Lyubimov dlie...@gmail.com wrote:
Got it. This must be the reason. Cause it is a laugh check, and i do
see 6 regions for 40 rows so it can span them, although i can't
confirm it for sure.
Run a scan from the shell:
hbase scan 'YOUR_TABLE'
... and
I meant specifying Integer.MAX_VALUE as maxSize along with config.
On Wed, Apr 20, 2011 at 10:17 AM, Venkatesh vramanatha...@aol.com wrote:
If I use default ..i can't share/pass my HBaseConfiguration object..atleast
i don't see a constructor/setter..
that would go against previous suggestion
On the host that is carrying the regionserver do:
./bin/hbase-daemon.sh stop regionserver
Then start it again.
Or, since 0.90.2, see bin/graceful_stop.sh. It will let you do a
gradual decommission optionally restarting the regionserver after the
regions have been gently offloaded and then
On Wed, Apr 20, 2011 at 5:23 AM, Gan, Xiyun ganxi...@gmail.com wrote:
when use scan feature, i would fetch 1 row every time inside iterator, what
if the column qualifier reach millions entries. how do you loop through
that, and will there be a cache issue?
You do in-row scan. You set an
sorry..yeah..that's dumb of me..clearly i'm not thinking anything..just
frustrated with upgrade
thx
-Original Message-
From: Ted Yu yuzhih...@gmail.com
To: user@hbase.apache.org
Sent: Wed, Apr 20, 2011 1:24 pm
Subject: Re: java.lang.IndexOutOfBoundsException
I meant
On Tue, Apr 19, 2011 at 11:58 AM, Venkatesh vramanatha...@aol.com wrote:
I was hoping that too..
I don't have scripts to generate # requests from shell..I will try that..
Did you try it?
Above you seem to say that a simple put of 100 bytes takes 2 seconds
where in 0.20.6 it took 10
Thanks
-Pete
-Original Message-
From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of Stack
Sent: Wednesday, April 20, 2011 10:23 AM
To: user@hbase.apache.org
Subject: Re: Restarting a Region Server
On the host that is carrying the regionserver do:
./bin/hbase-daemon.sh
shell is no problems..ones/twos..i've tried mass puts from shell
we cant handle our production load (even 1/3 of it)
700 mill per day is full load..same load we handled with absolutely no issues
in 0.20.6..
there is several pause between batch of puts as wel
-Original
Hi Joerg,
Try changing the table attribute name to COPROCESSOR$1 -- it is currently
case sensitive (we should probably change that).
After doing that, look for lines like the following in the region server
log:
Load coprocessor ... from HTD of tablename successfully.
or
attribute '...' has
Want to paste your configuration up in pastebin?
Is that 700million puts a day?
Remind us of your cluster size.
Paste some of a regionserver log too. That can be informative.
St.Ack
On Wed, Apr 20, 2011 at 10:41 AM, Venkatesh vramanatha...@aol.com wrote:
shell is no
Hi,
I'm now using CDH3u0 at 16nodes cluster (hdp0-hdp15).
The configuraiton is below.
hdp0: zk + master + region + nn + dn + jt + tt
hdp1: zk + master + region + snn + dn + tt
hdp2: zk + region + dn + tt
hdp3 to hdp15: region + dn + tt
Usually, it works really well. But once the user throws
Ok actually we do have 1 region for these exact tables... so back to
square one.
FWIW i do get 8% quartile under 3ms TTLB. So it is algorithmically
sound it seems. question is why outliers spread is so much longer than
in tests on one machine. must be network. What else.
On Wed, Apr 20, 2011 at
What is meant by 8% quartile? 75th %-ile? 98%-ile? Should quartile have
been quantile?
On Wed, Apr 20, 2011 at 12:00 PM, Dmitriy Lyubimov dlie...@gmail.comwrote:
Ok actually we do have 1 region for these exact tables... so back to
square one.
FWIW i do get 8% quartile under 3ms TTLB. So
Hello,
I am seeing reloading region because regionserver didn't accept updates
message in the middle of populating the table and was wondering what could
be wrong with it?
Thank you so much,
8% percentile. sorry. 8% requests do land under 3ms or less.
On Wed, Apr 20, 2011 at 12:06 PM, Ted Dunning tdunn...@maprtech.com wrote:
What is meant by 8% quartile? 75th %-ile? 98%-ile? Should quartile have
been quantile?
On Wed, Apr 20, 2011 at 12:00 PM, Dmitriy Lyubimov
Kazuki-san,
Setting the ZK timeout to a large value will stop the expirations but may not
provide sufficiently fast failure detection for your use case of course.
However if even Ganglia stops working during a large mapreduce job, I think you
need to question the adequacy of the network
On Wed, Apr 20, 2011 at 12:09 PM, Weishung Chung weish...@gmail.com wrote:
Hello,
I am seeing reloading region because regionserver didn't accept updates
message in the middle of populating the table and was wondering what could
be wrong with it?
This is a client-side message right?
On Wed, Apr 20, 2011 at 12:25 PM, Stack st...@duboce.net wrote:
On Tue, Apr 19, 2011 at 4:46 PM, Dmitriy Lyubimov dlyubi...@apache.org
wrote:
Right now i am shooting scans returning between 3 and 40 rows and
regardless of data size, approximately 500-400 QPS. The data tables
are almost empty
I'm upgrading our code base now to HBase 0.90.1 (CDH3-U0) from 0.89.something.
I'm facing a very hard time tracking failures in our unit tests and working
around them. Here's the situation:
1- Many of our unit tests run in-process M/R jobs using the LocalJobRunner.
2- We often do things before
Yes, this is the only stress test running on the cluster and nothing else.
And it all goes to block cache as evidenced by metrics.
2011-04-20 12:28:48,375 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=8.26
MB, free=190.08 MB, max=198.34 MB, blocks=112, accesses=55732209,
If one region only, then its located on a single regionserver. Tail
that regionservers logs. It might tell us something.
St.Ack
On Wed, Apr 20, 2011 at 12:25 PM, Stack st...@duboce.net wrote:
On Wed, Apr 20, 2011 at 12:25 PM, Stack st...@duboce.net wrote:
On Tue, Apr 19, 2011 at 4:46 PM,
On Wed, Apr 20, 2011 at 12:18 PM, Robert Mahfoud
rmahf...@mediosystems.com wrote:
I'm upgrading our code base now to HBase 0.90.1 (CDH3-U0) from 0.89.something.
I'm facing a very hard time tracking failures in our unit tests and working
around them. Here's the situation:
1- Many of our unit
i just enabled debug logging for o.a.h.hbase logger in that particular
region server... so far not much except for LRUBlock cache spitting
metrics ..
2011-04-20 12:28:48,375 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=8.26
MB, free=190.08 MB, max=198.34 MB, blocks=112,
Hello Stack,
Thank you.
You are right, it's on the client side. I was populating 30 batches of
datasets one after another continuously to the same table, each batch
contains 100 rows with ~6 cells in one column family. (Total: 3000 rows
with 6 cells each row) The test cluster consists of
Is there any way to log 'inner' TTLB times the region server incurs for reads?
On Wed, Apr 20, 2011 at 12:43 PM, Dmitriy Lyubimov dlie...@gmail.com wrote:
i just enabled debug logging for o.a.h.hbase logger in that particular
region server... so far not much except for LRUBlock cache spitting
Thanks a lot, case sensitivity was the issue...
Jörg
P.S. Should I open a Jira for that?
-Ursprüngliche Nachricht-
Von: Gary Helmling ghelml...@gmail.com
Gesendet: 20.04.2011 19:49:21
An: user@hbase.apache.org
Betreff: Re: Coprocessor not executing (current Snapshot)
Hi Joerg,
Try
Enable rpc logging. Will show in your ganglia. See metrics article
on hbase home page.
On Wed, Apr 20, 2011 at 12:44 PM, Dmitriy Lyubimov dlie...@gmail.com wrote:
Is there any way to log 'inner' TTLB times the region server incurs for reads?
On Wed, Apr 20, 2011 at 12:43 PM, Dmitriy
On Wed, Apr 20, 2011 at 12:48 PM, Stack st...@duboce.net wrote:
On Tue, Apr 19, 2011 at 12:08 PM, George P. Stathis
gstat...@traackr.com wrote:
We have several unit tests that have started mysteriously failing in
random
ways as soon as we migrated our EC2 CI build to the new 0.90 CDH3.
Dmitriy, what are you seeing on your side? Missing inserts? Deletes that are
never applied? Both?
On Wed, Apr 20, 2011 at 3:04 PM, Dmitriy Lyubimov dlie...@gmail.com wrote:
exactly my symptoms with 0.90.1. What gives.
On Wed, Apr 20, 2011 at 9:54 AM, George P. Stathis gstat...@traackr.com
Yes -- I already looked thru 'regionserver' metrics some time ago in
hbase book. And i am not sure there's a 'inner ttlb' metric.
There are fs latency metrics there but nothing for the respons times.
fs latency is essentially hdfs latency AFAICT and that would not be
relevant to what i am asking
FS latency shouldn't matter with your 99.9% cache hit rate as reported.
On Wed, Apr 20, 2011 at 12:55 PM, Dmitriy Lyubimov dlie...@gmail.comwrote:
Yes -- I already looked thru 'regionserver' metrics some time ago in
hbase book. And i am not sure there's a 'inner ttlb' metric.
There are fs
This is your problem. Sounds like a very deficient switch.
On Wed, Apr 20, 2011 at 11:41 AM, Kazuki Ohta kazuki.o...@gmail.com wrote:
The problem is that shuffle network transfer dominates the switch,
and important zk packets are not transferred properly at that time.
Yes that's what i said. there's metric for fs latency but we are not
hitting it so it's not useful.
Question is which one might be useful to measure inner ttlb, and i
don't see it there.
On Wed, Apr 20, 2011 at 1:14 PM, Ted Dunning tdunn...@maprtech.com wrote:
FS latency shouldn't matter with
On Wed, Apr 20, 2011 at 5:06 AM, Otis Gospodnetic
otis_gospodne...@yahoo.com wrote:
That's some old email :)
Sorry I'm catching up just now :D
I think what Jason is doing is not so much about trying to get (N)RT search
(which already exists in raw Lucene, in ES, in Zoie, Sensei, and
On Wed, Apr 20, 2011 at 10:12 AM, Peter Haidinyak phaidin...@local.com wrote:
Sorry, my bad, I assumed each thread would have its own Table instance not
using a shared instance.
Yeah you'd need to use one HTable instance per thread.
plugAlternatively, you can look at asynchbase, an
On Wed, Apr 20, 2011 at 10:04 AM, Venkatesh vramanatha...@aol.com wrote:
On 0.90.2, do you all think using HTablePool would help with performance
problem?
What performance problems are you seeing?
BTW, if you want a thread-safe client that's highly scalable for
high-throughput, multi-threaded
I'm now using CDH3u0 at 16nodes cluster (hdp0-hdp15).
The configuraiton is below.
hdp0: zk + master + region + nn + dn + jt + tt
hdp1: zk + master + region + snn + dn + tt
hdp2: zk + region + dn + tt
hdp3 to hdp15: region + dn + tt
I would also look at the memory configuration for your
Thanks a lot, case sensitivity was the issue...
Jörg
P.S. Should I open a Jira for that?
Yes, please do. I'm sure others will hit it as well.
--gh
btw, Ted, your version of YCSB in github should show TTLBs, right?
On Wed, Apr 20, 2011 at 1:14 PM, Ted Dunning tdunn...@maprtech.com wrote:
FS latency shouldn't matter with your 99.9% cache hit rate as reported.
On Wed, Apr 20, 2011 at 12:55 PM, Dmitriy Lyubimov dlie...@gmail.comwrote:
Yes
Thanks, I'll do that. I need to redesign my client to be multi-threaded so it's
a good time to look at it.
-Pete
-Original Message-
From: tsuna [mailto:tsuna...@gmail.com]
Sent: Wednesday, April 20, 2011 1:28 PM
To: user@hbase.apache.org
Subject: Re: HTable not thread safe
On Wed, Apr
Yes. In the sense that it measures time until operation is complete
according to the client.
And assuming that TTLB = time to last bit.
YCSB is, however, a frail vessel. I have been unable to stress even
moderate sized clusters with it. It is fine
as a starting point.
On Wed, Apr 20, 2011 at
Sorry if this is a naive question but can you explain why you consider
that ElasticSearch isn't a distributed solution for realtime search?
I wasn't referring just to ES, mainly to Katta and Solr. Taking a
step back, RT in Lucene should enable immediate consistency making it
symmetrical with
Take a look at this section of the book:
http://hbase.apache.org/book/performance.html
J-D
On Wed, Apr 20, 2011 at 12:44 PM, Weishung Chung weish...@gmail.com wrote:
Hello Stack,
Thank you.
You are right, it's on the client side. I was populating 30 batches of
datasets one after another
Hmm, by any chance are either of you disabling auto flush on table
instances? ie,
HTable.setAutoFlush(false)
I don't see it in the example code you posted, but just wondering if there's
any way this could be a case of:
https://issues.apache.org/jira/browse/HBASE-3750
This fix came after 0.90.2,
So i can't seem to be able to immediately find the explanation for those metrics
- rpcQueueTime -- do I assume it correctly it's the time a request
sits waiting int the incoming rpc queue before being picked up by
handler ?
-rpcProcessingTime -- do i assume it correctly it's time of request
Thanks J-D great info !
On Wed, Apr 20, 2011 at 4:45 PM, Jean-Daniel Cryans jdcry...@apache.orgwrote:
Take a look at this section of the book:
http://hbase.apache.org/book/performance.html
J-D
On Wed, Apr 20, 2011 at 12:44 PM, Weishung Chung weish...@gmail.com
wrote:
Hello Stack,
So far just the latency problems, sorry. It may not be hbase related
still although not very likely.
On Wed, Apr 20, 2011 at 12:54 PM, George P. Stathis
gstat...@traackr.com wrote:
Dmitriy, what are you seeing on your side? Missing inserts? Deletes that are
never applied? Both?
On Wed, Apr
Hello -- we have an issue that looks like this. We have php app
front end and thrift servers that live on seperate boxes away from
HBASE cluster. Everytime we do compaction on one of our 8 RS servers,
we cause a thread pile up on Thrift servers that delay _all_ queries
to HBASE. Our usual
Regarding the test:
- Try to only keep one HBaseAdmin, one HTablePool and always reuse
the same conf between tests, creating a new HBA or HTP creates a new
HBaseConfiguration thus a new connection. Use methods like
setUpBeforeClass. Another option is to close the connection once you
used those
How does get ave. time differs from read average time? What is
the definition of thereof?
-Jack
On Wed, Apr 20, 2011 at 12:38 PM, Stack st...@duboce.net wrote:
On Wed, Apr 20, 2011 at 12:18 PM, Robert Mahfoud
rmahf...@mediosystems.com wrote:
I'm upgrading our code base now to HBase 0.90.1 (CDH3-U0) from
0.89.something.
I'm facing a very hard time tracking failures in our unit tests
Hi, All
Thanks for the helpful comments!
Nice to see this happens rarely in other environments.
Actually I've changed the configuration not to run the task on the master node,
but the same problem happened.
So at first, upgrade the switch. Report again if the problem will be fixed.
Thanks
Hi Stack,
Thanks for your reply!
Two major queries in my system are select queries:
1. Select main comments by page, n items per page, ordered by last
update DESC. The last update of a main comment is the timestamp of its
newest sub-comment (if it doesn't have any sub-comment, last update is
Hello, with 0.89 HBASE, we see the following, all REST servers get
locked on trying to connect to one of our RS servers, the error in the
.out file on that Region Server looks like this:
Exception in thread pool-1-thread-3 java.lang.OutOfMemoryError: Java
heap space
at
85 matches
Mail list logo
