I have not applied hdfs-347, but done some other experiments.
I increased client thread to 2000 to put enough pressure on cluster. I
disabled RS block cache. The total TPS is still low (with Month+User
as row key, it is about 1300 for 10 RS+DN and with User+Month it is
700).
I used BTrace to log
This problem seems happened with high probability, I have saw it several times.
Here's the description about this problem:
1.From the WEBUI, I can see the regions count is:35087.
2.From the HBase shell, I counted the '.META.' table, and it showed there's
35085 regions.
hbase(main):001:0 count
Hello,
I am writing a MR job where each reducer will output one HFile containing some
of the rows of the table that will be created.
At first I thought to use the HashPartitioner to achieve load balancing, but
this would mix the rows and the output of each reducer will not be a continuous
Hi,
We have a setup with 4 regionservers and a replication factor of 3. We are
running MapReduce tasks using Hbase as data-source and sink. When running
MapReduce tasks over data stored on the 4 nodes we noticed that in the
statistics of a successfully completed job, the majority of the maps are
Are your keys arranged so that you have a problem with a hot region?
On Mon, May 16, 2011 at 11:18 PM, Weihua JIANG weihua.ji...@gmail.comwrote:
I have not applied hdfs-347, but done some other experiments.
I increased client thread to 2000 to put enough pressure on cluster. I
disabled RS
No. The key is generated randomly. In theory, it shall distributed to
all the RSs equally.
Thanks
Weihua
2011/5/17 Ted Dunning tdunn...@maprtech.com:
Are your keys arranged so that you have a problem with a hot region?
On Mon, May 16, 2011 at 11:18 PM, Weihua JIANG
If I understand hbase bulk loading correctly each hfile generated needs its
keys to fit within one existing region - that is the reason the total order
partitioner is used. I believe however that within one region before
compaction you can have multiple hfiles for a given column family and each
Isn't a 20mb young generation pretty small these days? For a 4gb heap, you
might want to try giving 5-10% of it to the young generation by setting
-XX:NewSize=256M
Some of the older
docshttp://www.oracle.com/technetwork/java/gc-tuning-5-138395.html#1.1.Sizing%20the%20Generations%7Coutlinehave
a
Nice analysis.
Can you figure the most popular blocks requested? You could figure
which files they belong too by grepping the blocks in namenode log.
It is odd that you have the sort of a request profile if your loading
was even. I'd expect the DN distribution to be even.
Sounds like hdfs-347
.META. does not include region information for itself or -ROOT- so a count
of .META. should always be two regions less than the total number.
-chris
On Tue, May 17, 2011 at 4:32 AM, bijieshan bijies...@huawei.com wrote:
This problem seems happened with high probability, I have saw it several
Sorry to jump in on the tail end.
What do you mean to say that they key is generated randomly?
I mean are you using a key and then applying a SHA-1 hash?
Which node is serving your -ROOT- and META tables?
Have you applied the GC hints recommended by Todd L in his blog?
Also you said:
'
And
When running map reduce jobs against HBase, a task needs to be scheduled on the
region server serving the region you're reading from to be considered local.
You have three replicas of the data at the HDFS level, but not at the HBase
level.
-Joey
On May 17, 2011, at 5:44, Felix Sprick
I am having a problem adding filerLists to my scans. These are the standard
filters, nothing custom. When I use them on the client, I get the following
errors. Maybe I need to include some additional jars on my classpath? I am
using row filters in the filter list.
pseudo-ish code:
Scan s = new
Yes, that was it. My scans are now back on the fast track. Thanks!
On Tue, May 17, 2011 at 12:45 PM, Todd Lipcon t...@cloudera.com wrote:
Hi Barney,
I think the clever syntax you're using might be confusing things.
Try:
FilterList fl = new FilterList(FilterList.Operator.MUST_PASS_ALL);
My rule of thumb is as big as possible but no bigger. The young gen size
determines the pause time and frequency of your young generation
collections, which are stop-the-world. In my experience the speed is about
0.5-1 second per GB on modern hardware.
You get better throughput and less
(Running Hbase 0.90.0 on 700+ nodes.)
You may have seen many (or mostly all) of the following issues already:
1. HConnection.isTableAvailable: This doesn't seem to be working all the
time. In particular, I had this code after creating a table asynchronously:
do {
LOG.info(Table +
For 3, we have the following check in createTable():
if(lastKey != null Bytes.equals(splitKey, lastKey)) {
throw new IllegalArgumentException(All split keys must be unique,
+
found duplicate: + Bytes.toStringBinary(splitKey) +
I wonder why you used
400 regions a day is way too much, also in 0.20.6 there's a high risk
of collision when you get near the 10 thousands of regions. But that's
most probably not your current issue.
That HDFS message 99% of the time means that the region server went
into GC and when it came back the master already
For 1, the check in HCM.isTableAvailable() is:
return available.get() (regionCount.get() 0);
This explains why some regions aren't available.
For 3, can you provide a unit test so that we can investigate further ?
Thanks
On Tue, May 17, 2011 at 4:25 PM, Vidhyashankar Venkataraman
Well, our issue was that we take an 8 core box, doing 18MB ParNew GC,
do it super fast (10x1sec), receive RPC calls to RS at the same time,
pile them up due to CPU contention, do GC even faster, run out of user
CPU. This is basically the cycle of cpu death.
It seems that larger RAM boxes doing
For 1, the check in HCM.isTableAvailable() is:
return available.get() (regionCount.get() 0);
This explains why some regions aren't available.
The javadoc says the function returns true if all regions are available.
Clearly this statement is wrong going by what is there in the code.
thanks J-D as always
-Original Message-
From: Jean-Daniel Cryans jdcry...@apache.org
To: user@hbase.apache.org
Sent: Tue, May 17, 2011 8:04 pm
Subject: Re: mapreduce job failure
400 regions a day is way too much, also in 0.20.6 there's a high risk
of collision when you get
Yes, you're right. While count the .META., the result will exclude the -ROOT-
region and the .META. region. Pardon me ,I should not mention about that.
Maybe the less 2 region is just a coincidence here, I can show another scenario
about this problem:
From WebUI, we can see that:
Address Start
Also some of the source for which we had used this function may be broken
(for example in LoadIncrementalHFiles.java)
Can you be more specific ?
Thanks
On Tue, May 17, 2011 at 5:54 PM, Vidhyashankar Venkataraman
vidhy...@yahoo-inc.com wrote:
For 1, the check in HCM.isTableAvailable() is:
Did you mean that coming out of the following loop, the table might still be
unavailable if there were many regions ?
while (!conn.isTableAvailable(table.getTableName())
(ctrTABLE_CREATE_MAX_RETRIES)) {
Cheers
On Tue, May 17, 2011 at 7:10 PM, Ted Yu yuzhih...@gmail.com wrote:
Also some
-ROOT- and .META. table are not served by these hot region servers.
I generate the key in random and verified at client by grepping .META.
table and record the mapping from each query to its serving region
server. It shows that each RS serves almost the same number of query
requests.
For GC
26 matches
Mail list logo
