Re: Problem with performance with many columns in column familie

[Sebastian Bauer]

path has stupid bug with double lock...

Index: core/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
===================================================================
--- core/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java  
(wersja 942215)
+++ core/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java  
(kopia robocza)
@@ -1449,6 +1449,14 @@
       // Run a GET scan and put results into the specified list
       scanner.get(result);
+
+      this.memstore.readLockLock();
+      if (!result.isEmpty()) {
+          KeyValue kv = result.get(0);
+          this.memstore.add(kv);
+        }
+      this.memstore.readLockUnlock();
+
     } finally {
       this.lock.readLock().unlock();
     }

W dniu 12.05.2010 17:16, Sebastian Bauer pisze:
I figured out what is taking so long, test data was 1 row with 100000 
columns and 1 with 100

when i try to increament column this huge row data didnt land in 
MemStore and times was(test in python after warmup):

before path:
#get one column from big row
1 0:00:00.919464
#get one column from small row
2 0:00:00.009650
#atomicIncrement one column from big row
3 0:00:00.081196
#atomicIncrement one column from small row
4 0:00:00.006530

after path:
#get one column from big row
1 0:00:00.009909
#get one column from small row
2 0:00:00.003489
#atomicIncrement one column from big row
3 0:00:00.004890
#atomicIncrement one column from small row
4 0:00:00.004820


path:

Index: core/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
===================================================================
--- 
core/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java  
(wersja 942215)
+++ 
core/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java  
(kopia robocza)
@@ -1449,6 +1449,14 @@

       // Run a GET scan and put results into the specified list
       scanner.get(result);
+
+      this.memstore.readLockLock();
+      if (!result.isEmpty()) {
+          KeyValue kv = result.get(0);
+          this.memstore.add(kv);
+        }
+      this.memstore.readLockLock();
+
     } finally {
       this.lock.readLock().unlock();
     }

what do you think about this change?
all suggestions welcome because i dont even know java ;)



Sebastian B.

W dniu 11.05.2010 18:58, Ted Yu pisze:
jstack is a handy tool:
http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstack.html

On Tue, May 11, 2010 at 9:50 AM, Sebastian Bauer<ad...@ugame.net.pl>  
wrote:

Ram is not a problem, second region server using about 550mB and first
about 300mB problem is with CPU, when i making queries to both column
famielies second region server is using ablut 40% - 80% first about 
10%,
after turning off queries to AdvToUsers(this big) CPU on both 
servers are
2-7%.

Sorry but i dont know how to make thread-dumping and i dont know java.

W dniu 11.05.2010 18:40, Stack pisze:

You could try thread-dumping the regionserver to try and figure where
its hung up.  Counters are usually fast so maybe its something to do
w/ 8k of them in the one row.  What kinda numbers are you seeing?  How
much RAM you throwing at the problem?

Yours,
St.Ack



On Tue, May 11, 2010 at 8:51 AM, Sebastian Bauer<ad...@ugame.net.pl>
  wrote:



Hi,

maybe i'll get help here :)

I have 2 tables, UserToAdv and AdvToUsers.

UserToAdv is simple:
{ "row_id" =>   [ {"adv:<id>":<counter>   },
                            {"adv:<id>":<counter>   },
                            .....about 100 columns
                        ]
only one kind of operation is perform - increasing counter:
client.atomicIncrement("UsersToAdv", ID, column, 1)


AdvToUsers have one column familie: "user:" inside this i have 
about 8000
columns with format: "user:<cookie>"
what i'm doing on DB is increasing counter inside "user:<cookie>":

client.atomicIncrement("AdvToUsers", ID, column, 1)

i have 2 regions:


first one:
        UsersToAdv,6FEC716B3960D1E8208DE6B06993A68D,1273580007602
            stores=1, storefiles=1, storefileSizeMB=8, 
memstoreSizeMB=9,
storefileIndexSizeMB=0
        UsersToAdv,0FDD84B9124B98B05A5E40F47C12DC45,1273580531847
            stores=1, storefiles=1, storefileSizeMB=4, 
memstoreSizeMB=4,
storefileIndexSizeMB=0
        AdvToUsers,5735,1273580575873
            stores=1, storefiles=1, storefileSizeMB=15, 
memstoreSizeMB=10,
storefileIndexSizeMB=0
        UsersToAdv,67CB411B48A7B83F0B863AC615285060,1273580533380
            stores=1, storefiles=1, storefileSizeMB=4, 
memstoreSizeMB=4,
storefileIndexSizeMB=0
        UsersToAdv,4012667F3E78C6431E3DD84641002FCE,1273580532995
            stores=1, storefiles=1, storefileSizeMB=4, 
memstoreSizeMB=4,
storefileIndexSizeMB=0
        UsersToAdv,5FE4A7506737CE0F38E254E62E23FE45,1273580533380
            stores=1, storefiles=1, storefileSizeMB=4, 
memstoreSizeMB=4,
storefileIndexSizeMB=0
        UsersToAdv,47E95EE30A11EBE45F055AC57EB2676E,1273580532995
            stores=1, storefiles=1, storefileSizeMB=4, 
memstoreSizeMB=4,
storefileIndexSizeMB=0
        UsersToAdv,37F9573415D9069B7E5810012AAD9CB7,1273580532258
            stores=1, storefiles=1, storefileSizeMB=4, 
memstoreSizeMB=4,
storefileIndexSizeMB=0
        UsersToAdv,1FFFDF082566D93153B34BFE0C44A9BF,1273580532173
            stores=1, storefiles=1, storefileSizeMB=4, 
memstoreSizeMB=4,
storefileIndexSizeMB=0
        UsersToAdv,17C93FB0047BC4D660C6570B734CBE17,1273580531847
            stores=1, storefiles=1, storefileSizeMB=4, 
memstoreSizeMB=4,
storefileIndexSizeMB=0
        UsersToAdv,27DFD8F02CD98FF57E8334837C73C57A,1273580532173
            stores=1, storefiles=1, storefileSizeMB=4, 
memstoreSizeMB=4,
storefileIndexSizeMB=0

second one:
        UsersToAdv,57C568066D35D09B4AF6CD7D68681144,1273580533427
            stores=1, storefiles=1, storefileSizeMB=4, 
memstoreSizeMB=4,
storefileIndexSizeMB=0
        UsersToAdv,4FA6A1A2681E2D252CCF765B140369EF,1273580533427
            stores=1, storefiles=1, storefileSizeMB=4, 
memstoreSizeMB=4,
storefileIndexSizeMB=0
        AdvToUsers,,1273580575966
            stores=1, storefiles=1, storefileSizeMB=1, 
memstoreSizeMB=1,
storefileIndexSizeMB=0
        UsersToAdv,07B296AC590061025B382B163E3C149E,1273580533023
            stores=1, storefiles=1, storefileSizeMB=4, 
memstoreSizeMB=4,
storefileIndexSizeMB=0
        UsersToAdv,3015D5DB07E2F4D30A19DEB354A85B52,1273580532258
            stores=1, storefiles=1, storefileSizeMB=4, 
memstoreSizeMB=4,
storefileIndexSizeMB=0
        AdvToUsers,5859,1273580580940
            stores=1, storefiles=1, storefileSizeMB=9, 
memstoreSizeMB=9,
storefileIndexSizeMB=0
        AdvToUsers,5315,1273580575966
            stores=1, storefiles=1, storefileSizeMB=14, 
memstoreSizeMB=12,
storefileIndexSizeMB=0
        AdvToUsers,5825,1273580580940
            stores=1, storefiles=1, storefileSizeMB=8, 
memstoreSizeMB=8,
storefileIndexSizeMB=0
        AdvToUsers,5671,1273580578114
            stores=1, storefiles=1, storefileSizeMB=8, 
memstoreSizeMB=7,
storefileIndexSizeMB=0
        UsersToAdv,,1273580533023
            stores=1, storefiles=1, storefileSizeMB=4, 
memstoreSizeMB=4,
storefileIndexSizeMB=0
        AdvToUsers,5457,1273580578114
            stores=1, storefiles=1, storefileSizeMB=8, 
memstoreSizeMB=8,
storefileIndexSizeMB=0

number of queries on both tables are equal, but load is greater on 
second
region because of AdvToUsers

is there any solution to increase performance atomicIncrement 
operation
on
column families with so many(8000) columns?

Thank You,

Sebastian Bauer