1) when there are only 2 client threads, vmstat output is procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 29236 1610000 557488 12027072 0 0 19 53 0 0 3 1 97 0 0 0 0 29236 1610152 557488 12027076 0 0 0 0 4630 23177 2 1 97 0 0 0 0 29236 1610484 557488 12027076 0 0 0 603 4604 22758 2 1 97 0 0 0 0 29236 1610544 557488 12027076 0 0 0 11 4643 23009 2 1 97 0 0 0 0 29236 1610912 557488 12027080 0 0 0 5 4578 22900 2 1 97 0 0 3 0 29236 1610600 557488 12027080 0 0 0 21 4627 22922 2 1 97 0 0 1 0 29236 1610616 557488 12027080 0 0 0 0 4552 23077 2 1 97 0 0 0 0 29236 1610992 557488 12027084 0 0 0 608 4608 22995 2 1 97 0 0 0 0 29236 1611900 557488 12027084 0 0 0 144 4641 23000 2 1 97 0 0 0 0 29236 1612040 557488 12027084 0 0 0 0 4621 22952 2 1 97 0 0 0 0 29236 1611428 557488 12027088 0 0 0 593 4668 23133 2 1 96 0 0 1 0 29236 1611812 557488 12027088 0 0 0 0 4623 23673 2 1 97 0 0 0 0 29236 1612068 557488 12027088 0 0 0 145 4627 23245 2 1 97 0 0 1 0 29236 1612440 557488 12027092 0 0 0 1 4654 23351 2 1 97 0 0 1 0 29236 1612120 557488 12027092 0 0 0 131 4695 23137 4 1 95 0 0 2 0 29236 1612492 557492 12027096 0 0 0 39 4647 22818 2 1 97 0 0 1 0 29236 1610164 557492 12027096 0 0 0 633 4666 22914 2 1 97 0 0 0 0 29236 1610380 557492 12027096 0 0 0 73 4668 22957 2 1 97 0 0 1 0 29236 1610420 557492 12027100 0 0 0 47 4642 22907 2 1 97 0 0 0 0 29236 1611596 557492 12027100 0 0 0 0 4664 22999 2 1 97 0 0 1 0 29236 1612236 557492 12027084 0 0 0 657 4665 22552 2 1 97 0 0 1 0 29236 1612732 557492 12027088 0 0 0 0 4586 22640 2 1 97 0 0 0 0 29236 1612504 557492 12027088 0 0 0 33 4605 22670 2 1 97 0 0 2 0 29236 1611500 557492 12027092 0 0 0 161 4575 22763 2 1 97 0 0 0 0 29236 1612160 557500 12027092 0 0 0 5 4570 23265 2 1 97 0 0 0 0 29236 1612568 557500 12027092 0 0 0 592 4616 23285 2 1 97 0 0 0 0 29236 1612700 557500 12027096 0 0 0 0 4688 23754 3 1 96 0 0 1 0 29236 1612192 557504 12027092 0 0 0 27 4649 23501 2 1 97 0 0 1 0 29236 1611748 557504 12027104 0 0 0 12 4612 23664 2 1 97 0 0 0 0 29236 1611932 557504 12027116 0 0 0 768 4606 22910 2 1 97 0 0 2 0 29236 1611788 557504 12027116 0 0 0 25 4545 22991 2 1 97 0 0 1 0 29236 1611916 557504 12027120 0 0 0 0 4615 23138 2 1 97 0 0 0 0 29236 1612176 557504 12027120 0 0 0 139 4604 23231 2 1 97 0 0 0 0 29236 1612848 557504 12027108 0 0 0 0 4624 23673 2 1 97 0 0 0 0 29236 1612820 557504 12027108 0 0 0 616 4657 23247 2 1 97 0 0 0 0 29236 1613196 557508 12027108 0 0 0 107 4581 23193 2 1 97 0 0 1 0 29236 1611772 557508 12027112 0 0 0 185 4594 22807 2 1 97 0 0 2 0 29236 1610432 557508 12027116 0 0 0 1 4603 23395 2 1 97 0 0 2 0 29236 1610348 557512 12027116 0 0 0 653 4724 23562 2 1 97 0 0 0 0 29236 1610612 557512 12027132 0 0 0 0 4621 23533 2 1 97 0 0 1 0 29236 1612368 557512 12027116 0 0 0 77 4609 23223 2 1 97 0 0 0 0 29236 1612628 557512 12027120 0 0 0 0 4571 22697 2 1 97 0 0 1 0 29236 1610908 557512 12027120 0 0 0 19 4585 22614 2 1 97 0 0 1 0 29236 1610020 557512 12027124 0 0 0 629 4656 23382 2 1 97 0 0
2) up client threads number to 8, vmstat output is procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 29236 1610000 557488 12027072 0 0 19 53 0 0 3 1 97 0 0 0 0 29236 1610152 557488 12027076 0 0 0 0 4630 23177 2 1 97 0 0 0 0 29236 1610484 557488 12027076 0 0 0 603 4604 22758 2 1 97 0 0 0 0 29236 1610544 557488 12027076 0 0 0 11 4643 23009 2 1 97 0 0 0 0 29236 1610912 557488 12027080 0 0 0 5 4578 22900 2 1 97 0 0 3 0 29236 1610600 557488 12027080 0 0 0 21 4627 22922 2 1 97 0 0 1 0 29236 1610616 557488 12027080 0 0 0 0 4552 23077 2 1 97 0 0 0 0 29236 1610992 557488 12027084 0 0 0 608 4608 22995 2 1 97 0 0 0 0 29236 1611900 557488 12027084 0 0 0 144 4641 23000 2 1 97 0 0 0 0 29236 1612040 557488 12027084 0 0 0 0 4621 22952 2 1 97 0 0 0 0 29236 1611428 557488 12027088 0 0 0 593 4668 23133 2 1 96 0 0 1 0 29236 1611812 557488 12027088 0 0 0 0 4623 23673 2 1 97 0 0 0 0 29236 1612068 557488 12027088 0 0 0 145 4627 23245 2 1 97 0 0 1 0 29236 1612440 557488 12027092 0 0 0 1 4654 23351 2 1 97 0 0 1 0 29236 1612120 557488 12027092 0 0 0 131 4695 23137 4 1 95 0 0 2 0 29236 1612492 557492 12027096 0 0 0 39 4647 22818 2 1 97 0 0 1 0 29236 1610164 557492 12027096 0 0 0 633 4666 22914 2 1 97 0 0 0 0 29236 1610380 557492 12027096 0 0 0 73 4668 22957 2 1 97 0 0 1 0 29236 1610420 557492 12027100 0 0 0 47 4642 22907 2 1 97 0 0 0 0 29236 1611596 557492 12027100 0 0 0 0 4664 22999 2 1 97 0 0 1 0 29236 1612236 557492 12027084 0 0 0 657 4665 22552 2 1 97 0 0 1 0 29236 1612732 557492 12027088 0 0 0 0 4586 22640 2 1 97 0 0 0 0 29236 1612504 557492 12027088 0 0 0 33 4605 22670 2 1 97 0 0 2 0 29236 1611500 557492 12027092 0 0 0 161 4575 22763 2 1 97 0 0 0 0 29236 1612160 557500 12027092 0 0 0 5 4570 23265 2 1 97 0 0 0 0 29236 1612568 557500 12027092 0 0 0 592 4616 23285 2 1 97 0 0 0 0 29236 1612700 557500 12027096 0 0 0 0 4688 23754 3 1 96 0 0 1 0 29236 1612192 557504 12027092 0 0 0 27 4649 23501 2 1 97 0 0 1 0 29236 1611748 557504 12027104 0 0 0 12 4612 23664 2 1 97 0 0 0 0 29236 1611932 557504 12027116 0 0 0 768 4606 22910 2 1 97 0 0 2 0 29236 1611788 557504 12027116 0 0 0 25 4545 22991 2 1 97 0 0 1 0 29236 1611916 557504 12027120 0 0 0 0 4615 23138 2 1 97 0 0 0 0 29236 1612176 557504 12027120 0 0 0 139 4604 23231 2 1 97 0 0 0 0 29236 1612848 557504 12027108 0 0 0 0 4624 23673 2 1 97 0 0 0 0 29236 1612820 557504 12027108 0 0 0 616 4657 23247 2 1 97 0 0 0 0 29236 1613196 557508 12027108 0 0 0 107 4581 23193 2 1 97 0 0 1 0 29236 1611772 557508 12027112 0 0 0 185 4594 22807 2 1 97 0 0 2 0 29236 1610432 557508 12027116 0 0 0 1 4603 23395 2 1 97 0 0 2 0 29236 1610348 557512 12027116 0 0 0 653 4724 23562 2 1 97 0 0 0 0 29236 1610612 557512 12027132 0 0 0 0 4621 23533 2 1 97 0 0 1 0 29236 1612368 557512 12027116 0 0 0 77 4609 23223 2 1 97 0 0 0 0 29236 1612628 557512 12027120 0 0 0 0 4571 22697 2 1 97 0 0 1 0 29236 1610908 557512 12027120 0 0 0 19 4585 22614 2 1 97 0 0 1 0 29236 1610020 557512 12027124 0 0 0 629 4656 23382 2 1 97 0 0 and the "VM Thread" is very busing, take up ~90% one cpu time. On Thu, Feb 24, 2011 at 10:55 AM, Schubert Zhang <[email protected]> wrote: > Now, I am trying the 0.90.1, but this issue is still there. > > I attach the jstack output. Coud you please help me analyze it. > > Seems all the 8 client threads are doing metaScan! > > On Sat, Jan 29, 2011 at 1:02 AM, Stack <[email protected]> wrote: > >> On Thu, Jan 27, 2011 at 10:33 PM, Schubert Zhang <[email protected]> >> wrote: >> > 1. The .META. table seems ok >> > I can read my data table (one thread for reading). >> > I can use hbase shell to scan my data table. >> > And I can use 1~4 threads to put more data into my data table. >> > >> >> Good. This would seem to say that .META. is not locked out (You are >> doing these scans while your 8+client process is hung?). >> >> >> > Before this issue happen, about 800 millions entities (column) have >> been >> > put into the table successfully, and there are 253 regions for this >> table. >> > >> >> >> So, you were running fine with 8+ clients until you hit the 800million >> entries? >> >> >> > 3. All clients use HBaseConfiguration.create() for a new Configuration >> > instance. >> > >> >> Do you do this for each new instance of HTable or do you pass them all >> the same Configuration instance? >> >> >> > 4. The 8+ client threads running on a single machine and a single JVM. >> > >> >> How many instances of this process? One or many? >> >> >> > 5. Seems all 8+ threads are blocked in same location waiting on call to >> > return. >> > >> >> If you want to paste a thread dump of your client, some one of us will >> give it a gander. >> >> St.Ack >> > > -- Best Regards Anty Rao
