10k scanners fetching 200-500 keys of size 0.5-1kb each This is somewhere between 1GB and 5GB of data and 2.5 - 12.5 seconds in theory - not 1-2 sec. If your clients are off the rack than everything depends on your rack's upstream link capacity.
Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: [email protected] ________________________________________ From: Shaosu Liu [[email protected]] Sent: Thursday, April 24, 2014 12:02 PM To: [email protected] Subject: concurrent scan optimization Hi, I want to large amount of concurrent scan (10k), each fetching 200 - 500 keys, each key is around 500 - 1k bytes. I have 5 regions servers using offheap cache, data is fully cached and all machines are on the same rack with gigabit connection. So GC or disk should not slow down me much. Scan caching is set to 10000, each region server has 200 handlers. Currently it takes around 30 seconds to process 10k scan concurrently. If I use asynchbase, it takes around 18seconds. It should be able to possible do this around 1 - 2 seconds. Could anybody shed light on what I am doing wrong here? Thanks, ~Shaosu Liu Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or [email protected] and delete or destroy any copy of this message and its attachments.
