<<I am really curious why spy cannot store up to 6 million... I'd definitely spend some more time analyzing what's going on if I were you before going down that road. Turn up gc logging (-verbose:gc), see if the app is heavily gc-ing when the program "stops"; dump threads (kill -3 pid) etc....
On Sat, Oct 16, 2010 at 8:27 PM, Shi Yu <[email protected]> wrote: > Follow up my previous question. I tried to read those 6 million > <key,value> records out. Both API are capable, but the spymemcache API > is faster (16 minutes) than the Whalin's v2.5.1 distribution (24 > minutes) (http://github.com/gwhalin/Memcached-Java-Client/downloads) > .Since the spymemcache cannot insert that much, I came to a strange > hybrid settings using Whalin's API to store, and using spymemcache to > read ... > > I am really curious why spy cannot store up to 6 million... > > > On Sat, Oct 16, 2010 at 12:59 PM, Shi Yu <[email protected]> wrote: >> Hi, >> >> I have two problems when using memcached java clients the spymemcached >> (http://code.google.com/p/spymemcached/) and the gwhalin java client >> for memcached (http://github.com/gwhalin/Memcached-Java-Client). I >> found that the spymemcached failed to store more than 4.3 million >> records sometimes 3.7 million (please see my code below). There was no >> error no exception, but simply the code automatically stopped at the >> 4.3 million and didn't even hit the final line. In contrast, the >> gwhalin java client was able to insert 6 million records without >> problem, however, comparing the speed of inserting the first 4 million >> records that the gwhalin client is much slower than the spymemcached. >> The memcached server is set up using the following command >> "./memcached -d -m 4000 127.0.0.1 -p 11211" and I think there is no >> problem at the server side. What is the problem here, should I adjust >> any settings? Thanks? >> >> -Shi >> >> >> //spymemcached code >> public static void main(String[] args) throws Exception { >> MemcachedClient mc=new MemcachedClient(new >> InetSocketAddress("ocuic32.research", 11211)); >> mc.flush(); >> System.out.println("Memchaced flushed ..."); >> int count = 0; >> for(int i=0;i<6000000;i++){ >> String a = "String"+i; >> String b = "Value"+i; >> >> mc.add(a,i,(String) b); >> count ++; >> if (String.valueOf(count).endsWith("00000")) >> System.out.println(count+ " elements added."); >> } >> >> System.out.println("done "+ count +" records inserted"); >> //spymemcached aint able to get this line >> } >> >> >> >> //gwhalin memcached code >> public static void main(String[] args) throws Exception { >> BasicConfigurator.configure(); >> String[] servers = { "ocuic32.research:11211" }; >> SockIOPool pool = SockIOPool.getInstance(); >> pool.setServers( servers ); >> pool.setFailover( true ); >> pool.setInitConn( 10 ); >> pool.setMinConn( 5 ); >> pool.setMaxConn( 250 ); >> pool.setMaintSleep( 30 ); >> pool.setNagle( false ); >> pool.setSocketTO( 3000 ); >> pool.setAliveCheck( true ); >> pool.initialize(); >> >> MemcachedClient mcc = new MemcachedClient(); >> mcc.flushAll(); >> int count = 0; >> int maxlength = 0; >> //while((line=br.readLine())!=null){ >> >> for(int i=0;i<6000000;i++){ >> String a = "String"+i; >> String b = "Value"+i; >> String sha1_ad1 = AeSimpleSHA1.SHA1(a); >> mcc.set(sha1_ad1,(String) b); >> count ++; >> if (String.valueOf(count).endsWith("00000")) >> System.out.println(count+ " elements added."); >> >> } >> >> System.out.println("done "+ count +" records >> inserted"); //gwhalin 's client is able to get this line, but very >> slow >> } >> >
