Okay, nice to know that. Is there any asynchronous writing method in spy? I don't understand, I have set the first two maps to null when writing the third one, so you mean the memclient still reads over all the existing records in server when inserting new data? Or its a heap size error at the server side?
On Sun, Oct 17, 2010 at 12:36 AM, Jonathan Leech <[email protected]> wrote: > Sounds like bug 125 to me. Your thread inserting the records can write them > to the queue faster than they are written from the queue to memcached, the > queue fills up with more and more records, and each one takes longer than > the last to get written. If you are also bouncing up against the upper limit > of the heap, then the VM will compound the issue spending a lot of time > garbage collecting. > > > > On Oct 16, 2010, at 6:53 PM, Boris Partensky <[email protected]> > wrote: > >> <<I am really curious why spy cannot store up to 6 million... >> >> I'd definitely spend some more time analyzing what's going on if I >> were you before going down that road. Turn up gc logging >> (-verbose:gc), see if the app is heavily gc-ing when the program >> "stops"; dump threads (kill -3 pid) etc.... >> >> On Sat, Oct 16, 2010 at 8:27 PM, Shi Yu <[email protected]> wrote: >>> >>> Follow up my previous question. I tried to read those 6 million >>> <key,value> records out. Both API are capable, but the spymemcache API >>> is faster (16 minutes) than the Whalin's v2.5.1 distribution (24 >>> minutes) (http://github.com/gwhalin/Memcached-Java-Client/downloads) >>> .Since the spymemcache cannot insert that much, I came to a strange >>> hybrid settings using Whalin's API to store, and using spymemcache to >>> read ... >>> >>> I am really curious why spy cannot store up to 6 million... >>> >>> >>> On Sat, Oct 16, 2010 at 12:59 PM, Shi Yu <[email protected]> wrote: >>>> >>>> Hi, >>>> >>>> I have two problems when using memcached java clients the spymemcached >>>> (http://code.google.com/p/spymemcached/) and the gwhalin java client >>>> for memcached (http://github.com/gwhalin/Memcached-Java-Client). I >>>> found that the spymemcached failed to store more than 4.3 million >>>> records sometimes 3.7 million (please see my code below). There was no >>>> error no exception, but simply the code automatically stopped at the >>>> 4.3 million and didn't even hit the final line. In contrast, the >>>> gwhalin java client was able to insert 6 million records without >>>> problem, however, comparing the speed of inserting the first 4 million >>>> records that the gwhalin client is much slower than the spymemcached. >>>> The memcached server is set up using the following command >>>> "./memcached -d -m 4000 127.0.0.1 -p 11211" and I think there is no >>>> problem at the server side. What is the problem here, should I adjust >>>> any settings? Thanks? >>>> >>>> -Shi >>>> >>>> >>>> //spymemcached code >>>> public static void main(String[] args) throws Exception { >>>> MemcachedClient mc=new MemcachedClient(new >>>> InetSocketAddress("ocuic32.research", 11211)); >>>> mc.flush(); >>>> System.out.println("Memchaced flushed ..."); >>>> int count = 0; >>>> for(int i=0;i<6000000;i++){ >>>> String a = "String"+i; >>>> String b = "Value"+i; >>>> >>>> mc.add(a,i,(String) b); >>>> count ++; >>>> if (String.valueOf(count).endsWith("00000")) >>>> System.out.println(count+ " elements added."); >>>> } >>>> >>>> System.out.println("done "+ count +" records inserted"); >>>> //spymemcached aint able to get this line >>>> } >>>> >>>> >>>> >>>> //gwhalin memcached code >>>> public static void main(String[] args) throws Exception { >>>> BasicConfigurator.configure(); >>>> String[] servers = { "ocuic32.research:11211" }; >>>> SockIOPool pool = SockIOPool.getInstance(); >>>> pool.setServers( servers ); >>>> pool.setFailover( true ); >>>> pool.setInitConn( 10 ); >>>> pool.setMinConn( 5 ); >>>> pool.setMaxConn( 250 ); >>>> pool.setMaintSleep( 30 ); >>>> pool.setNagle( false ); >>>> pool.setSocketTO( 3000 ); >>>> pool.setAliveCheck( true ); >>>> pool.initialize(); >>>> >>>> MemcachedClient mcc = new MemcachedClient(); >>>> mcc.flushAll(); >>>> int count = 0; >>>> int maxlength = 0; >>>> //while((line=br.readLine())!=null){ >>>> >>>> for(int i=0;i<6000000;i++){ >>>> String a = "String"+i; >>>> String b = "Value"+i; >>>> String sha1_ad1 = AeSimpleSHA1.SHA1(a); >>>> mcc.set(sha1_ad1,(String) b); >>>> count ++; >>>> if (String.valueOf(count).endsWith("00000")) >>>> System.out.println(count+ " elements added."); >>>> >>>> } >>>> >>>> System.out.println("done "+ count +" records >>>> inserted"); //gwhalin 's client is able to get this line, but very >>>> slow >>>> } >>>> >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "spymemcached" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/spymemcached?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "spymemcached" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/spymemcached?hl=en. > >
