Hi Dormando,
Again I got new error in memcached
java.lang.IllegalStateException: Timed out waiting to add Cmd: set Key:
ECO-MAIL-CACHE:1d41402abc4b2a76b9719d9219828753875 Flags: 1 Exp: 0 Data
Length: 394(max wait=10000ms)
at
net.spy.memcached.protocol.TCPMemcachedNodeImpl.addOp(TCPMemcachedNodeImpl.java:346)
at
net.spy.memcached.MemcachedConnection.addOperation(MemcachedConnection.java:697)
at
net.spy.memcached.MemcachedConnection.addOperation(MemcachedConnection.java:677)
at
net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:639)
at
net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:296)
at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:727)
at com.ecomail.emx.core.cache.DefaultCache.set(DefaultCache.java:63)
at
com.ecomail.emx.exchange.db.SubscriptionCacheThread.createSubcriptionCache(SubscriptionCacheThread.java:88)
at
com.ecomail.emx.exchange.db.SubscriptionCacheThread.call(SubscriptionCacheThread.java:52)
at
com.ecomail.emx.exchange.db.SubscriptionCacheThread.call(SubscriptionCacheThread.java:22)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Please suggest me, what should I do to over-come this issues.
On Wednesday, August 1, 2012 12:05:24 PM UTC+5:30, U.S. Adha wrote:
>
> Hi Dormando,
> I got another issue with memcached. Once I am trying
> with 14 GB some time, it able to pull 60 million data into memcached but
> some time, my java client is keep thorwing this error and unable to set 60
> million into memcached.
>
> 2012-08-01 11:54:29.255 INFO net.spy.memcached.MemcachedConnection:
> sun.nio.ch.SelectionKeyImpl@5975d6ab has a ready op, handling IO
> 2012-08-01 11:54:29.255 INFO net.spy.memcached.MemcachedConnection:
> sun.nio.ch.SelectionKeyImpl@5975d6ab has 1, interested in 5
> 2012-08-01 11:54:29.255 INFO net.spy.memcached.MemcachedConnection:
> sun.nio.ch.SelectionKeyImpl@5975d6ab has a ready op, handling IO
> 2012-08-01 11:54:29.256 INFO net.spy.memcached.MemcachedConnection:
> sun.nio.ch.SelectionKeyImpl@2993a66f has 1, interested in 5
> 2012-08-01 11:54:29.256 INFO net.spy.memcached.MemcachedConnection:
> sun.nio.ch.SelectionKeyImpl@2993a66f has a ready op, handling IO
> 2012-08-01 11:54:29.256 INFO net.spy.memcached.MemcachedConnection:
> sun.nio.ch.SelectionKeyImpl@2993a66f has 1, interested in 5
> 2012-08-01 11:54:29.256 INFO net.spy.memcached.MemcachedConnection:
> sun.nio.ch.SelectionKeyImpl@2993a66f has a ready op, handling IO
> 2012-08-01 11:54:29.256 WARN net.spy.memcached.MemcachedConnection: Shut
> down with 12462 bytes remaining to write
> 2012-08-01 11:54:29.257 INFO net.spy.memcached.MemcachedConnection: Shut
> down memcached client
>
>
> Please view stats detail of memcached
>
> STAT libevent 1.4.8-stable
> STAT pointer_size 64
> STAT rusage_user 1395.867236
> STAT rusage_system 179.303205
> STAT curr_connections 5
> STAT total_connections 60
> STAT connection_structures 46
> STAT reserved_fds 20
> STAT cmd_get 100
> STAT cmd_set 44122955
> STAT cmd_flush 0
> STAT cmd_touch 0
> STAT get_hits 0
> STAT get_misses 100
> STAT delete_misses 0
> STAT delete_hits 0
> STAT incr_misses 0
> STAT incr_hits 0
> STAT decr_misses 0
> STAT decr_hits 0
> STAT cas_misses 0
> STAT cas_hits 0
> STAT cas_badval 0
> STAT touch_hits 0
> STAT touch_misses 0
> STAT auth_cmds 0
> STAT auth_errors 0
> STAT bytes_read 3759717871
> STAT bytes_written 353029296
> STAT limit_maxbytes 10485760000
> STAT accepting_conns 1
> STAT listen_disabled_num 0
> STAT threads 4
> STAT conn_yields 1272597
> STAT hash_power_level 25
> STAT hash_bytes 268435456
> STAT hash_is_expanding 0
> STAT expired_unfetched 0
> STAT evicted_unfetched 0
> STAT bytes 6009690477
> STAT curr_items 44122955
> STAT total_items 44122955
> STAT evictions 0
> STAT reclaimed 0
> END
>
>
> On Monday, July 30, 2012 4:02:54 PM UTC+5:30, U.S. Adha wrote:
>>
>> Hi Dormando,
>> Appreciated your help. Now am able to upload 60
>> million as well.
>>
>>
>> Thanks
>> USA
>>
>>
>> On Monday, July 30, 2012 2:13:56 PM UTC+5:30, Dormando wrote:
>>>
>>> bytes_written is the number of bytes written to the network.
>>>
>>> You're actually looking for the "bytes" value, which for you is about
>>> 7.5G. The rest overhead from item headers and loss from slab class
>>> offsets.
>>>
>>> On Mon, 30 Jul 2012, U.S. Adha wrote:
>>>
>>> > Yes, thats true, I over come with previous issue, but as stats detail
>>> show, I have consume 5 GB(STAT bytes_written 512116959) and I have assign 8
>>> GB (STAT limit_maxbytes 8388608000 ) to my memcached, so it seem to me, yet
>>> I have 3 GB available. I have more RAM
>>> > available, Let me try with 12 GB and let see if this would work..
>>> >
>>> >
>>> > Thanks Dormando. I really appreciate your help.
>>> >
>>> >
>>> >
>>> > On Monday, July 30, 2012 1:40:08 PM UTC+5:30, Dormando wrote:
>>> > Well if you have evictions, you're probably out of memory? This
>>> isn't what
>>> > you were saying before though.
>>> >
>>> > Get moar memoryyy?
>>> >
>>> > On Mon, 30 Jul 2012, U.S. Adha wrote:
>>> >
>>> > > Hi Dormando, Thanks for reply. Looks like
>>> this works a bit, but it start evictions objects after 50 Millions object
>>> inserted into memcached. I am using memcached 1.4.7 at this time. Also I
>>> have tried with memcached 1.4.13 too, but same
>>> > issue was
>>> > > there too. Now am starting memcached with this
>>> > >
>>> > >
>>> > > memcached -m 8000 -l 127.0.0.1 -p 11211 9000 9 >
>>> log_nine_seconds
>>> > >
>>> > > yet my memcached evicting object...look stats
>>> > >
>>> > > STAT reclaimed 0
>>> > > END
>>> > > stats
>>> > > STAT pid 5300
>>> > > STAT uptime 3801
>>> > > STAT time 1343635673
>>> > > STAT version 1.4.7
>>> > > STAT libevent 2.0.12-stable
>>> > > STAT pointer_size 64
>>> > > STAT rusage_user 500.823299
>>> > > STAT rusage_system 486.058376
>>> > > STAT curr_connections 5
>>> > > STAT total_connections 37434
>>> > > STAT connection_structures 27
>>> > > STAT cmd_get 154795
>>> > > STAT cmd_set 63614971
>>> > > STAT cmd_flush 0
>>> > > STAT get_hits 154795
>>> > > STAT get_misses 0
>>> > > STAT delete_misses 0
>>> > > STAT delete_hits 0
>>> > > STAT incr_misses 0
>>> > > STAT incr_hits 0
>>> > > STAT decr_misses 0
>>> > > STAT decr_hits 0
>>> > > STAT cas_misses 0
>>> > > STAT cas_hits 0
>>> > > STAT cas_badval 0
>>> > > STAT auth_cmds 0
>>> > > STAT auth_errors 0
>>> > > STAT bytes_read 5421185438
>>> > > STAT bytes_written 512116959
>>> > > STAT limit_maxbytes 8388608000
>>> > > STAT accepting_conns 1
>>> > > STAT listen_disabled_num 0
>>> > > STAT threads 4
>>> > > STAT conn_yields 19532
>>> > > STAT bytes 7523088744
>>> > > STAT curr_items 55177103
>>> > > STAT total_items 63614971
>>> > > STAT evictions 8325645
>>> > > STAT reclaimed 0
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > On Saturday, July 28, 2012 1:06:37 AM UTC+5:30, Dormando
>>> wrote:
>>> > > Your startup options are bizarre...
>>> > >
>>> > > memcached -m 8000 -l 127.0.0.1 -p 11211
>>> > > ^ just do that. don't add -L or -R and see if that helps
>>> any.
>>> > >
>>> > > Also, we need to know the version and make of the client
>>> you're using. Any
>>> > > example code or data for how you're trying to load it
>>> will be required to
>>> > > see what's happening.
>>> > >
>>> > > Your client should be doing something weird to end up in
>>> that state.
>>> > > Memcached is trying to write back to your client but the
>>> socket is closed
>>> > > for some reason.
>>> > >
>>> > > On Fri, 27 Jul 2012, U.S. Adha wrote:
>>> > >
>>> > > > Yes, I tried without -vv option. It seem like, it's
>>> speed increase a bit faster but not much as I expect. But this issue
>>> > > > remain...................."Failed to write, and not
>>> due to blocking: Connection reset by peer "
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Friday, July 27, 2012 3:05:59 PM UTC+5:30, U.S.
>>> Adha wrote:
>>> > > > Yes, I tried with -vv option. It seem like, it's
>>> speed increase a bit faster but not much as I expect. But this issue
>>> > > > remain...................."Failed to write, and
>>> not due to blocking: Connection reset by peer "
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Friday, July 27, 2012 3:03:48 PM UTC+5:30, Martin
>>> Hellmich wrote:
>>> > > > Have you tried the same without the -vv switch?
>>> > > > I experienced memcached to be much faster
>>> without the logging.
>>> > > > Martin
>>> > > >
>>> > > > On 07/27/2012 11:29 AM, U.S. Adha wrote:
>>> > > > > Hi, thanks for reply me. yet am fighting with
>>> same issue, am trying to put data into memcached via java client and am
>>> trying to
>>> > > > put 60 million object into memcached. I am using
>>> this command to start memcached
>>> > > > >
>>> > > > > memcached -d -vv -m 8000 -l 127.0.0.1 -p 11211
>>> -c 250 -L 2048 -R 100 1000 10 > log_ten_seconds
>>> > > > >
>>> > > > > but after putting 20 million objects memcached
>>> give me error like
>>> > > > >
>>> > > > > Failed to write, and not due to blocking:
>>> Connection reset by peer
>>> > > > > <80 connection closed.
>>> > > > > Failed to write, and not due to blocking:
>>> Connection reset by peer
>>> > > > >
>>> > > > >
>>> > > > > Am not sure, where am having problem......I
>>> really appreciate if you help me to get-out of this hall.
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > On Wednesday, November 30, 2011 3:33:53 PM
>>> UTC+5:30, ktechie wrote:
>>> > > > >
>>> > > > > My configuration is VMware on Red hat
>>> 64bit linux OS. Memcached 1.4.5
>>> > > > > and Xmemcached 1.3.5.
>>> > > > > I have a program which does eager loading
>>> of data, which is about 3GB,
>>> > > > > this happens through a multithreaded
>>> application, with 10 threads
>>> > > > > I have also created a connection pool of
>>> 50 connections to memcached.
>>> > > > > There is also a listener which tells when
>>> there is disconnection or
>>> > > > > the connection is healed.
>>> > > > >
>>> > > > > Sometimes the program works fine without
>>> errors, but at times when I
>>> > > > > see disconnections and connections
>>> happening many times.
>>> > > > > And the memcached server gives the
>>> following errors.
>>> > > > >
>>> > > > > Failed to write, and not due to blocking:
>>> Connection reset by peer
>>> > > > > Failed to read, and not due to blocking:
>>> > > > > errno: 104 Connection reset by peer
>>> > > > > rcurr=127980b8 ritem=1442bd8a
>>> rbuf=12797c00 rlbytes=550 rsize=2048
>>> > > > >
>>> > > > > When this happens the data is not fully
>>> loaded.
>>> > > > >
>>> > > > > At times this issue is resolved after the
>>> VM is rebooted. but that is
>>> > > > > not always the case.
>>> > > > > I tried reducing the connection pool size
>>> to 1 and thread pool to 1,
>>> > > > > even then the disconnections could happen
>>> > > > >
>>> > > > > What could be the possible reasons for
>>> this.
>>> > > > >
>>> > > >
>>> > > > --
>>> > > > Martin Hellmich Information
>>> Technology Department
>>> > > > [email protected]
>>> CERN
>>> > > > +41 22 76 765 26
>>> CH-1211 Geneva 23
>>> > > >
>>> > > >
>>> > > >
>>> > >
>>> > >
>>> > >
>>> >
>>> >
>>> >
>>
>>
