#x27;ll need to switch to multi-threaded
to get this benefit. A dedicated accept thread is a common design pattern
memcached just hadn't adopted until just now.
Any chance you/Don/etc could run the stable tree on a box or two and see
if it removes *or* reduces the connection timeout?
-Dorma
ug in
pecl/memcache...
Any other client authors (libmemcached folks? :P) willing to put out a
native benchmark for the aforementioned case? I'd love to but I'm
overextended for the next week :)
-Dormando
On Wed, 27 Aug 2008, Clint Webb wrote:
> Assuming it takes approximately a
Yo,
That goofy google groups footer that appeared when I moved the list(s).
Anyone mind terrible if I turn that off?
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"memcached" group.
To post to this group, send em
add a few more mutexes used in stats to spread the
load a touch. Easy to do now, then we can ignore the problem for a while
longer.
-Dormando
On Mon, 25 Aug 2008, nick wrote:
>
> Argh. Let me try to put the formatting back in.
>
> The attached patch (against subversion rev793,
Heh. Applied.
On Thu, 31 Jul 2008, Jonathan Bastien-Filiatrault wrote:
>
> Signed-off-by: Jonathan Bastien-Filiatrault <[EMAIL PROTECTED]>
> ---
> configure.ac |2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
>
> >
>
Hey,
We should probably be using zeroed out memory there anyway. Although the
fact that this only comes up in a specific condition under centos makes me
want to believe redhat's doing something retarded.
I've committed this and pushed to stable. Thanks!
-Dormando
On Sat, 2 Aug 2
one while
implementing it, I'll attempt to queue this change for 1.2.7. Patches
welcome too :P
-Dormando
> Yeah, pros and cons both ways. Another proposal, just for fun: we could
> return NOT_FOUND (pro: no change to clients!) when it's not a number, and
> _also_ delete the value.
7;ll run in the foreground. So you can
see why it doesn't start, but you'll have to remember to restart it with
-d once you're sure it works.
-Dormando
On Fri, 5 Sep 2008, pedalpete wrote:
Thanks Steve,
I am kinda new to admining my own server, so no, I hadn't tried
telnet.
And
The -u option is for specifying the user to drop privileges to?
If you see mentions of "managed instances" - that's unfinished code, which
we should silence. You can safely ignore that.
-Dormando
On Fri, 5 Sep 2008, TK wrote:
thanks, steve.
Was looking at the help
ecuted:0.00% of 3
/usr/include/gentoo-multilib/amd64/stdlib.h:creating 'stdlib.h.gcov'
Anyone else have comments?
-Dormando
On Thu, 4 Sep 2008, Victor Kirkebo wrote:
> This patch adds test coverage to the 'make test' target.
> gcov is used for gcc and tcov is used for Sun studio compiler.
>
> Victor
>
Toru and Trond continue to kick ass.
If you have any patches, fixes, minor requests, or bug reports that you'd
like to see hit 1.2.7, please respond to the list with details or code. I
would like to stamp this a 1.2.7-rc1 sometime tomorrow at the latest.
have fun,
-Dormando
e file executed
items.c : 2.39 Percent of the file executed
memcached.c : 8.15 Percent of the file executed
slabs.c : 15.65 Percent of the file executed
stats.c : 1.72 Percent of the file executed
-Victor
dormando wrote:
Hey,
Think this could be modified to not spit coverage on files not in the
so
opy default_engine tree to lockscale_engine (or whatever)
-> port your patches onto that
-> isolate the patches which touch the main tree
... then we can decide on if we want to distribute both engines and give
users a choice, or keep one in the repo and slowly adopt the scaling
changes from one to the other (if possible).
Thanks,
-Dormando
it simpler and easier to
> integrate the changes into default_engine. Would you like me to experiment
> with this approach?
Yes, if you could test that and show the results of either approach that'd
be great! You should still put it into its own engine for now, though.
-Dormando
Hey,
http://memcached.org/feedme :)
Preemptive thanks to anyone who decides to stand up and help!
-Dormando
> +dormando (who seems to have been dropped from the cc list)
>
> On Fri, Sep 23, 2011 at 9:46 AM, Mark Wong wrote:
> On Fri, Sep 23, 2011 at 9:30 AM, Josh Berkus wrote:
> > On 9/23/11 3:24 AM, Paul Lindner wrote:
> >> Contact Josh Berkus he may be able to get
Intel/AMD or a 10g card vendor could get you access to hardare
> > too..
> >
> >
>
> Could also try the folks at Oregon State University. They support a
> number of FOSS projects in that way and are generally a very cool bunch
> of folks to work with. Can do an intro if you like.
If they can help, sure I'd love an intro!
-Dormando
> Hi,
> I am trying to do some scalability analysis (scaling the number of clients)
> for Memcached.
> Is there any available benchmark for such experiments ?
>
> Also, I am curious to know about the number of clients that a memcached
> server can service in a typical deployment.
> By default, th
> I know that problably it would depend of my specific scenario, but in
> general, do you think it would be better to have a single instance of
> 60MB or 6 instances of 10MB on the same server?
> I wonder if having multiples instances would highly imprrove the
> concurrency capacity and if having
> > How many do you want it to run? After a point you have to start tuning
> > your OS kernel to reserve less RAM per TCP connection, but it'll scale to
>
> Which parameter is that?
It's a lot of parameters. Google for linux TCP tuning.
> Thanks for the reply.
> I would like to scale and see where the performance saturates.
> So, in a typical deployment, is 30-35K memcached clients (per server)
> possible ?
> Is it a common case ?
It's going to vary on hardware, kernel, OS, etc and moreso on the active
number of requests per s
p
as it's difficult to do this automatically, correctly, for most users.
It's coming up pretty soon in my TODO list though; we've been catching up
on the backlog with 1.4.
-Dormando
On Tue, 27 Sep 2011, Gonzalo de Pedro wrote:
> > It's coming up pretty soon in my TODO list though; we've been catching up
> > on the backlog with 1.4.
>
> Are you planning to implement this for version 1.6?
I can't/won't predict what version number that change will be in.
way.
This time you get some fancypants FEATURES like COMMANDS and SWITCHES and
FIDDLYDINKS and some awesome COUNTERS for judging how well you do things.
enjoy,
-Dormando
ly your read buffer
fills and you won't be able to send any packets until you first read some.
iirc libmemcached has a workaround for this by starting to call the read
handler callbacks while sending requests still.
-Dormando
> Hi,
>
> Thanks for your responses.
> Sorry, I should not have said C++ API. This was my wrapper over the
> libmemcached C API, 0.43
>
> Dormando, yes it is the binary protocol and I think what you mention
> is exactly what might be happening.
> The client seems to ge
ch is also
> > available here:
> >
> > git://github.com/vmiklos/memcached.git
>
> Hi,
>
> Ping?
>
> Thanks,
>
> Miklos
Thanks for this!
I don't have a systemd setup available; is there anyone who could verify
the script for me and give it a +1? If so it should go into 1.4.9, no
problem :)
Thanks,
-Dormando
Looks correct to me... Apologies to everyone if it isn't :P I applied it
and it's going into 1.4.8.
thanks!
On Thu, 29 Sep 2011, Nate wrote:
> At some point it looks like start-memcached was changed to fork an
> extra time, as the comments put it, "now that the tty is closed." I'm
> not sure wh
Also the rest of the stuff we added/fixed are totally fawesome.
-Dormando
Hey,
Enjoying 1.4.8? Thought I'd share some rough things that you guys may
enjoy:
https://github.com/dormando/mc-crusher
^ I've thrown my hat into the ring of benchmark utilities. This is
probably on par with some work Dustin's been doing, but I went in a
slightly different
> Hello all,
>
> Nice to be a part of the memcached community.
>
> Is there any way, using the current libmemcached library, to
> dynamically assign expiration to certain keys ?
> What I am trying to do is the following :
>
> # expire key in 100 seconds
> set key value 100
> # after 10 seconds, dec
e view.
If you're planning to deploy, this sounds like a perfect opportunity to go
test it. Run your cluster with the traffic you expect to hit and try it
both ways with a simple bench script. Keep the one you're happiest with,
or pick the simplest if it doesn't make a difference.
-Dormando
> Right. It does sound like we'll have to conduct some experiments.
> Would've been nice to get some other inputs though.
>
> We might make 2000-3000 mgets/sec. 100-500 keys each.
>
It's straight up math.
if (keylength * keycount > tcp_buffer) -> latency benefit from having
shorter mgets (or just
shop's support about their implementation of
memcached. I'm guessing that it doesn't support what you're trying to do,
but it's unlikely people here know how they set things up.
-Dormando
> Hello,
>
> I see from the API that the memcached counters are unsigned long
> integers.
> My questions is : why unsigned ? Is there any possibility to use
> signed integer counters with memcached ?
>
> Regards,
> Vlad
The counters are documented in the server's doc/protocol.txt (most of
them...)
lue without too much effort, I suppose? A "value" of 0 in uint64_t would
be the signed -9223372036854775808 or whatever it is.
-Dormando
>
> The Dev server has this issue:
> - items that are not yet expired (I had a sample with 35 minutes
> lifetime left) get removed from the cache
> - the memcache has plenty of free memory from the default 64MB
> - the server has 12GB of free memory
>
> Any good reason why a valid item is removed w
, lru, etc) is up to your application.
It still has a weakness of not being able to reassign memory if you put it
all into one slab class pool.
-Dormando
> Folks -- should I add this to the Fedora memcached rpm directly first?
>
Actually if you can hold *one* day on that, that'd be great. I'm going to
ship 1.4.9 as a bugfix release with this in there and a few other tiny
patches. The perf changes will be pushed to 1.4.10
ce release.
I've pushed over 10 billion requests through what's in the 14perf tree at
rates of over 1.2 million keys/sec sustained. I still have a few important
things to stress test before I mark it final, but that could happen within
the next week.
have fun,
-Dormando
o implement in C, and the
ascii protocol should be trivial to implement in any dynamic language, as
it already is with the ascii protocol.
We may not even do this, who knows. I'd rather have this so I can use it.
Sick of not being able to use binprot features wherever I want.
-Dormando
s with the numbers. That may end up with clients doing horrific
signed/unsigned incr/decr emulation layers though.
Argh. So much hate.
> # Quiet as a Flag
>
> Again, happy to take it as a consistent bit, but it's not a different
> command, just different handling of the command both in the server and the
> client.
I wrote my justification above. Hit me back with a stronger response, or
is it fine?
Thanks!
-Dormando
> > We had previously talked about an even tighter binary protocol, but perhaps
> > harder to generalize a parser around. This doesn't seem different
> > enough from the existing binary protocol to warrant introducing an
> > incompatibility.
>
> I honestly can't remember what else was removed in
> I don't have the notes from that discussion, but there was the dream of the
> "five byte get"
>
> A byte of magic, a byte of flags describing the parts, a byte of opcode,
> and a byte of key length, then a key, "a"
>
> The flags would include things like quiet, CAS?, keylenlen, etc...
>
>
On Wed, 19 Oct 2011, dormando wrote:
> > I don't have the notes from that discussion, but there was the dream of
> > the "five byte get"
> >
> > A byte of magic, a byte of flags describing the parts, a byte of opcode,
> > and a byte of key leng
y very wrong, but it should work. The
default will forever stay at 250 bytes because it's a good idea, and
increasing the limit may seriously inflate the internal item structure due
to alignment issues, at minimum one byte per item.
-Dormando
> On Wednesday, October 19, 2011 2:49:17 PM UTC-7, Dormando wrote:
>
> That startup option should flog you a dozen times first, and force you
> to
> agree that you're doing something very very wrong, but it should work.
> The
> default will forever
> [root@events memcached-1.2.8]# rpm -q libevent
just an aside; 1.2.8 is very very old. you should be on 1.4.9 if you're
building yourself.
> hello,
> i am trying to install memcached-1.4.9 on Centos 5.7
>
> i get these errors (the root one is easy) but not sure what i need to
> change for the others
>
> t/item_size_max.t 1/7 Item max size cannot be less than 1024
> bytes.
> t/item_size_max.t 2/7 Cannot set item size limit hi
ab size to get the slabs closer
to ideal.
Should get you a lot closer.
-Dormando
are binary blobs and it doesn't matter
what's in them. There may be an option you have to swith off in the
client, or the client may not actually be using binprot (run memcached
with -vvv to see what it thinks of your connecting clients).
-Dormando
lose 4 bytes.
Which gets you an awful lot closer. However; if your keys are 24 bytes
because they're base64 encoded, but you can get them down to 8 or 16 bytes
by hashing them via sha1, you can adjust the -n even lower.
-Dormando
> There's nothing like that currently. Last discussion I remember is that
> we decided against allowing binary keys at the client because we don't
> know what other clients may expect when trying to get that item.
>
> We can certainly reconsider that, but it's not been needed thus far.
What the h
> Calm down. It clearly wasn't 50% of the use cases given that it's just
> now come up. :)
It was part of the whole damn point of implementing the protocol. We
wanted three things: binary keys, proper quiet commands, and CAS on
everything. The rest is exactly the fucking same in ASCII. I reserve
> I'd prefer a flag that I have to _enable_ to have the library verify my
> damn keys. Let the user do what he wants to do and don't expect every
> client user to be a moron. (just like the stupid ubuntu installations that
> adds all sorts of stupid aliases for rm etc).
Most users won't ever know
> I'd prefer a flag that I have to _enable_ to have the library verify my
> damn keys. Let the user do what he wants to do and don't expect every
> client user to be a moron. (just like the stupid ubuntu installations that
> adds all sorts of stupid aliases for rm etc).
(yes, I'm saying users tend
> >> I'd prefer a flag that I have to _enable_ to have the library verify my
> >> damn keys. Let the user do what he wants to do and don't expect every
> >> client user to be a moron. (just like the stupid ubuntu installations
> >>that
> >> adds all sorts of stupid aliases for rm etc).
> >
> >Most
>
> Nobody fucking does that. Get over it, yo. People read the minimum amount
> of crap they have to read until it works. Everyone else doesn't have a
> hard time finding work.
>
> Also; because when you don't, people switch to to other systems because
> they believe it's "easier", or they complain
> There were strong arguments for keeping keys compatible with both ASCII and
> binary clients, to the point where it was decided to keep
> parity between the two.
>
> We can certainly revisit that now, but I don't remember anyone advocating
> for arbitrary bytes in keys other than asking if
> On Monday, October 24, 2011 10:50:39 PM UTC-4, Dormando wrote:
>
> Just... add a flag so it can be turned off? It's a sane default, but
> hurtful if you ever need blob keys. None of the original clients checked
> keys, and that sucked.
>
> Makes sen
> > It still has a weakness of not being able to reassign memory if you put it
> > all into one slab class pool.
>
> I have 16 fixed sizes (ranging from 1K-6K, all slightly irregular size
> [e.g. 1028 bytes]), so I will use the slab allocator every time I need
> one of these fixed sizes. So I will
catches on fire or if you restart
memcached. If you can't handle a restart, you can't handle failure. If
you're not designing for failure, you suck.
I'm not saying such a feature will never be added; but it won't be to
solve the issue in the way you frame it.
-Dormando
You're yelling at a moderation queue, for what it's worth.
On Mon, 31 Oct 2011, rusherding wrote:
> sorry for all-caps subject ,because my previous email was deleted...i
> wrote another and all-caps to attract your attention.
>
> On 10月30日, 上午2时38分, Dustin wrote:
> > On Saturday, October 29, 20
Hey,
http://code.google.com/p/memcached/wiki/ReleaseNotes1410
See release notes or code for more information. I would be surprised if
any decent host would top out before overloading its network card. This
one is a beast.
-Dormando
Thanks, as always!
On Wed, 9 Nov 2011, Paul Lindner wrote:
> Fedora RPMs up for testing. This release looks nce!
>
> On Wed, Nov 9, 2011 at 5:00 PM, dormando wrote:
> Hey,
>
> http://code.google.com/p/memcached/wiki/ReleaseNotes1410
>
> See releas
rns out you're just hitting maxconns; try increasing -c further
(perhaps also lowering -m by a few megs), or try the new "-o
maxconns_fast" option in 1.4.9+, which immediately rejects conns over the
limit.
-Dormando
> Hi All,
> My scenario needs me to retrieve multiple objects that have the same key.
> Infact my scenario needs me to identify objects using multiple keys too,
> but I can solve the multiple keys problem by adding one more entry to
> memcached. So thats not my question. Is it possible to store m
> Well, if I need to put another object in the collection, I need to first get
> it the existing object from the cache. And then insert this new object
> within that collection. Reducing performance by that much. But I understand
> that perf will not drop considerably since a get is much faster,
> Dormando,Quick question.
>
> So if I were to
> put (key, array_of_size_3)
> and then
> append (key, new_item)
>
> value = get (key)
> size of value will be 4 ?
if array_of_size_3 is "3 bytes", and new_item is "1 byte", then yes.
remember that if
Just specify 0 and it won't expire.
On Wed, 23 Nov 2011, Siddharth Jagtiani wrote:
> I noticed exp is int, not uint. So I wonder if I give -1 will it consider no
> expiry :).
> Siddharth
>
> On Wed, Nov 23, 2011 at 4:25 PM, Siddharth Jagtiani
> wrote:
> Hi,
> I am wondering if there is a
d
> also, i know this is unlikely in probably 99.999% of cases but it wud be
> great to factor in the maths behind science.
I sorta answered this above. Should put this into a wiki page I guess...
-Dormando
> But which client? Usually if you need memcache to scale you will be
> running many clients in parallel - and if they are doing single-key
> operations in many cases adding more servers will make them completely
> separate. It is only multi-gets with many small keys that don't
> scale forever.
Keys can have any characters except newlines or spaces while using the
ASCII protocol. If using the binary protocol exclusively there's no limit
to what, just the length (250 bytes)
On Wed, 30 Nov 2011, Siddharth Jagtiani wrote:
> Can it be "-613748077" ?
>
> On Wed, Nov 30, 2011 at 1:57 PM, smal
> I'm surprised how little I can find on this topic... references to the
> stampeding problem (which is resolved by using get_locked) are about
> all the info I can find. Sorry in advance if i'm missing some earlier
> discussion - i've searched quite a bit and can't find anything about
> it.
>
> Ar
> Hi,
>
> I'm developing some applications using memcached and I saw that the
> incr operation does not update the key's expiration, requiring the
> usage of get + set if I want to update the expiration.
>
> Is there a special reason to this behavior?
It's more flexible if that's the default. you
> @dormando, great response this is almost exctly what i had in mind, i.e.
> grouping all of your memcached servers into logical pools so as to
> avoid hitting all of them for every request. infact, a reasonable design, for
> a very large server installation base, would be to ai
of your applications, or adjust it to run many
connections in parallel, or to use binary protocol and stack SET's.
memcached will go much faster, but a single process with a single
connection will be limited by the roundtrip latency.
Using mc-crusher and multigets or multisets over local
> Dormando,
>
> What exactly do you mean by stack? Do you mean buffering?
>
> I've tried a few options from libmemcached: buffering and no_reply.
> Both of them seem to make things much much faster.
>
> With buffering it does 150k/s with no_reply ~750k/s. So i
> Hello,
>
> After 3 weeks with memcached 1.4.10 in production, today we have start
> getting randomly this error:
>
> SERVER_ERROR out of memory storing object with memcached
>
> I can reproduce it with a simple set+get loop, this is the Python
> script that I have used (running the script from 6
tps://github.com/memcached/memcached/commit/ca5016c54111e062c771d20fcc4662400713c634
>
> - using exptime=0 it doesn't happen.
I probably fixed this a few weeks ago in my branch, but I'm still wringing
the bugs from it. If you can hold on for a few days for the beta and test
that when I post it, it should be better.
Thanks!
-Dormando
> We're using memcached to cache content generated by another
> application on the same server. Items that are cached are set to never
> expire, as the application currently self decides when it will refresh
> items already in the cache.
>
> We run memcache with verbose output, and I occasionally s
ith it, and what the exact errors are.
If you could get it to crash under gdb and get a backtrace as well, that's
major bonus points.
Thanks, and sorry,
-Dormando
> dormando, with a new script setting a random exptime I can reproduce the
> problem in a fresh memcached 1.4.10 (it doesn't happen with earlier versions):
>
> https://gist.github.com/1564556
>
> With the first evictions memcached starts reporting "SERVER_ERROR ou
y very long
time. *Please* spend some time hacking at the beta if you can, and provide
any feedback.
Thanks,
-Dormando
conns_fast? That way you'll rule out many of the bugfixes we've had
since then, and if the one running maxconns_fast starts EPIPE'ing, you'll
see an error message when you attempt to connect to it if it is connected
to a max conns error.
-Dormando
> El 09/01/12 06:12, dormando escribió:
>
> > Hey, could you please try to reproduce the issue with 1.4.11-beta1:
> > http://code.google.com/p/memcached/wiki/ReleaseNotes1411beta1
> >
> > I've closed the logic issues and fixed a few other things besides. Would
> There are archived discussions floating around about this subject,
> particularly with SSD. Are there open source works (implementation,
> src code, interface description, etc) available for people to take a
> look ?
There is a working and well used engine implementation in the engine-pu
branch:
http://code.google.com/p/memcached/wiki/ReleaseNotes1411rc1
Tossed in a few more fixes from the issue tracker, punting on the rest for
1.4.12 since this is so late.
I'm going to leave this up for another day or two while I work on the wiki
a bit and try to come up with other tests. Slab rebalanci
s and some complications with getting
slab reassign to work.
have fun,
-Dormando
> El 17/01/12 06:36, dormando escribió:
>
> > http://code.google.com/p/memcached/wiki/ReleaseNotes1411
>
> We're having problems building this release with old GCC versions, for
> example:
>
> * RHEL-4 (GCC 3.4.6), on both 32 and 64 bits:
> thread.c:98: warnin
> El 17/01/12 06:36, dormando escribió:
>
> > http://code.google.com/p/memcached/wiki/ReleaseNotes1411
>
> We're having problems building this release with old GCC versions, for
> example:
>
> * RHEL-4 (GCC 3.4.6), on both 32 and 64 bits:
> thread.c:98: warnin
> > El 17/01/12 06:36, dormando escribió:
> >
> > > http://code.google.com/p/memcached/wiki/ReleaseNotes1411
> >
> > We're having problems building this release with old GCC versions, for
> > example:
> >
> > * RHEL-4 (GCC 3.4.6), on
> El 17/01/12 21:27, dormando escribió:
>
> > > could you please try 1.4.11 with the attached patch on all platforms? I'm
> > > taking a bit of a guess since google isn't very familiar with this macro.
> > >
> > > for bonus points; add
> > El 17/01/12 21:27, dormando escribió:
> >
> > > > could you please try 1.4.11 with the attached patch on all platforms?
> > > > I'm
> > > > taking a bit of a guess since google isn't very familiar with this
> > > > macro.
> @dormando, results after applying your last patch:
Thank you!
> - build works on RHEL-{4,5,6} both i386 and x86_64 :)
>
> - RHEL 4 and 5 return "emulated atomics" and RHEL-6 "gcc atomics"
Even RHEL5 64bit says "emulated atomics" ? That shouldn't
Yo all,
I'll be speaking at SCALE (http://socallinuxexpo.org/) on saturday on the
topic of "modern memcached" - underused existing features, and talking
about current and future work on the project.
> Is there any guarantee that search is not NULL on Line 133? I think if
> Line 107 is true and takes the branch on Line 108, there is nothing
> between there and Line 133 that sets the value for search. So, if
> slabs_alloc fails to allocate memory in all the instances and it
> remains NULL, we ca
> El 18/01/12 19:04, dormando escribió:
>
> > > Closer... Can you try the attached patch against 1.4.11? could've sworn
> > > I'd tested the other path, but I guess not :(
> > >
> > > I added some prints at the bottom that should print some
> Today Jan 18, 2012 at 09:51 dormando wrote:
>
> > > b) i386 = "manual" compilation works, but a I'm getting an error
> > > building
> > > the RPM package, it might be related with gcc flags, I'm working on it..
> > > rpmbuild
> We're having some issues with retrieving cached keys from our
> memcached server.
>
> Tested different data with expiry times of 30s, 120s, and 600s - the
> test was: try to get the data three times in quick succession.
>
> Obviously, we're expecting the first one from db but then the next two
>
1 - 100 of 1190 matches
Mail list logo
