On 12. apr. 2011, at 18.56, Nelz wrote: > What is the difference between GAT and GATQ? >
GAT will return with "KEY NOT FOUND" if the item isn't in the cache, whereas GATQ only returns information about the items found. Cheers, Trond > - Nelz > > On Tue, Apr 12, 2011 at 02:52, Trond Norbye <[email protected]> wrote: >> >> On 12. apr. 2011, at 05.55, dormando wrote: >> >>> ps. folks please look this over and evaluate. Do you understand >>> everything? Does anything suck? Need more clarification? Whatever? >>> >>> http://code.google.com/p/memcached/downloads/detail?name=memcached-1.6.0_beta1.tar.gz >>> ^ easy-bake oven form beta release. Passes tests on a bunch of platforms, >>> but possibly not OpenBSD. >>> >> >> I've fixed that bug. See: >> https://github.com/memcached/memcached/commit/cc3941084188195fc8b43fcdc05cec3dab5a4bd4 >> >> Cheers, >> >> Trond >> >> >>> Make evaluating! Give major feedback. >>> >>> -Dormando >>> >>> On Mon, 11 Apr 2011, Trond Norbye wrote: >>> >>>> What's new in memcached >>>> ======================= >>>> >>>> (part two - new feature proposals) >>>> >>>> Table of Contents >>>> ================= >>>> 1 Protocol >>>> 1.1 Virtual buckets! >>>> 1.2 TAP >>>> 1.3 New commands >>>> 1.3.1 VERBOSITY >>>> 1.3.2 TOUCH, GAT and GATQ >>>> 1.3.3 SET_VBUCKET, GET_VBUCKET, DEL_VBUCKET >>>> 1.3.4 TAP_CONNECT >>>> 1.3.5 TAP_MUTATION, TAP_DELETE, TAP_FLUSH >>>> 1.3.6 TAP_OPAQUE >>>> 1.3.7 TAP_VBUCKET_SET >>>> 1.3.8 TAP_CHECKPOINT_START and TAP_CHECKPOINT_END >>>> 2 Modularity >>>> 2.1 Engines >>>> 2.2 Extensions >>>> 2.2.1 Logger >>>> 2.2.2 Daemon >>>> 2.2.3 ASCII commands >>>> 3 New stats >>>> 3.1 Stats returned by the default stats command >>>> 3.1.1 libevent >>>> 3.1.2 rejected_conns >>>> 3.1.3 stats related to TAP >>>> 3.2 topkeys >>>> 3.3 aggregate >>>> 3.4 settings >>>> 3.4.1 extension >>>> 3.4.2 topkeys >>>> >>>> >>>> 1 Protocol >>>> ~~~~~~~~~~~ >>>> >>>> Intentionally, there is no significant difference in protocol over >>>> 1.4.x. There is one minor change, but it should be transparent to >>>> most users. >>>> >>>> 1.1 Virtual buckets! >>>> ===================== >>>> >>>> We don't know who originally came up with the idea, but we've heard >>>> rumors that it might be Anatoly Vorobey or Brad Fitzpatrick. In lieu >>>> of a full explanation on this, the concept is that instead of mapping >>>> each key to a server we map it to a virtual bucket. These virtual >>>> buckets are then distributed across all of the servers. To ease the >>>> introduction of this we've assigned the two reserved bytes in the >>>> binary protocol for specifying the vbucket id, which allowed us to >>>> avoid protocol extensions. >>>> >>>> Note that this change should allow for complete compatibility if the >>>> clients and the server are not aware of vbuckets. These should have >>>> been set to 0 according to the original binary protocol specification, >>>> which means that they will always use vbucket 0. >>>> >>>> The idea is that we can move these vbuckets between servers such that >>>> you can "grow" or "shrink" your cluster without losing data in your >>>> cache. The classic memcached caching engine does _not_ implement >>>> support for multiple vbuckets right now, but it is on the roadmap to >>>> create a version of the engine in memcached to support this (it is a >>>> question of memory efficiency, and there are currently not many >>>> clients that support them). >>>> >>>> Defining this now will allow us to start moving down the path to >>>> vbuckets in the default_engine and allow other engine implementors to >>>> consider vbuckets in their design. >>>> >>>> You can read more about the mechanics of it here: >>>> [http://dustin.github.com/2010/06/29/memcached-vbuckets.html] >>>> >>>> However, you _cannot_ use a mix of clients that are vbucket aware and >>>> clients who don't use vbuckets, but then again it doesn't make sense >>>> to use a vbucket aware backend if your clients don't know how to >>>> access them. This is why we believe a protocol change isn't >>>> warranted. >>>> >>>> Defining this now will allow us to start moving down the path to >>>> vbuckets in the default_engine and allow other engine implementors to >>>> consider vbuckets in their design. >>>> >>>> 1.2 TAP >>>> ======== >>>> >>>> In order to facilitate vbucket transfers, among other use cases where >>>> people want to see what's inside the server, we added to the binary >>>> protocol a set of commands collectively called TAP. The intention is >>>> to allow "clients" to receive a stream of notifications whenever data >>>> change in the server. It is solely up to the backing store to >>>> implement this, so it can make decisions about what resources are used >>>> to implement TAP. This functionality is commonly needed enough though >>>> that the core is aware of it, leaving specific implementation to >>>> engines. >>>> >>>> 1.3 New commands >>>> ================= >>>> >>>> There are a few new commands available. The following sections >>>> provides a brief description of them. Please check protocol_binary.h >>>> for the implementation details. >>>> >>>> 1.3.1 VERBOSITY >>>> ---------------- >>>> >>>> We did not have an equivalent of the verbosity command in the textual >>>> protocol. This command allows the user to change the verbosity level >>>> on your running server by using the binary protocol. Why do we need >>>> this? There is a command line option you may use to disable the ascii >>>> protocol, so we need this command in order to change the logging level >>>> in those configurations. >>>> >>>> 1.3.2 TOUCH, GAT and GATQ >>>> -------------------------- >>>> >>>> One of the problems with the existing commands in memcached is that >>>> you couldn't tell the memcached server that the object is still valid >>>> and we just want a longer expiration. Normally you want to put an >>>> expiry time on the objects, so that you can get an indication if your >>>> cache is big enough (by watching the eviction stats.. if your >>>> memcached server has a high eviction rate your cache isn't big enough >>>> for what you want to have in there). The normal idea is that the items >>>> you're normally using would be bumped to the front of your LRU (and >>>> hence not be kicked out immediately). >>>> >>>> The touch command lets you set the expiry time for an object without >>>> retrieving the object. In most cases, you will not want to do this >>>> unless you provide a CAS value to ensure that you're touching the >>>> correct version of the object. >>>> >>>> GAT means "get and touch" and returns the object in addition to >>>> setting a new expiration time. This allows you to have a rolling >>>> window of expiry that has a TTL in addition to the access time. For >>>> example, you can instruct memcached to allow an object to live no >>>> later than five minutes after the last time it was access (but as >>>> always, it may expire sooner). >>>> >>>> 1.3.3 SET_VBUCKET, GET_VBUCKET, DEL_VBUCKET >>>> -------------------------------------------- >>>> >>>> These commands are used to set, get or delete a vbucket on the server. >>>> >>>> 1.3.4 TAP_CONNECT >>>> ------------------ >>>> >>>> Connect and request that the server initialize a TAP stream. >>>> >>>> The point of this command is to allow clients to connect and specify a >>>> few things about the data they wish to receive. Specifically, the >>>> client will typically specify a date either in the past or in the >>>> future along with specifying a vbucket. The server will then stream >>>> data mutated since that given date or if a future date is specified, >>>> only stream new mutations as they arrive. The specific details about >>>> which mutations to send may vary on implementation. >>>> >>>> 1.3.5 TAP_MUTATION, TAP_DELETE, TAP_FLUSH >>>> ------------------------------------------ >>>> >>>> TAP_MUTATION is a notification that an item changed value in the >>>> server. >>>> >>>> The mutation typically comes with the new value. >>>> >>>> TAP_DELETE is a notification that a key was deleted on the server. >>>> >>>> Finally, to avoid having to send a complete list of all the keys in >>>> the server when the user issues a flush, we can send a single message >>>> (TAP_FLUSH) representing the flush. Please note that the FLUSH >>>> message means _ALL_ vbuckets, and not just a single vbucket. >>>> >>>> 1.3.6 TAP_OPAQUE >>>> ----------------- >>>> >>>> To allow storage engines to send their own messages over the tap >>>> stream between each other, a tap opaque message is defined. It is >>>> completely up to the storage engine to specify the internal layout of >>>> the package. >>>> >>>> 1.3.7 TAP_VBUCKET_SET >>>> ---------------------- >>>> >>>> This is a message requesting a vbucket set. It is similar to the >>>> set_vbucket command, with the difference that this message comes over >>>> a tap connection (with the extra info a tap message contains) >>>> >>>> 1.3.8 TAP_CHECKPOINT_START and TAP_CHECKPOINT_END >>>> -------------------------------------------------- >>>> >>>> The checkpoint start and end messages may be used by engine who wants >>>> to use checkpoints. Checkpoints are an optional feature that may be >>>> used by some engines to allow clients to start at a checkpoint >>>> position. By doing so, the client need not do a full "backfill" even >>>> if it is revisiting a server after having been gone for a while. The >>>> TAP_CHECKPOINT_START tells a client that it's the start of a new >>>> checkpoint, and the TAP_CHECKPOINT_END tells the client when it's >>>> received everything for that given checkpoint. >>>> >>>> 2 Modularity >>>> ~~~~~~~~~~~~~ >>>> >>>> As we mentioned in the first email on changes, one big difference with >>>> this new work is that we've tried to refactor memcached into being a >>>> modular application instead of being monolithic. In the future, we'd >>>> like to make the command parser as a separate module, so that we may >>>> load the parsers separately. >>>> >>>> 2.1 Engines >>>> ============ >>>> >>>> We've done a lot of work trying to refactor the code in memcached to >>>> avoid the tight coupling between the command protocol parser and the >>>> actual item storage. >>>> >>>> The idea with the engine interface is that the memcached process loads >>>> a dynamically loadable object and calls a well known function to get a >>>> set of function pointers. All communication between the memcached >>>> process and the engine is performed through these function >>>> pointers. The memcached process provides a set of services to the >>>> engine as well through another set of function pointers. >>>> >>>> The beauty of this is that the user may choose between a set of >>>> different storage engines that suites their runtime >>>> environment. People have different requirements for their server. Some >>>> people need ACID, others may prefer ecstacy ;-) The storage interface >>>> may let them design their app by using the memcached protocol, and >>>> they can just swap in the backend that suites their needs (may it be >>>> persistence, replication (sync or lazily) etc..) >>>> >>>> 2.2 Extensions >>>> =============== >>>> >>>> The item storage isn't the only place we've tried to create a level of >>>> modularity. People run memcached in different environments with >>>> different requirements. You specify the extensions you want to use by >>>> adding the -X command line argument. >>>> >>>> 2.2.1 Logger >>>> ------------- >>>> >>>> We've seen a lot of different requests when it comes to logging. Some >>>> want it to a file, some to syslog (or Windows event log) and some want >>>> it to standard out. By default memcached will print to stderr, but >>>> you may specify a different logger by loading the appropriate module >>>> with the -X command line argument >>>> >>>> 2.2.2 Daemon >>>> ------------- >>>> >>>> You might want to have some daemons providing extra services inside >>>> your memcached server. Examples would be things like a doors server >>>> to provide additional access to your server (Trond's favorite), or >>>> perhaps a "dispatcher" offering a threadpool for your engines to >>>> use?). >>>> >>>> 2.2.3 ASCII commands >>>> --------------------- >>>> >>>> If you really need to extend the ASCII protocol, you may now load >>>> additional ASCII commands as loadable modules. We don't need a >>>> separate module for binary commands, because those are already handled >>>> inside memcached due to the fixed semantics on the protocol. This >>>> isn't necessarily encouraged, but sometimes it is required to get >>>> something done quick. >>>> >>>> 3 New stats >>>> ~~~~~~~~~~~~ >>>> >>>> There are a number of new stats introduced. The key supplied in the >>>> status command is passed to the storage engine to allow the storage >>>> engine to add extra information to the existing stats commands, and to >>>> create their own stat commands. >>>> >>>> 3.1 Stats returned by the default stats command >>>> ================================================ >>>> >>>> 3.1.1 libevent >>>> --------------- >>>> >>>> Over the time we've seen a lot of bugs around people using an old >>>> version of libevent. That's part of the reason why we bundle a well >>>> known version of libevent in the release distribution. Memcached >>>> checks the libevent version during startup, and will refuse to start >>>> if the one used is too old. Since most operating systems use shared >>>> libraries these days, you might be using another version than the one >>>> you originally used when you first built memcached. In order for us to >>>> see which library people are using we decided to put it into the stats >>>> as well. >>>> >>>> 3.1.2 rejected_conns >>>> --------------------- >>>> >>>> The number of times a connection attempt was refused (due when we're >>>> hitting the maximum number of connections. >>>> >>>> 3.1.3 stats related to TAP >>>> --------------------------- >>>> >>>> There are a number of stats related to the packages used in the TAP >>>> protocol. These stats will only appear if they are non-zero: >>>> >>>> tap_checkpoint_start_received tap_checkpoint_start_sent >>>> tap_checkpoint_end_received tap_checkpoint_end_sent >>>> tap_connect_received tap_delete_received tap_delete_sent >>>> tap_flush_received tap_flush_sent tap_mutation_received >>>> tap_mutation_sent tap_opaque_received tap_opaque_sent >>>> tap_vbucket_set_received tap_vbucket_set_sent >>>> >>>> 3.2 topkeys >>>> ============ >>>> >>>> You may get information about the most popular keys in memcached by >>>> exporting the environment variable MEMCACHED_TOP_KEYS to the number of >>>> keys you would want memcached to keep track of. There is no such thing >>>> as a free lunch, so enabling this can have a small memory and speed >>>> impact. We've decided to _disable_ this by default, so you need to >>>> export this variable to enable the feature. Ex: >>>> >>>> me@localhost:> MEMCACHED_TOP_KEYS=10 ./memcached >>>> >>>> Running "stats topkeys" would return something like >>>> >>>> STAT my-key2 get_hits=0,get_misses=1,cmd_set=0,incr_hits=0, >>>> incr_misses=0,decr_hits=0,decr_misses=0,delete_hits=0, >>>> delete_misses=0,evictions=0,cas_hits=0,cas_badval=0, >>>> cas_misses=0,ctime=2,atime=2 >>>> STAT my-key1 >>>> get_hits=1,get_misses=0,cmd_set=1,incr_hits=0, >>>> incr_misses=0,decr_hits=0,decr_misses=0,delete_hits=0, >>>> delete_misses=0,evictions=0,cas_hits=0,cas_badval=0, >>>> cas_misses=0,ctime=12,atime=12 >>>> >>>> (Line breaks and indentations added to make it more readable in this >>>> document): >>>> >>>> 3.3 aggregate >>>> ============== >>>> >>>> The combination of the storage engine interface and the SASL auth >>>> allows for the combination of a connection-based stats. The aggregate >>>> subcommand is used to aggregate the stats from all of the connections >>>> on the server. The stats returned from the aggregate subcommand is the >>>> same as the normal stats command. >>>> >>>> 3.4 settings >>>> ============= >>>> >>>> There are times an engine may want to share details about it's >>>> configuration through stats. This argument to stats will get you >>>> there. >>>> >>>> Just to show a couple of examples... >>>> >>>> 3.4.1 extension >>>> ---------------- >>>> >>>> Displays one of the extensions loaded (may appear multiple times). >>>> >>>> ex: >>>> >>>> STAT logger syslog >>>> STAT ascii_extension scrub >>>> STAT ascii_extension noop >>>> STAT ascii_extension echo >>>> >>>> 3.4.2 topkeys >>>> -------------- >>>> >>>> The number of keys we are monitoring. >>>> >>>> There may be many other settings exposed, depending on the engine's >>>> configuration. >>>> >>>> >> >>
