That's pretty much exactly what we do in some cases. prepend/append packed bytes so the list is in order. Then we pull the data and read as much as we need without touching serialization.
If the overhead is a big deal you could fiddle with it a bit. Store 100 or 1,000 items per list and use multiple keys (if you can divide the numbers by something to be able to do that easily). Or have a main 10k list and then a separate "recent" key that you expire once it gets past 100 items or so. Then you're still occasionally doing the full fetch, but not nearly as often. Updating both keys ends up being the same prepend operation. -Dormando On Tue, 19 Jan 2010, jim wrote: > At least initially we would use normal java serialization. Since we > are talking about using prepend to put the most recent entries in our > list, we cannot use gzip anymore since it would not gel with > prepending bytes. > > Did you use a custom serialization scheme? Or do you mean the binary > serializer? Because at some point you HAVE to serialize something to > a byte[], right? Did you prepend data at all? I don't see many > people talking about utilizing the prepend/append methods within the > protocol, I'm also trying to figure out why this is. > > Even though it appears the network fetch of even 10,000 longs isn't > much data, since we have so many of these lists to process it ends up > becoming a large network hit once you've done it 200k+ times. > > Jim > > On Jan 19, 3:05 pm, dormando <[email protected]> wrote: > > How're you serializing the list? > > > > In cases where I've had to work with that, we ensure the value is a flat > > list of packed numerics, and run no serialization on them. Then you have > > the overhead of a network fetch, but testing values within the list is > > nearly free. > > > > -Dormando > > > > On Tue, 19 Jan 2010, jim wrote: > > > Hi: > > > > > I've recently started on a project that is utilizing memcached to > > > buffer lists of 'long's that represent key values for larger more > > > detailed objects to load if needed. This list will have a maximum of > > > 10,000 items. > > > > > ex: > > > > > List of longs: > > > key : actList value : maximum 10,000 longs sorted chronologically > > > based on their 'detailed' object timestamp. > > > > > Detailed entry: > > > key : act_<id> value : text serialized object representing detailed > > > activity object. <id> is contained on the 'List of longs' above. > > > > > My question is there are times where we would like to load only the > > > first block of entries from the list of longs, say 10 records. We > > > would then look at those 10 records and see if they are new based on > > > what a currently displayed list shows and if so grab these new entries > > > without pulling all 10,000 across from cache only to utilize the > > > relatively small number of new entries on the top of the list. This > > > kind of goes hand in hand with the prepend (or append) operations > > > where when new activities arrive we push these new activities into the > > > front of the 'List of longs' and it's these new entries that clients > > > may be interested in if they do not yet have them. > > > > > My question is, is there a way to do this? Is there a way to grab > > > only X bytes from a value in memcache? I read over the protocol > > > document and it doesn't appear there is. Is there any interest or > > > valid use case that this seems to fill for other users? > > > > > An alternate solution I can see if to store the 'list of longs' as a > > > flat list of keys with a naming convention, such as actList_1, > > > actList_2, etc. However, this will obviously lead to an extremely > > > long key name in the multi-key 'get' as well as lots of churn in > > > managing these objects .. so it appears far less than ideal. However, > > > we have many (200,000) lists of these 10,000 item 'List of longs' > > > which leads to pulling loads of data over the wire when a large update > > > occurs that needs to be communicated with cache ... it would be much > > > more efficient to only go after a certain number of bytes in cache vs. > > > the entire cached value. > > > > > Any other thoughts? Ideas? > > > > > Thank you. > > > Jim >
