At least initially we would use normal java serialization. Since we are talking about using prepend to put the most recent entries in our list, we cannot use gzip anymore since it would not gel with prepending bytes.
Did you use a custom serialization scheme? Or do you mean the binary serializer? Because at some point you HAVE to serialize something to a byte[], right? Did you prepend data at all? I don't see many people talking about utilizing the prepend/append methods within the protocol, I'm also trying to figure out why this is. Even though it appears the network fetch of even 10,000 longs isn't much data, since we have so many of these lists to process it ends up becoming a large network hit once you've done it 200k+ times. Jim On Jan 19, 3:05 pm, dormando <[email protected]> wrote: > How're you serializing the list? > > In cases where I've had to work with that, we ensure the value is a flat > list of packed numerics, and run no serialization on them. Then you have > the overhead of a network fetch, but testing values within the list is > nearly free. > > -Dormando > > On Tue, 19 Jan 2010, jim wrote: > > Hi: > > > I've recently started on a project that is utilizing memcached to > > buffer lists of 'long's that represent key values for larger more > > detailed objects to load if needed. This list will have a maximum of > > 10,000 items. > > > ex: > > > List of longs: > > key : actList value : maximum 10,000 longs sorted chronologically > > based on their 'detailed' object timestamp. > > > Detailed entry: > > key : act_<id> value : text serialized object representing detailed > > activity object. <id> is contained on the 'List of longs' above. > > > My question is there are times where we would like to load only the > > first block of entries from the list of longs, say 10 records. We > > would then look at those 10 records and see if they are new based on > > what a currently displayed list shows and if so grab these new entries > > without pulling all 10,000 across from cache only to utilize the > > relatively small number of new entries on the top of the list. This > > kind of goes hand in hand with the prepend (or append) operations > > where when new activities arrive we push these new activities into the > > front of the 'List of longs' and it's these new entries that clients > > may be interested in if they do not yet have them. > > > My question is, is there a way to do this? Is there a way to grab > > only X bytes from a value in memcache? I read over the protocol > > document and it doesn't appear there is. Is there any interest or > > valid use case that this seems to fill for other users? > > > An alternate solution I can see if to store the 'list of longs' as a > > flat list of keys with a naming convention, such as actList_1, > > actList_2, etc. However, this will obviously lead to an extremely > > long key name in the multi-key 'get' as well as lots of churn in > > managing these objects .. so it appears far less than ideal. However, > > we have many (200,000) lists of these 10,000 item 'List of longs' > > which leads to pulling loads of data over the wire when a large update > > occurs that needs to be communicated with cache ... it would be much > > more efficient to only go after a certain number of bytes in cache vs. > > the entire cached value. > > > Any other thoughts? Ideas? > > > Thank you. > > Jim
