Hi, I tried several different way, but still can't get it work appropriately.
First, I flush cache, no luck. Second, not using namespace, no luck. Then I wrote a simple loop which fetch users object from DataStore and store them to memcache which isn't related any other codes. The code fetches 12k object and stored 12k without error/exception, but memcache.stats() reported that only about half of them are stored. Finally, I use memcache.set() instead of set_multi(), but the result is same. Tried 12k stores, stored about 6k. memcache.stats() reports: bytes 13518695 items 6071 I think this doesn't hit any limit. On Dec 3, 1:58 pm, naan <[email protected]> wrote: > No, they are not in the same request. get_multi is called within 5 > minutes after set_multi is called. User object is firly small (less > than 1k) and total number of items in memcache is less than 50k, so it > won't be more than 100MB. User object contains strings, integers and > integer list. No blobs, nor string list. Again, all set_multi call > seems success, no error, no exception occurred. > > Thanks, > Kazuho > > On Dec 3, 1:14 pm, "Ikai L (Google)" <[email protected]> wrote: > > > Are you doing the get_multi and set_multi within the same request, or are > > the items disappearing afterwards? There's a 100mb limit to the number of > > items you can be storing in Memcache, but I'm not sure this is what you're > > hitting - the least recently used objects would be getting expired. What > > kind of data is stored in the User object? > > > ---------- Forwarded message ---------- > > From: naan <[email protected]> > > Date: Thu, Dec 3, 2009 at 12:51 PM > > Subject: [google-appengine] Re: Weird memcached issue > > To: Google App Engine <[email protected]> > > > I try to store 100 users object per request. memcache.set_multi seems > > working (return an empty array which means all data stored > > successfully), but get_multi returns only 9-15 of them. It worked > > yesterday morning, then stopped working afternoon. it seems that I > > can't store new user data into memcache, but still can get old cached > > data with get_multi. > > > The test code you gave me works fine. And my code works fine too on > > another GAE instance. Hmm.... Does call memcache.flush_all() will > > solve the issue? > > > On Dec 3, 12:04 pm, "Ikai L (Google)" <[email protected]> wrote: > > > What is being returned by get_multi? None or empty dictionary? Is a > > > namespace being specified? > > > > Here's some test code that is working for me when deployed. What might you > > > be doing differently from this? > > > > from google.appengine.ext import webapp > > > from google.appengine.ext.webapp.util import run_wsgi_app > > > from google.appengine.ext import db > > > > from google.appengine.api import memcache > > > > class MyThing(db.Model): > > > name = db.StringProperty() > > > > def __str__(self): > > > return "MyThing: %s" % self.name > > > > class MemcacheTest(webapp.RequestHandler): > > > def get(self): > > > stats = memcache.get_stats() > > > > println(self, "<b>Cache Hits:%s</b><br>" % stats['hits']) > > > println(self, "<b>Cache Misses:%s</b><br><br>" % stats['misses']) > > > > memcache.set("data", "My data") > > > data = memcache.get("data") > > > println(self, "Set 'data' -> 'My data'. Get 'data' -> " + data) > > > > key_range = range(1000) > > > mapping = { } > > > > for i in key_range: > > > mapping["key%d" % i] = "value %d" % i > > > > memcache.set_multi(mapping) > > > println(self, "Set %d values using set_multi" % len(mapping)) > > > > cached_mapping = memcache.get_multi(mapping.keys()) > > > println(self, "Retrieved %d values using get_multi" % > > > len(cached_mapping)) > > > > namespace = "synccache" > > > println(self, "Testing namespaces with namespace '%s'" % namespace) > > > memcache.set_multi(mapping, namespace=namespace) > > > println(self, "Set %d values using set_multi in namespace '%s'" % ( > > > len(mapping), namespace )) > > > > cached_mapping = memcache.get_multi(mapping.keys(), > > namespace=namespace) > > > println(self, "Retrieved %d values using get_multi in namespace '%s'" > > % > > > ( len(cached_mapping), namespace )) > > > > println(self, "Testing objects") > > > obj_map = {} > > > for i in key_range: > > > thing = MyThing(name="thing %d" % i) > > > obj_map[thing.name] = thing > > > > namespace = "things" > > > println(self, "Testing namespaces with namespace '%s'" % namespace) > > > memcache.set_multi(obj_map, namespace=namespace) > > > println(self, "Set %d values using set_multi in namespace '%s'" % ( > > > len(mapping), namespace )) > > > > cached_mapping = memcache.get_multi(obj_map.keys(), > > namespace=namespace) > > > println(self, "Retrieved %d values using get_multi in namespace '%s'" > > % > > > ( len(cached_mapping), namespace )) > > > > for key in cached_mapping: > > > println(self, str(obj_map[key])) > > > > def println(handler, string): > > > handler.response.out.write(string + "<br/>") > > > > application = webapp.WSGIApplication([ ('/cachetest', MemcacheTest), > > > ], debug=True) > > > > def main(): > > > run_wsgi_app(application) > > > > if __name__ == "__main__": > > > main() > > > > On Thu, Dec 3, 2009 at 11:09 AM, naan <[email protected]> wrote: > > > > I'm using memcache.set_multi() / add_multi() for caching DataStore > > > > data. Code is something like following: > > > > > users = User.all().query(....).fetch(...) > > > > > mapping = {} > > > > for u in users: > > > > mapping[str(u.key())] = u > > > > > memcache.set_multi(mapping, namespace='synccache') > > > > > It seems that set_multi/add_multi returns correct value and there's no > > > > error logs, error messages, nor exception. However, I can't retrieve > > > > them with memcache.get_multi(). > > > > > Thanks, > > > > Kazuho > > > > > On Dec 3, 10:46 am, "Ikai L (Google)" <[email protected]> wrote: > > > > > We've just tried this and it seems to work for us. What are you > > storing > > > > in > > > > > Memcache? How are you generating the keys? > > > > > > On Thu, Dec 3, 2009 at 10:31 AM, Ikai L (Google) <[email protected]> > > > > wrote: > > > > > > > Is there an error message when you try to store data? Do you see any > > > > > > information in the logs? > > > > > > > On Wed, Dec 2, 2009 at 5:49 PM, naan <[email protected]> wrote: > > > > > > >> My app id is echofonsync and the issue remains. I can get a result > > > > > >> from memcache.get_stats() now, but can not store new data. Thanks. > > > > > > >> On Dec 2, 5:05 pm, "Ikai L (Google)" <[email protected]> wrote: > > > > > >> > Is this issue still occurring for you? If so, let me know your > > > > > >> application > > > > > >> > ID. > > > > > > >> > Michael, let me know if this is causing you problems in > > production. > > > > I > > > > > >> can > > > > > >> > attempt to migrate your application to a different pool, but this > > is > > > > not > > > > > >> > something I'd advise if you're running in production and things > > are > > > > > >> working > > > > > >> > fine. > > > > > > >> > On Wed, Dec 2, 2009 at 4:26 PM, naan <[email protected]> wrote: > > > > > >> > > Hi, > > > > > > >> > > This happens for me too. I can't store any data into memcache. > > > > Also, I > > > > > >> > > can't get any results with memcache.get_stats(). The function > > > > returns > > > > > >> > > None. > > > > > > >> > > On Dec 2, 1:45 pm, Michael <[email protected]> wrote: > > > > > >> > > > The problem with consistency is: the same key maps to the > > > > different > > > > > >> > > > values (when I'm hitting another memcached server), so > > randomly > > > > I'm > > > > > >> > > > getting old value from memcache and it produces weird effects > > on > > > > my > > > > > >> > > > application (like new message appearing, though they were > > marked > > > > as > > > > > >> > > > read). > > > > > > >> > > > Is it going to be fixed soon? > > > > > > >> > > > On Dec 2, 11:33 pm, Michael <[email protected]> wrote: > > > > > > >> > > > > At the time of the first post it was going for about 10 > > > > minutes. > > > > > > >> > > > > It's still happening. > > > > > > >> > > -- > > > > > > >> > > You received this message because you are subscribed to the > > Google > > > > > >> Groups > > > > > >> > > "Google App Engine" group. > > > > > >> > > To post to this group, send email to > > > > > >> [email protected]. > > > > > >> > > To unsubscribe from this group, send email to > > > > > >> > > [email protected]<google-appengine%[email protected]> > > <google-appengine%[email protected]<google-appengine%[email protected]> > > > > > <google-appengine%[email protected]<google-appengine%[email protected]> > > <google-appengine%[email protected]<google-appengine%[email protected]> > > > > > > >> <google-appengine%[email protected]<google-appengine%[email protected]> > > <google-appengine%[email protected]<google-appengine%[email protected]> > > > > > <google-appengine%[email protected]<google-appengine%[email protected]> > > <google-appengine%[email protected]<google-appengine%[email protected]> > > > > > > >> > > . > > > > > >> > > For more options, visit this group at > > > > > >> > >http://groups.google.com/group/google-appengine?hl=en. > > > > > > >> > -- > > > > > >> > Ikai Lan > > > > > >> > Developer Programs Engineer, Google App Engine > > > > > > >> -- > > > > > > >> You received this message because you are subscribed to the Google > > > > Groups > > > > > >> "Google App Engine" group. > > > > > >> To post to this group, send email to > > > > [email protected]. > > > > > >> To unsubscribe from this group, send email to > > > > > >> [email protected]<google-appengine%[email protected]> > > <google-appengine%[email protected]<google-appengine%[email protected]> > > > > > <google-appengine%[email protected]<google-appengine%[email protected]> > > <google-appengine%[email protected]<google-appengine%[email protected]> > > > > > > >> . > > > > > >> For more options, visit this group at > > > > > >>http://groups.google.com/group/google-appengine?hl=en. > > > > > > > -- > > > > > > Ikai Lan > > > > > > Developer > > ... > > read more » -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
