This is dragging a bit but its an interesting debate: By using 30 poles - what happens is as use pole in a nano second the pole n will be read and set with a 29 second removal span. So when the last nano second polling user hits that pole it sets it to remove in 30 seconds - so if ten users pole in that second they all simply re-set the removal for 30 seconds - and all we cause is a nano second gloat. If however you use two poles as you suggest if someone poles at 29 seconds the pole is set and it wont get removed for another 29 seconds - which basically means the first person to pole at 1 second will be kept for what is that 58 seconds? if you then think OK ill calculate a removal time based on how many seconds are left - ie if someone poles at 15 seconds when you set mcache you set 15 seconds remaning to remove not 30 then that persons pole is kept for 15 not 30 as i said in my last post. All of this sounds highly complex. As i said 30 poles solves all these worries and is simple to code and execute or am i missing the "rabit in the hat trick"?
Interesting debate :-) This still will not work at if you use two sets they both expire within 30 secs so if i poll at 29 secs - in 1 sec the my poll is removed it should be kept for 29 secods - if i poll on say 45 seconds my poll is lost in 15. The idea is to show a clear window of "who" polled in the last the seconds. Or that's how i understand the requirement So if a poll come at 03:12:09(XX:YY:09) AM it will go to cache MY_USER_CACHE_FIRST - #this person must exist for 30secs not 21# and if request comes at 03:12:35(XX:YY:35) AM it will go to cache MY_USER_CACHE_LAST. #this person must exist for 30 secs not 25# the only way to keep a track to the nearest second is with 30 poles 15 poles gives you a two second error 10 poles gives you a three second error 5 poles gives you a 6 second error 2 poles gives you a 15 second error 30 pole solution is the only way to do it if you want to use mcache in this way. Further on the note of reading the users. I don't know what the original developers was planing on doing with the data - he mentioned he wanted a task to do "something" every few seconds and their maybe a lot of users. On that note assuming he is NOT updating the users model - assuming maybe its an admin page with an Ajax update to show who is polling then each time a user poles(which i know would not be a task) he should store a dictionary with the data from the user he needs in the pole - when they poll - not load it every n second in his task. Its better to read it once - but again this is all guess work. Finally if he is updating the users table it would be just as easy to update a polled time and run a task that draws which ones polled in the time frame and do the updateamd as you rightly say perhaps do 100 at a time and make the task re-run when their are poles left. The solution i posed was merely the approach that would work in mcache and was a simple one. I hope this is now agreeable? ________________________________ From: Ravi Sharma <[email protected]> To: [email protected] Sent: Thu, 24 June, 2010 19:40:50 Subject: Re: [google-appengine] Re:What is a pattern for keeping track of current users in google app engine? Yes i understand about that there might be lost in few entries. That you can fix in this solution by many ways..one of the way by keeping two sets then.( i thought of explaning it before but mail was getting longer :) ) Ok let me explain in detail But this to work the both cache need to have seprate Id like and expire time say 30 second or more. Key Value description MY_USER_CACHE_FIRST Set<Key> users for poll request between 00 to 30 seconds MY_USER_CACHE_LAST Set<Key> users for poll request between 31 to 60 seconds and each poll handler will see if i am between second 00 to 30 or 31 to 60 So if a poll come at 03:12:09(XX:YY:09) AM it will go to cache MY_USER_CACHE_FIRST and if request comes at 03:12:35(XX:YY:35) AM it will go to cache MY_USER_CACHE_LAST. And run your task at every 11:29:35((XX:YY:35) and 11:30:05((XX:YY:05), so 11:29:35(XX:YY:35) task will receive all users polled between 11:29:00(XX:YY:00) to 11:29:30(XX:YY:30). and 11:29:35(XX:YY:05) task will receive all users polled between 11:29:00(XX:(YY-1):00) to 11:29:30(XX:(YY-1):30). at the end of hour even XX will be XX-1 11:29:35 task will always read thye cache MY_USER_CACHE_FIRST and work on it as i explained before, in case u receive millions of records and divide into small task put seprately into small memcahe object etc and 11:30:05 task will always work on MY_USER_CACHE_LAST. I kept 5 second different between last entry going into cache and starting the task you can choose this difference as you want, may be just one second. Ravi. On Thu, Jun 24, 2010 at 7:04 PM, Martin Webb <[email protected]> wrote: @ping2ravi >this wont do what was required as if every-time you set mcache you will renew >the 30 second lifespan. further if mcache does clear on the 30 sec count a >poll 1 sec ago would be lost - that was the whole point of using 30 1 sec >poles as they all clear on deadline and don't get repeatedly refreshed? > > > >Regards > > >Martin Webb > > > >The information contained in this email is confidential and may contain >proprietary information. It is > meant solely for the intended recipient. Access to this email by anyone else > is unauthorised. If you are not the intended recipient, any disclosure, > copying, distribution or any action taken or omitted in reliance on this, is > prohibited and may be unlawful. No liability or responsibility is accepted if > information or data is, for whatever reason corrupted or does not reach its > intended recipient. No warranty is given that this email is free of viruses. > The views expressed in this email are, unless otherwise stated, those of the > author > > > > > > > ________________________________ From: > ping2ravi <[email protected]> >To: Google App Engine <[email protected]> >Sent: Thu, 24 June, 2010 13:41:50 >Subject: [google-appengine] Re: What is a pattern for keeping track of current > users in google app engine? > > >Hi, >If reading 1000+ user records(say keys) in ONE task execution is >problem then it doesnt matter if you use Memcache or data store, your >solution is not gonna be scalable to update 1000+ user record(for last >>poll time) in ONE task execution. > >What i understood your problem is that you want to keep every user's >last poll time updated against there user data. And as a solution you >are running kind of Timer thread in the form of Task Queus and update >>all user's last poll time who accessed/polled in last 30/n seconds. >I am > specifying a little different approach depending on my >understanding about your problem > >Using memcache Solution 1) > >>You dont need to create 30 poll cache or n poll cache. Just create one >>Set<key> object and put it in memcache and your polling request handler will >>keep adding user's id/key in this cache.Using Set/HashSet will make sure that >>you dont see to worry about merging or duplicacy. > >> After 30/n seconds your task start executing it will take all the >> data(Set<Key> from cache and then remove everything from set in memcache, so >> that your polling request handler will find it empty and then they will >> start adding adding new user id/key in it. > >> Now it may be possible that you got a set of 10000 or more from cache and >> task execution may not be able to update all in 30 seconds, so you may want >> to create another sub task who will take left over user key and do the same >> thing again. Basically you will put > the left over ids in memcache again with different id and pass that memcach > id to sub-task and that task can read it from there. Or to make it more > efficient, if you already know by experience that you are able to update on > 500 records in one execution then divide your original left over user ids and > divide it into set of 500 and create as many task as you want and save set of > 500 in memcache with different ids like LEFT_OVER_SET_1, LEFT_OVER_SET_2 etc. > and each task can read daat from memcache using these provided ids and if > something left over it will try to invoke sub task again(divide by 500, but > this time you will find total number is already less then 500). > >But still i say memcach edoesnt guarntee that over a span of time data >will still be avaiable so instead of memcache you can use datastore in >above solution. > >Ravi. > > > > >-- >You received this message because you are subscribed to the Google Groups >"Google App > Engine" group. >To post to this group, send email to [email protected]. >To unsubscribe from this group, send email to >[email protected]. >>For more options, visit this group at >>http://groups.google.com/group/google-appengine?hl=en. > > > >> > >-- >>You received this message because you are subscribed to the Google Groups >>"Google App Engine" group. >>To post to this group, send email to [email protected]. >>To unsubscribe from this group, send email to >>[email protected]. >> >For more options, visit this group at >http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
