yes. i decided to proceed with using mapreduce. i plan to randomly select in advance a more manageable number of just around 100 to 200 questions from the table using mapreduce. everytime i would be needing a random set of questions i'll just get from this store rather than the main questions table. i expect that the 100 to 200 questions would last me for around 3-5 days so i expect to scan the whole questions table not very often.
thanks very much for your comments. it was really very helpful. On Nov 30, 4:20 am, "Ikai Lan (Google)" <[email protected]> wrote: > Thanks for the feedback Stephen. Yeah, I can see where having an option to > not return the Entity could be useful. > > Yep, Stephen's way is the way to go (we talked about this at lunch the other > day). Create a new set of Entities with random keys that point to the > original entities, then randomly select a few of these based on key. > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > Blogger:http://googleappengine.blogspot.com > Reddit:http://www.reddit.com/r/appengine > Twitter:http://twitter.com/app_engine > > On Sat, Nov 27, 2010 at 10:27 AM, Stephen Johnson > <[email protected]>wrote: > > > To extend on Tim's comments. I find that map reduce can be your best friend > > in a lot of situations. Why not use map reduce to pre-generate random test > > lists of keys. Then you're only going through all the keys every so often > > and not every time you need a new test. Create a new Entity kind to hold > > these pre-generated lists and then when you need a new test grab the first > > one from the pre-generated list of keys and then delete it (or just mark it > > as used if you want a historical record of the test). Have your map reduce > > job create as many as you think you'll need over a certain period and then > > just have a task to that runs the map reduce job every so often. > > > As a side note to any of the Googlers that maintain and update map-reduce, > > this is another example of where it would be nice to have an option of > > map-reduce only returning keys and not also the entire entity because as in > > this example there is no need to incur the CPU cost of returning the entire > > entity. This was discussed also in the Bulk Deletion Woe topic a couple of > > weeks ago. > > > On Fri, Nov 26, 2010 at 10:44 PM, Tim Hoffman <[email protected]> wrote: > > >> Hi > > >> Do you know your key structure and general distribution? > > >> If for instance you keys are ints, you could randomly choose a key > >> do a > query and grab the first past that value or a random number > >> past the first one, > >> that way you only need to return a few keys/entities. > > >> To make it more likely to get hits, your could have a task queue that > >> maps the > >> keys ranges, then your random choice can come from a set of values > >> more likely to give you an entity. > > >> Just a thought ;-) > > >> T > > >> On Nov 26, 11:35 pm, lein <[email protected]> wrote: > >> > hello, > > >> > i have a table of questions that has many rows. every now and then i > >> > would have to generate a quiz by randomly selecting questions from > >> > this table. making a key only query and randomly selecting the > >> > question keys from the resulting list of keys, takes a lot of CPU > >> > resources. any suggestions on how i could better do this? > > >> > thanks > > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Google App Engine" group. > >> To post to this group, send email to [email protected]. > >> To unsubscribe from this group, send email to > >> [email protected]<google-appengine%[email protected]> > >> . > >> For more options, visit this group at > >>http://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine%[email protected]> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
