The Mapper API is actually quite good at inserting lots of data quickly, at least in my tests with data coming in from the Blobstore. Since creating a new index row is not dependent on the total size of data, it wouldn't really make index building slower. Once the process starts, however, creating a few thousand indexes really doesn't take that long.
On Fri, Sep 10, 2010 at 3:03 PM, Robert Kluin <[email protected]>wrote: > There are constantly posts about how slow it is to load data. > Assuming 'tons' means thousands and thousands of entities, then from > watching the groups it seems like it would be very hard to insert a > ton of data after index building begins. Unless your planning to make > index building slower. > > > > > > > On Fri, Sep 10, 2010 at 16:58, Ikai L (Google) <[email protected]> wrote: > > Index building isn't done in what we call "user space", but that's > certainly > > one possibility. I think we're being a bit more intelligent about how we > > schedule work, however, and it might be a bit more involved of a solution > > than just creating multiple queues - for instance, you could effective > > "game" the system by inserting a ton of data after the index building > begins > > (and we don't want to lock your datastore). We recognize that this is a > > major pain point for developers and there's some work going on in this > area > > to make index building more fair all around. > > > > On Fri, Sep 10, 2010 at 1:25 PM, Jeff Schwartz <[email protected]> > > wrote: > >> > >> I think the problem with 2 tasks queues is that they could still be > >> hitting the same nodes on the network. Maybe just a better time slicing > >> algorithm is what's needed. > >> > >> On Fri, Sep 10, 2010 at 3:36 PM, Nate Bauernfeind > >> <[email protected]> wrote: > >>> > >>> Couldn't you create two task queue's? That should fix the starvation > >>> problem. I suppose the trick would be to determine which queue to stick > it > >>> on. > >>> > >>> On Fri, Sep 10, 2010 at 1:15 PM, Ikai L (Google) <[email protected]> > >>> wrote: > >>>> > >>>> I think one thing we can do better is to schedule some index builds > for > >>>> apps with no data ahead of apps with a lot of data, since they are > >>>> essentially no-ops. Of course, one wonders if this means large index > builds > >>>> will be delayed forever? *rubs hands together for Computer Science > fun* > >>>> > >>>> On Thu, Sep 9, 2010 at 7:11 PM, Jeff Schwartz < > [email protected]> > >>>> wrote: > >>>>> > >>>>> Soon after I posted the issue it was fixed. I don't know how or by > whom > >>>>> but somehow it got resolved. I waited a few hours for the index to > build > >>>>> even though there was no data to which the index was referring to. > Thank > >>>>> you, Ikai. > >>>>> > >>>>> On Thu, Sep 9, 2010 at 9:39 PM, Ikai L (Google) <[email protected]> > >>>>> wrote: > >>>>>> > >>>>>> The indexes all look to be serving to me. > >>>>>> > >>>>>> On Thu, Sep 9, 2010 at 2:07 PM, Jeff Schwartz > >>>>>> <[email protected]> wrote: > >>>>>>> > >>>>>>> I have 1 index that was building for quite a while though there was > >>>>>>> no data associated with the index. When I clicked on the Datastore > Indexes > >>>>>>> link in the admin console I am directed to a error page that says: > >>>>>>> > >>>>>>> Server Error (500) > >>>>>>> > >>>>>>> A server error has occurred. > >>>>>>> > >>>>>>> Return to Applications screen ยป > >>>>>>> > >>>>>>> My application's id is i-emote and its version is 1. > >>>>>>> > >>>>>>> Can anyone fix this for me and as soon as posible? Thanks in > advance. > >>>>>>> > >>>>>>> Jeff > >>>>>>> > >>>>>>> -- > >>>>>>> You received this message because you are subscribed to the Google > >>>>>>> Groups "Google App Engine" group. > >>>>>>> To post to this group, send email to > >>>>>>> [email protected]. > >>>>>>> To unsubscribe from this group, send email to > >>>>>>> [email protected]<google-appengine%[email protected]> > . > >>>>>>> For more options, visit this group at > >>>>>>> http://groups.google.com/group/google-appengine?hl=en. > >>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Ikai Lan > >>>>>> Developer Programs Engineer, Google App Engine > >>>>>> Blog: http://googleappengine.blogspot.com > >>>>>> Twitter: http://twitter.com/app_engine > >>>>>> Reddit: http://www.reddit.com/r/appengine > >>>>>> > >>>>>> -- > >>>>>> You received this message because you are subscribed to the Google > >>>>>> Groups "Google App Engine" group. > >>>>>> To post to this group, send email to > >>>>>> [email protected]. > >>>>>> To unsubscribe from this group, send email to > >>>>>> [email protected]<google-appengine%[email protected]> > . > >>>>>> For more options, visit this group at > >>>>>> http://groups.google.com/group/google-appengine?hl=en. > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> -- > >>>>> Jeff > >>>>> > >>>>> -- > >>>>> You received this message because you are subscribed to the Google > >>>>> Groups "Google App Engine" group. > >>>>> To post to this group, send email to > [email protected]. > >>>>> To unsubscribe from this group, send email to > >>>>> [email protected]<google-appengine%[email protected]> > . > >>>>> For more options, visit this group at > >>>>> http://groups.google.com/group/google-appengine?hl=en. > >>>> > >>>> > >>>> > >>>> -- > >>>> Ikai Lan > >>>> Developer Programs Engineer, Google App Engine > >>>> Blog: http://googleappengine.blogspot.com > >>>> Twitter: http://twitter.com/app_engine > >>>> Reddit: http://www.reddit.com/r/appengine > >>>> > >>>> -- > >>>> You received this message because you are subscribed to the Google > >>>> Groups "Google App Engine" group. > >>>> To post to this group, send email to > [email protected]. > >>>> To unsubscribe from this group, send email to > >>>> [email protected]<google-appengine%[email protected]> > . > >>>> For more options, visit this group at > >>>> http://groups.google.com/group/google-appengine?hl=en. > >>> > >>> -- > >>> You received this message because you are subscribed to the Google > Groups > >>> "Google App Engine" group. > >>> To post to this group, send email to [email protected] > . > >>> To unsubscribe from this group, send email to > >>> [email protected]<google-appengine%[email protected]> > . > >>> For more options, visit this group at > >>> http://groups.google.com/group/google-appengine?hl=en. > >> > >> > >> > >> -- > >> -- > >> Jeff > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Google App Engine" group. > >> To post to this group, send email to [email protected]. > >> To unsubscribe from this group, send email to > >> [email protected]<google-appengine%[email protected]> > . > >> For more options, visit this group at > >> http://groups.google.com/group/google-appengine?hl=en. > > > > > > > > -- > > Ikai Lan > > Developer Programs Engineer, Google App Engine > > Blog: http://googleappengine.blogspot.com > > Twitter: http://twitter.com/app_engine > > Reddit: http://www.reddit.com/r/appengine > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine%[email protected]> > . > > For more options, visit this group at > > http://groups.google.com/group/google-appengine?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
