hi Erwin Streur,
Thanks for your reply, i have some questions :
For a query select on a data-store like:
authority = Authority.all().filter('twitterUser =',user).get() . (
user attribute is key of Authority)
Although google app engine provide many ways such as using index, zizag
merger,... But when the total records is two large.
So the query takes too long times to complete and lead to DeadlineExceed
error.
In this case how can we over come it.
Every body, does task queue make sense in this case. I think task queue is
suitable for some background works like fetching data from some sites,... But
in my case, when searching on large records, and must show result after the
seaching process finish, how can we use task queue.
I'm considering using Mapreduce frame work.
So please give me some suggestion.
Thanks.
On Thu, Apr 28, 2011 at 6:52 AM, Erwin Streur <[email protected]>wrote:
> To make sure that we are on the right track. You do get the deadline
> ExceededException on the get statement and the work upto that
> statement do not cost so much time (you have verified this by
> profiling/ checking timestamps of log statement)
>
> I primarily develop in Java, but what I understand from Python the
> get() has an implied fetch limit of 1. You are performing a single
> equality filter on an indexed property (GAE requirement) with no
> ordering therefore no index iterating, no zigzag join and no large
> data transfers are done. This should always be fast. Not as fast as
> querying on primary key, but close anyway.
>
> In case there are long running operations, like URL fetch on external
> sites and large data transfer, performing the operation offline by
> means of TaskQueue is good suggestion because then the deadline is not
> 30 sec, but 10 minutes. In case of large volume processing, you might
> consider the MapReduce framework.
>
> But again, your initial post doesn't indicate that to be the case and
> the problem purely to be on the database get.
> Does the problem happen always/often or was it just a fluke and was
> there some temporary overall bad performance on the datastore? Is the
> get part of a transaction? (it shouldn't matter because you are not
> querying by ID, but just to be thorough;)
>
> Regards,
>
> Erwin
>
> On 27 apr, 17:38, Noah McIlraith <[email protected]> wrote:
> > Look into storing the user inside the key (key name), that way you will
> be
> > able to very quickly grab the entity from the datastore (without needing
> to
> > query).
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
