Since you mentioned ofy, here are some useful tips which helped us to solve
similar issues:
1) if you are making lot's of queries and don't need to cache, best is to
disable cache for query, with :
ofy().cache(false).
2) use iterator to process big data sets in chunks.
Here is sample code, which works very robustly for us. It will process
chunks for 8 minutes and will pass the rest to be run on next task. It uses
deferred tasks.
chunks defined 'limit' variable, entity type by 'modelCalss', cursor by
'startCursor' which is null to start with. There are some other variables
you can figure out yourself.
Hope this helps.
try {
do {
Query<T> query = ofy().cache(false).load()
.type(this.modelClass)
.limit(this.limit)
.chunkAll();
if (this.startCursor != null) {
query = query.startAt(this.startCursor);
}
QueryResultIterator<T> iterator = null;
try {
iterator = query.iterator();
if (!iterator.hasNext()) {
stillNotFinished = false;
}
while (iterator.hasNext()) {
T entity = iterator.next();
process(entity);
this.countSoFar++;
}
} catch (Exception dxe) {
log.error("Exception error should stop from {}",
this.countSoFar, dxe);
isException = true;
} finally {
ofy().flush();
if (iterator != null) {
this.startCursor = iterator.getCursor();
ofy().clear();
}
}
log.info("Processed so far {} items. ", this.countSoFar);
} while (stillNotFinished && sw.elapsed(TimeUnit.MINUTES) < 8);
} finally {
sw.stop();
ModelProcessor.log.info("Finished reading {} items in total for
{}", this.countSoFar, sw.toString());
if (stillNotFinished) {
if (isException || !this.spawnNewProcessOnDisruption) {
log.warn("Has Stopped on item {}", this.countSoFar);
} else {
log.info("Rerunning from {}", this.countSoFar);
TaskController.addToTaskQueue(this);
}
}
}
On Wednesday, June 24, 2015 at 9:54:19 PM UTC+1, [email protected] wrote:
>
> Thank you Chad Vincent for your answer.
>
> I added some logging, but it didn't gave me more relevant information.
> Everything is running fine until the "unexplainable" failure. Therefore, I
> guess that it is related to a memory issue. It retrieves data and caches
> until the memory is full and then crashes.
>
> I spent some time tonight reading the Objectify documentation. I came
> across this:
> https://code.google.com/p/objectify-appengine/wiki/BasicOperations#The_Session_Cache
>
> . It could really be related to my issue. Unfortunately calling
> ofy().clear() sometimes did not solve anything. Moreover, I read that
> assigning ObjectifyService.ofy() was a bad idea. But requesting ofy()
> directly each time I needed it did not help neither.
>
> Any other idea or suggestion is welcome. Thank you :) .
> G.
>
> Le mercredi 24 juin 2015 22:30:38 UTC+2, Chad Vincent a écrit :
>>
>> Then I would definitely add more logging so you can see what is and is
>> not getting done before the instance bails.
>>
>> On Wednesday, June 24, 2015 at 1:48:07 AM UTC-5, [email protected]
>> wrote:
>>>
>>> Thank you for answering me.
>>>
>>> Here is the only log I'm getting :
>>> /task/update-known-devices 500 151600ms 0kb AppEngine-Google; (+
>>> http://code.google.com/appengine) module=default version=dev
>>>
>>> 1.
>>>
>>> 0.1.0.2 - - [23/Jun/2015:23:23:07 -0700] "POST
>>> /task/update-known-devices HTTP/1.1" 500 0
>>> "http://freemobile-netstat.appspot.com/cron/update-known-devices";
>>> "AppEngine-Google; (+http://code.google.com/appengine)"
>>> "freemobile-netstat.appspot.com" ms=151600 cpu_ms=3530
>>> queue_name=update-queue task_name=81805458203132447311 exit_code=202
>>> app_engine_release=1.9.22 trace_id=b86eb2915714ad73aae5f54bfbef746d
>>> instance=00c61b117c5b3b509e154cd4930e3af3083a8a68
>>> <https://appengine.google.com/instances?app_id=e~freemobile-netstat&version_id=dev.385229375500191200&key=00c61b117c5b3b509e154cd4930e3af3083a8a68#00c61b117c5b3b509e154cd4930e3af3083a8a68>
>>>
>>> 2. E2015-06-24 08:23:07.492
>>>
>>> A problem was encountered with the process that handled this request,
>>> causing it to exit. This is likely to cause a new process to be used for
>>> the next request to your application. (Error code 202)
>>>
>>>
>>> This log appears for each retry (around 3 times per task).
>>>
>>> G.
>>> Le mercredi 24 juin 2015 01:00:49 UTC+2, Chad Vincent a écrit :
>>>>
>>>> What are you getting in your logs? Any uncaught exceptions?
>>>>
>>>> You might want to add some debug logging statements and lower the
>>>> logging level for a while to see what is happening. You do get 10 minutes
>>>> for queries from the TaskQueue, so it isn't a timeout issue.
>>>>
>>>> On Tuesday, June 23, 2015 at 1:31:59 PM UTC-5, [email protected]
>>>> wrote:
>>>>>
>>>>> Hi everybody,
>>>>> I have a long task to run under my App Engine application with a lot
>>>>> of computing. But after a while running the task (~2 minutes), it stops
>>>>> and
>>>>> just retries. Obviously, my task restarts from the beginning and cannot
>>>>> end.
>>>>> I would like to configure App Engine to run the task until it is
>>>>> finished. I've heard about a 10 minutes limit for the tasks. I guess it
>>>>> should be enough if it didn't retries everytime.
>>>>>
>>>>> My source code is (hopefully) freely available here :
>>>>> https://github.com/gilbsgilbs/freemobilenetstat-gae . The two tasks
>>>>> under *src/main/java/org/pixmob/freemobile/netstat/gae/web/task* are
>>>>> problematic (since it computes a lot of data). They are pushed the the
>>>>> queue from a cron under
>>>>> *src/main/java/org/pixmob/freemobile/netstat/gae/web/cron/CronServlet.java*
>>>>> .
>>>>>
>>>>> Note that I'm really new to App Engine and that my code might not be
>>>>> optimal. I'm not sure of the exact issue and maybe it is just a memory
>>>>> overflow (which I wouldn't known how to fix).
>>>>>
>>>>> Thank you !
>>>>> G.
>>>>>
>>>>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/0a3fa7dd-310d-409d-927c-f9744687e1f0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
