Thank you, Robert. I think you nailed it. After a lot of experimentation I
reached the same conclusion. GAE seems to be punishing long running tasks
(my tasks take about 20 seconds each). The longer the tasks run, the fewer
of them you can execute.
This seems odd to me because doing more work per task is more efficient. If
I split my 20 seconds work into 20 tasks (1 seconds each), then I'm
duplicating the overhead 20 times. And the App Engine team just recently
increased the task timeout to 10 minutes, which tells me that they want to
make it possible to do more work per task.
Kann,
Mapreduce is designed to do as much work per task as possible. So, yes,
I believe your tasks are being punished as well and that explains the 6x
increase in time. But it doesn't add up. When I changed my tasks to do less
work, suddenly the app started scaling better and more instances were
created. If the goal is to use less instances, then long running tasks
should be encouraged because they do more work with less instances. Maybe
it's a bug in their scaling algorithm?
On Sun, Jun 19, 2011 at 12:22 PM, Kaan Soral <[email protected]> wrote:
> I have your problem too,
>
> I haven't checked the situation since weeks but right after
> development a mapreduce job, which was doing 800k operations was
> showing results in ~30 minutes, but that time increased to 3 hours ...
>
> I guess this is because they are re-designing appengine in a wrong
> direction
>
> In future we will get charged by instance usage, so you wouldn't want
> 70 instances, and I guess the modifications are in that direction
>
> I don't like this, but I wouldn't want to pay a lot either,
>
> sad ...
>
> On Jun 19, 8:52 am, Waleed Abdulla <[email protected]> wrote:
> > I think I narrowed the issue a little bit. Added a lot of logging.info()
> > statements and I noticed the following:
> >
> > 1. The time between inserting a task and when it starts running is too
> long:
> > 5 - 20 seconds. Typically tasks should execute right away after inserting
> if
> > you're below the queue rate. That's issue #1
> >
> > 2. I put a timer in my code to calculate the time it takes to execute
> each
> > task. When I compare that time with the time app engine reports in the
> logs,
> > there is a big difference. 5 - 20 seconds. For example, processing a task
> > takes 25 seconds, of which only 10 seconds is actually spent inside my
> code.
> > So the infrastructure is keeping requests on hold for a while before they
> > get executed. That's issue #2
> >
> > When I compare the delay in #1 to the time difference calculated in
> #2,
> > they're very close and they correlate (i.e. they go up and down
> together).
> > So that means that issue #1 is probably caused by issue #2. (i.e. tasks
> do
> > get picked for execution right away, but the actual execution doesn't
> start
> > until 5 - 20 seconds later).
> >
> > There seems to be an infrastructure problem in GAE. Maybe GAE is
> > experiencing high load, too much network traffic, ..etc. Whatever the
> > reason, it seems that requests are waiting in a queue for too long before
> > they run.
> >
> > Does anyone notice any issues that confirm or deny the above? Any
> ideas
> > how to work around it?
> >
> > Waleed
> >
> > On Sat, Jun 18, 2011 at 7:45 PM, Chiguireitor
> > <[email protected]>wrote:
> >
> >
> >
> >
> >
> >
> >
> > > The other issue is that each task spawns a new instance... could you be
> > > hitting a max instance limit there? i really don't know if there's an
> > > instance limit... but could happen
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Google App Engine" group.
> > > To view this discussion on the web visit
> > >https://groups.google.com/d/msg/google-appengine/-/nsxW4XGycnwJ.
> >
> > > To post to this group, send email to [email protected]
> .
> > > To unsubscribe from this group, send email to
> > > [email protected].
> > > For more options, visit this group at
> > >http://groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
