I should add that the pauses are not from GC and also in tracing the CPU
call tree in the JVM it seems like nothing is doing any work, just seems to
be idling or blocking.

On Thu, Dec 3, 2015 at 11:24 AM, Richard Marscher <rmarsc...@localytics.com>
wrote:

> Hi,
>
> I'm doing some testing of workloads using local mode on a server. I see
> weird behavior where a job is submitted to the application and it just
> hangs for several minutes doing nothing. The stages are submitted as
> pending and in the application UI the stage view claims no tasks have been
> submitted. Suddenly after a few minutes things suddenly start and run
> smoothly.
>
> I'm running against tiny data sets the size of 10s to low 100s of items in
> the RDD. I've been attaching with JProfiler, doing thread and heap dumps
> but nothing is really standing out as to why Spark seems to periodically
> pause for such a long time.
>
> Has anyone else seen similar behavior or aware of some quirk of local mode
> that could cause this kind of blocking?
>
> --
> *Richard Marscher*
> Software Engineer
> Localytics
> Localytics.com <http://localytics.com/> | Our Blog
> <http://localytics.com/blog> | Twitter <http://twitter.com/localytics> |
> Facebook <http://facebook.com/localytics> | LinkedIn
> <http://www.linkedin.com/company/1148792?trk=tyah>
>



-- 
*Richard Marscher*
Software Engineer
Localytics
Localytics.com <http://localytics.com/> | Our Blog
<http://localytics.com/blog> | Twitter <http://twitter.com/localytics> |
Facebook <http://facebook.com/localytics> | LinkedIn
<http://www.linkedin.com/company/1148792?trk=tyah>

Reply via email to