The page size calculation is the issue here as there is plenty of free
memory, although there is maybe a fair bit of wasted space in some pages.
It is that when we have a lot of tasks each is only allowed to reach 1/n of
the available memory and several of the tasks bump in to that limit. With
tasks 4 times the number of cores there will be some contention and so they
remain active for longer.
So I think this is a test case issue configuring the number of executors
too high.
On 15 September 2015 at 18:54, Reynold Xin <r...@databricks.com> wrote:
> Maybe we can change the heuristics in memory calculation to use
> SparkContext.defaultParallelism if it is local mode.
>
>
> On Tue, Sep 15, 2015 at 10:28 AM, Pete Robbins <robbin...@gmail.com>
> wrote:
>
>> Yes and at least there is an override by setting spark.sql.test.master
>> to local[8] , in fact local[16] worked on my 8 core box.
>>
>> I'm happy to use this as a workaround but the 32 hard-coded will fail
>> running build/tests on a clean checkout if you only have 8 cores.
>>
>> On 15 September 2015 at 17:40, Marcelo Vanzin <van...@cloudera.com>
>> wrote:
>>
>>> That test explicitly sets the number of executor cores to 32.
>>>
>>> object TestHive
>>> extends TestHiveContext(
>>> new SparkContext(
>>> System.getProperty("spark.sql.test.master", "local[32]"),
>>>
>>>
>>> On Mon, Sep 14, 2015 at 11:22 PM, Reynold Xin <r...@databricks.com>
>>> wrote:
>>> > Yea I think this is where the heuristics is failing -- it uses 8 cores
>>> to
>>> > approximate the number of active tasks, but the tests somehow is using
>>> 32
>>> > (maybe because it explicitly sets it to that, or you set it yourself?
>>> I'm
>>> > not sure which one)
>>> >
>>> > On Mon, Sep 14, 2015 at 11:06 PM, Pete Robbins <robbin...@gmail.com>
>>> wrote:
>>> >>
>>> >> Reynold, thanks for replying.
>>> >>
>>> >> getPageSize parameters: maxMemory=515396075, numCores=0
>>> >> Calculated values: cores=8, default=4194304
>>> >>
>>> >> So am I getting a large page size as I only have 8 cores?
>>> >>
>>> >> On 15 September 2015 at 00:40, Reynold Xin <r...@databricks.com>
>>> wrote:
>>> >>>
>>> >>> Pete - can you do me a favor?
>>> >>>
>>> >>>
>>> >>>
>>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala#L174
>>> >>>
>>> >>> Print the parameters that are passed into the getPageSize function,
>>> and
>>> >>> check their values.
>>> >>>
>>> >>> On Mon, Sep 14, 2015 at 4:32 PM, Reynold Xin <r...@databricks.com>
>>> wrote:
>>> >>>>
>>> >>>> Is this on latest master / branch-1.5?
>>> >>>>
>>> >>>> out of the box we reserve only 16% (0.2 * 0.8) of the memory for
>>> >>>> execution (e.g. aggregate, join) / shuffle sorting. With a 3GB
>>> heap, that's
>>> >>>> 480MB. So each task gets 480MB / 32 = 15MB, and each operator
>>> reserves at
>>> >>>> least one page for execution. If your page size is 4MB, it only
>>> takes 3
>>> >>>> operators to use up its memory.
>>> >>>>
>>> >>>> The thing is page size is dynamically determined -- and in your
>>> case it
>>> >>>> should be smaller than 4MB.
>>> >>>>
>>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala#L174
>>> >>>>
>>> >>>> Maybe there is a place that in the maven tests that we explicitly
>>> set
>>> >>>> the page size (spark.buffer.pageSize) to 4MB? If yes, we need to
>>> find it and
>>> >>>> just remove it.
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Sep 14, 2015 at 4:16 AM, Pete Robbins <robbin...@gmail.com>
>>> >>>> wrote:
>>> >>>>>
>>> >>>>> I keep hitting errors running the tests on 1.5 such as
>>> >>>>>
>>> >>>>>
>>> >>>>> - join31 *** FAILED ***
>>> >>>>> Failed to execute query using catalyst:
>>> >>>>> Error: Job aborted due to stage failure: Task 9 in stage 3653.0
>>> >>>>> failed 1 times, most recent failure: Lost task 9.0 in stage 3653.0
>>> (TID
>>> >>>>> 123363, localhost): java.io.IOException: Unable to acquire 4194304
>>> bytes of
>>> >>>>> memory
>>> >>>>> at
>>> >>>>>
>>> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.acquireNewPage(UnsafeExternalSorter.java:368)
>>> >>>>>
>>> >>>>>
>>> >>>>> This is using the command
>>> >>>>> build/mvn -Pyarn -Phadoop-2.2 -Phive -Phive-thriftserver test
>>> >>>>>
>>> >>>>>
>>> >>>>> I don't see these errors in any of the amplab jenkins builds. Do
>>> those
>>> >>>>> builds have any configuration/environment that I may be missing?
>>> My build is
>>> >>>>> running with whatever defaults are in the top level pom.xml, eg
>>> -Xmx3G.
>>> >>>>>
>>> >>>>> I can make these tests pass by setting
>>> spark.shuffle.memoryFraction=0.6
>>> >>>>> in the HiveCompatibilitySuite rather than the default 0.2 value.
>>> >>>>>
>>> >>>>> Trying to analyze what is going on with the test it is related to
>>> the
>>> >>>>> number of active tasks, which seems to rise to 32, and so the
>>> >>>>> ShuffleMemoryManager allows less memory per task even though most
>>> of those
>>> >>>>> tasks do not have any memory allocated to them.
>>> >>>>>
>>> >>>>> Has anyone seen issues like this before?
>>> >>>>
>>> >>>>
>>> >>>
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>
>>
>
