[
https://issues.apache.org/jira/browse/SPARK-14367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom Hubregtsen updated SPARK-14367:
-----------------------------------
Summary: spark.memory.useLegacyMode=true in 1.6 does not yield the same
memory behavior as Spark 1.3 (was: spark.memory.useLegacyMode=true in 1.6 does
not yield the same memory behavior as in 1.3)
> spark.memory.useLegacyMode=true in 1.6 does not yield the same memory
> behavior as Spark 1.3
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-14367
> URL: https://issues.apache.org/jira/browse/SPARK-14367
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.6.0
> Environment: Ubuntu 15.10 with ibm-java-ppc64le-80.
> Reporter: Tom Hubregtsen
> Priority: Minor
> Labels: backwards-compatibility
>
> Hi,
> I am trying to get the same memory behavior in Spark 1.6 as I had in Spark
> 1.3 with default settings.
> I set
> --driver-java-options "--Dspark.memory.useLegacyMode=true
> -Dspark.shuffle.memoryFraction=0.2 -Dspark.storage.memoryFraction=0.6
> -Dspark.storage.unrollFraction=0.2"
> in Spark 1.6.
> But the numbers don't add up. For instance:
> --driver-java-options "-Dspark.shuffle.memoryFraction=0.1
> -Dspark.storage.memoryFraction=0.1"
> in Spark 1.3.1 leads to:
> 16/03/29 14:47:36 INFO MemoryStore: MemoryStore started with capacity 46.1 MB
> The same in Spark 1.6.0 with -Dspark.memory.useLegacyMode=true
> -Dspark.shuffle.memoryFraction=0.1 -Dspark.storage.memoryFraction=0.1.
> 16/03/29 14:50:55 INFO MemoryStore: MemoryStore started with capacity 92.2 MB
> If I then increase both fractions to 0.2, the numbers of the MemoryStore both
> double (as one would expect), but that means there is still a 2x difference
> in allocated memory between Spark 1.3 and Spark 1.6. So my question:
> I believe a parameter that reads
> spark.memory.useLegacyMode=true
> should yield the *exact* memory behavior as in the Legacy version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
