GitHub user zhangwei72 reopened a pull request:
https://github.com/apache/spark/pull/17548
Branch 1.6
## What changes were proposed in this pull request?
Find spark1.6 version of the memory management of the Bug,
spark.memory.storageFraction and the official website of the problem.
## How was this patch tested?
I added some log information to the UnifiedMemoryManager file as follows:
logInfo("storageMemoryPool.memoryFree
%f".format(storageMemoryPool.memoryFree/1024.0/1024.0))
logInfo("onHeapExecutionMemoryPool.memoryFree
%f".format(onHeapExecutionMemoryPool.memoryFree/1024.0/1024.0))
logInfo("storageMemoryPool.memoryUsed %f".format(
storageMemoryPool.memoryUsed/1024.0/1024.0))
logInfo("onHeapExecutionMemoryPool.memoryUsed
%f".format(onHeapExecutionMemoryPool.memoryUsed/1024.0/1024.0))
logInfo("storageMemoryPool.poolSize %f".format(
storageMemoryPool.poolSize/1024.0/1024.0))
logInfo("onHeapExecutionMemoryPool.poolSize
%f".format(onHeapExecutionMemoryPool.poolSize/1024.0/1024.0))
When I run the PageRank program, the input file for PageRank is generated
by the BigDataBench-Chinese Academy of Sciences and is used to evaluate large
data analysis system tools with a size of 676M. The information submitted is as
follows:
./bin/spark-submit --class org.apache.spark.examples.SparkPageRank \
--master yarn \
--deploy-mode cluster \
--num-executors 1 \
--driver-memory 4g \
--executor-memory 7g \
--executor-cores 6 \
--queue thequeue \
./examples/target/scala-2.10/spark-examples-1.6.2-hadoop2.2.0.jar \
/test/Google_genGraph_23.txt 6
The configuration is as followsï¼
spark.memory.useLegacyMode=false
spark.memory.fraction=0.75
spark.memory.storageFraction=0.2
Log information is as followsï¼
17/02/28 11:07:34 INFO memory.UnifiedMemoryManager:
storageMemoryPool.memoryFree 0.000000
17/02/28 11:07:34 INFO memory.UnifiedMemoryManager:
onHeapExecutionMemoryPool.memoryFree 5663.325877
17/02/28 11:07:34 INFO memory.UnifiedMemoryManager:
storageMemoryPool.memoryUsed 0.299123
17/02/28 11:07:34 INFO memory.UnifiedMemoryManager:
onHeapExecutionMemoryPool.memoryUsed 0.000000
17/02/28 11:07:34 INFO memory.UnifiedMemoryManager:
storageMemoryPool.poolSize 0.299123
17/02/28 11:07:34 INFO memory.UnifiedMemoryManager:
onHeapExecutionMemoryPool.poolSize 5663.325877
According to the configuration, storageMemoryPool.poolSize at least 1G or
more, but the log information is only 0.299123M, so there is an error.
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/spark branch-1.6
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17548.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17548
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
