[
https://issues.apache.org/jira/browse/SPARK-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040294#comment-16040294
]
Maarten Kesselaers commented on SPARK-20691:
--------------------------------------------
OK, some components use the function {noformat} byteStringAsMb {noformat} from
JavaUtils which return MebiBytes. (This is also called bij Utils.scala in the
function byteStringAsMb)
In the other hand in Utils.scala, we have Utils.bytesToString which returns
MegaBytes.
Does it make sense to have both and mix them?
This causes a lot of confusion, in my opinion.
> Difference between Storage Memory as seen internally and in web UI
> ------------------------------------------------------------------
>
> Key: SPARK-20691
> URL: https://issues.apache.org/jira/browse/SPARK-20691
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.3.0
> Reporter: Jacek Laskowski
> Labels: starter
>
> I set Major priority as it's visible to a user.
> There's a difference in what the size of Storage Memory is managed internally
> and displayed to a user in web UI.
> I found it while answering [How does web UI calculate Storage Memory (in
> Executors tab)?|http://stackoverflow.com/q/43801062/1305344] on StackOverflow.
> In short (quoting the main parts), when you start a Spark app (say
> spark-shell) you see 912.3 MB RAM for Storage Memory:
> {code}
> $ ./bin/spark-shell --conf spark.driver.memory=2g
> ...
> 17/05/07 15:20:50 INFO BlockManagerMasterEndpoint: Registering block manager
> 192.168.1.8:57177 with 912.3 MB RAM, BlockManagerId(driver, 192.168.1.8,
> 57177, None)
> {code}
> but in the web UI you'll see 956.6 MB due to the way the custom JavaScript
> function {{formatBytes}} in
> [utils.js|https://github.com/apache/spark/blob/master/core/src/main/resources/org/apache/spark/ui/static/utils.js#L40-L48]
> calculates the value. That translates to the following Scala code:
> {code}
> def formatBytes(bytes: Double) = {
> val k = 1000
> val i = math.floor(math.log(bytes) / math.log(k))
> val maxMemoryWebUI = bytes / math.pow(k, i)
> f"$maxMemoryWebUI%1.1f"
> }
> scala> println(formatBytes(maxMemory))
> 956.6
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
