[
https://issues.apache.org/jira/browse/SPARK-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004413#comment-16004413
]
Sean Owen commented on SPARK-20691:
-----------------------------------
I think that we have, unfortunately, not consistently differentiated between
megabytes (MB, 10^6 = 1000000 bytes) and mebibytes (MiB, 2^20 = 1048576 bytes).
The javascript function is actually correctly computing MB, and the rest of
Spark is not in {{Utils.bytesToString}}.
Where the user supplies a value like "700m" it's interpreted as mebibytes.
That's fine and at least unambiguous and not-wrong.
I don't think we want to change behavior, but we can make display and log
output more consistent and correct.
The inconsistency is bad of course. The simple change is to change the
javascript, and at least update its strings to say "MiB" etc correctly. I think
{{Utils.bytesToString}} should be changed too. If you want to be a hero, you
might look for anywhere "MB" or "KB" occurs in the code and see if the value is
being computed correctly.
> Difference between Storage Memory as seen internally and in web UI
> ------------------------------------------------------------------
>
> Key: SPARK-20691
> URL: https://issues.apache.org/jira/browse/SPARK-20691
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.3.0
> Reporter: Jacek Laskowski
>
> I set Major priority as it's visible to a user.
> There's a difference in what the size of Storage Memory is managed internally
> and displayed to a user in web UI.
> I found it while answering [How does web UI calculate Storage Memory (in
> Executors tab)?|http://stackoverflow.com/q/43801062/1305344] on StackOverflow.
> In short (quoting the main parts), when you start a Spark app (say
> spark-shell) you see 912.3 MB RAM for Storage Memory:
> {code}
> $ ./bin/spark-shell --conf spark.driver.memory=2g
> ...
> 17/05/07 15:20:50 INFO BlockManagerMasterEndpoint: Registering block manager
> 192.168.1.8:57177 with 912.3 MB RAM, BlockManagerId(driver, 192.168.1.8,
> 57177, None)
> {code}
> but in the web UI you'll see 956.6 MB due to the way the custom JavaScript
> function {{formatBytes}} in
> [utils.js|https://github.com/apache/spark/blob/master/core/src/main/resources/org/apache/spark/ui/static/utils.js#L40-L48]
> calculates the value. That translates to the following Scala code:
> {code}
> def formatBytes(bytes: Double) = {
> val k = 1000
> val i = math.floor(math.log(bytes) / math.log(k))
> val maxMemoryWebUI = bytes / math.pow(k, i)
> f"$maxMemoryWebUI%1.1f"
> }
> scala> println(formatBytes(maxMemory))
> 956.6
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
