Github user ivoson commented on the issue:
https://github.com/apache/spark/pull/22252
Based on the discussion above, I think updating the doc to guide users to
explicitly identify the unit may be necessary. The last commit update the doc.
cc @vanzin @srowen @xuanyuanking
Also I think the unit for the conf should be finally unified because even
for yarn-cluster or K8S, spark.executor.memory is not just used to allocate
resource where parsed as MiB in default, while some other places parsed as
bytes in default for some validation, such as:
https://github.com/apache/spark/blob/ff8dcc1d4c684e1b68e63d61b3f20284b9979cca/core/src/main/scala/org/apache/spark/SparkContext.scala#L465-L470
https://github.com/apache/spark/blob/ff8dcc1d4c684e1b68e63d61b3f20284b9979cca/core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala#L208-L233
https://github.com/apache/spark/blob/ff8dcc1d4c684e1b68e63d61b3f20284b9979cca/core/src/main/scala/org/apache/spark/memory/StaticMemoryManager.scala#L121-L143
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
