JoshRosen commented on code in PR #37027:
URL: https://github.com/apache/spark/pull/37027#discussion_r910520108
##########
core/src/main/scala/org/apache/spark/util/JsonProtocol.scala:
##########
@@ -512,6 +512,7 @@ private[spark] object JsonProtocol {
def storageLevelToJson(storageLevel: StorageLevel): JValue = {
("Use Disk" -> storageLevel.useDisk) ~
("Use Memory" -> storageLevel.useMemory) ~
+ ("Use Off Heap" -> storageLevel.useOffHeap) ~
Review Comment:
- The `useOffHeap` field was added in Spark 1.0 in PR #158 and was persisted
to JsonProtocol as a "Use Tachyon" field (since that was the original meaning
of that field).
- It was subsequently renamed to "Use ExternalBlockStore" in #5430.
- It was later removed in #10752 as part of removal of the legacy external
block store API.
- In #11805 for Spark 2.x I repurposed the then-orphaned `OFF_HEAP` storage
level for use with off-heap caching, but overlooked the fact that that the
`useOffHeap` field was no longer being recorded in `JsonProtocol`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
