This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 16bbe0f [SPARK-27486][CORE][TEST] Enable History server storage
information test in the HistoryServerSuite
16bbe0f is described below
commit 16bbe0f798d2a75ba0a9f2daff709292f6971755
Author: Shahid <[email protected]>
AuthorDate: Fri Apr 19 08:12:20 2019 -0700
[SPARK-27486][CORE][TEST] Enable History server storage information test in
the HistoryServerSuite
## What changes were proposed in this pull request?
We have disabled a test related to storage in the History server suite
after SPARK-13845. But, after SPARK-22050, we can store the information about
block updated events to eventLog, if we enable
"spark.eventLog.logBlockUpdates.enabled=true".
So, we can enable the test, by adding an eventlog corresponding to the
application, which has enabled the configuration,
"spark.eventLog.logBlockUpdates.enabled=true"
## How was this patch tested?
Existing UTs
Closes #24390 from shahidki31/enableRddStorageTest.
Authored-by: Shahid <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
---
.../one_rdd_storage_json_expectation.json | 10 ++++++++++
.../org/apache/spark/deploy/history/HistoryServerSuite.scala | 8 +++++---
2 files changed, 15 insertions(+), 3 deletions(-)
diff --git
a/core/src/test/resources/HistoryServerExpectations/one_rdd_storage_json_expectation.json
b/core/src/test/resources/HistoryServerExpectations/one_rdd_storage_json_expectation.json
new file mode 100644
index 0000000..09afdf5
--- /dev/null
+++
b/core/src/test/resources/HistoryServerExpectations/one_rdd_storage_json_expectation.json
@@ -0,0 +1,10 @@
+{
+ "id" : 0,
+ "name" : "0",
+ "numPartitions" : 8,
+ "numCachedPartitions" : 0,
+ "storageLevel" : "Memory Deserialized 1x Replicated",
+ "memoryUsed" : 0,
+ "diskUsed" : 0,
+ "partitions" : [ ]
+}
diff --git
a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
index c99bcf2..5ba27a0 100644
---
a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
+++
b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala
@@ -174,9 +174,11 @@ class HistoryServerSuite extends SparkFunSuite with
BeforeAndAfter with Matchers
"executor node blacklisting unblacklisting" ->
"applications/app-20161115172038-0000/executors",
"executor memory usage" ->
"applications/app-20161116163331-0000/executors",
- "app environment" -> "applications/app-20161116163331-0000/environment"
- // Todo: enable this test when logging the even of onBlockUpdated. See:
SPARK-13845
- // "one rdd storage json" ->
"applications/local-1422981780767/storage/rdd/0"
+ "app environment" -> "applications/app-20161116163331-0000/environment",
+
+ // Enable "spark.eventLog.logBlockUpdates.enabled", to get the storage
information
+ // in the history server.
+ "one rdd storage json" -> "applications/local-1422981780767/storage/rdd/0"
)
// run a bunch of characterization tests -- just verify the behavior is the
same as what is saved
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
