Baohe Zhang created SPARK-31584:
-----------------------------------
Summary: NullPointerException when parsing event log with
InMemoryStore
Key: SPARK-31584
URL: https://issues.apache.org/jira/browse/SPARK-31584
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 3.0.1
Reporter: Baohe Zhang
Fix For: 3.0.1
I compiled with the current branch-3.0 source and tested it in mac os. A
java.lang.NullPointerException will be thrown when below conditions are met:
# Using InMemoryStore as kvstore when parsing the event log file (e.g., when
spark.history.store.path is unset).
# At least one stage in this event log has task number greater than
spark.ui.retainedTasks (by default is 100000). In this case, kvstore needs to
delete extra task records.
# The job has more than one stage, so parentToChildrenMap in
InMemoryStore.java will have more than one key.
The java.lang.NullPointerException is thrown in InMemoryStore.java :296. In the
method deleteParentIndex().
{code:java}
private void deleteParentIndex(Object key) {
if (hasNaturalParentIndex) {
for (NaturalKeys v : parentToChildrenMap.values()) {
if (v.remove(asKey(key))) {
// `v` can be empty after removing the natural key and we can
remove it from
// `parentToChildrenMap`. However, `parentToChildrenMap` is a
ConcurrentMap and such
// checking and deleting can be slow.
// This method is to delete one object with certain key, let's make
it simple here.
break;
}
}
}
}{code}
In “if (v.remove(asKey(key)))”, if the key is not contained in v,
"v.remove(asKey(key))" will return null, and java will throw a
NullPointerException when executing "if (null)".
An exception stack trace is attached.
This issue can be fixed by updating if statement to
{code:java}
if (v.remove(asKey(key)) != null){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
