[
https://issues.apache.org/jira/browse/SPARK-31584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Baohe Zhang updated SPARK-31584:
--------------------------------
Attachment: errorstack.txt
> NullPointerException when parsing event log with InMemoryStore
> --------------------------------------------------------------
>
> Key: SPARK-31584
> URL: https://issues.apache.org/jira/browse/SPARK-31584
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 3.0.1
> Reporter: Baohe Zhang
> Priority: Minor
> Fix For: 3.0.1
>
> Attachments: errorstack.txt
>
>
> I compiled with the current branch-3.0 source and tested it in mac os. A
> java.lang.NullPointerException will be thrown when below conditions are met:
> # Using InMemoryStore as kvstore when parsing the event log file (e.g., when
> spark.history.store.path is unset).
> # At least one stage in this event log has task number greater than
> spark.ui.retainedTasks (by default is 100000). In this case, kvstore needs to
> delete extra task records.
> # The job has more than one stage, so parentToChildrenMap in
> InMemoryStore.java will have more than one key.
> The java.lang.NullPointerException is thrown in InMemoryStore.java :296. In
> the method deleteParentIndex().
> {code:java}
> private void deleteParentIndex(Object key) {
> if (hasNaturalParentIndex) {
> for (NaturalKeys v : parentToChildrenMap.values()) {
> if (v.remove(asKey(key))) {
> // `v` can be empty after removing the natural key and we can
> remove it from
> // `parentToChildrenMap`. However, `parentToChildrenMap` is a
> ConcurrentMap and such
> // checking and deleting can be slow.
> // This method is to delete one object with certain key, let's
> make it simple here.
> break;
> }
> }
> }
> }{code}
> In “if (v.remove(asKey(key)))”, if the key is not contained in v,
> "v.remove(asKey(key))" will return null, and java will throw a
> NullPointerException when executing "if (null)".
> An exception stack trace is attached.
> This issue can be fixed by updating if statement to
> {code:java}
> if (v.remove(asKey(key)) != null){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
