[
https://issues.apache.org/jira/browse/SPARK-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrei Ivanov updated SPARK-16859:
----------------------------------
Description:
It looks like job history storage tab in history server is broken for completed
jobs since *1.6.2*.
More specifically it's broken since
[SPARK-13845|https://issues.apache.org/jira/browse/SPARK-13845].
I've fixed for my installation by effectively reverting the above patch
([see|https://github.com/EinsamHauer/spark/commit/3af62ea09af8bb350c8c8a9117149c09b8feba08]).
IMHO, the most straightforward fix would be to implement
_SparkListenerBlockUpdated_ serialization to JSON in _JsonProtocol_ making sure
it works from _ReplayListenerBus_.
The downside will be that it will still work incorrectly with pre patch job
histories. But then, it doesn't work since *1.6.2* anyhow.
PS: I'd really love to have this fixed eventually. But I'm pretty new to Apache
Spark and missing hands on Scala experience. So I'd prefer that it be fixed by
someone experienced with roadmap vision. If nobody volunteers I'll try to patch
myself.
was:
It looks like job history storage tab in history server is broken for completed
jobs since *1.6.2*.
More specifically it's broken since
[SPARK-13845|https://issues.apache.org/jira/browse/SPARK-13845].
I've fixed for my installation by effectively reverting the above patch
([see|https://github.com/EinsamHauer/spark/commit/3af62ea09af8bb350c8c8a9117149c09b8feba08]).
IMHO, the most straightforward fix would be to implement
_SparkListenerBlockUpdated_ serialization to JSON in _JsonProtocol_ and making
sure it works from _ReplayListenerBus_.
The downside will be that it will still work incorrectly with pre patch job
histories. But then, it doesn't work since *1.6.2* anyhow.
PS: I'd really love to have this fixed eventually. But I'm pretty new to Apache
Spark and missing hands on Scala experience. So I'd prefer that it be fixed by
someone experienced with roadmap vision. If nobody volunteers I'll try to patch
myself.
> History Server storage information is missing
> ---------------------------------------------
>
> Key: SPARK-16859
> URL: https://issues.apache.org/jira/browse/SPARK-16859
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.6.2, 2.0.0
> Reporter: Andrei Ivanov
> Labels: historyserver, newbie
>
> It looks like job history storage tab in history server is broken for
> completed jobs since *1.6.2*.
> More specifically it's broken since
> [SPARK-13845|https://issues.apache.org/jira/browse/SPARK-13845].
> I've fixed for my installation by effectively reverting the above patch
> ([see|https://github.com/EinsamHauer/spark/commit/3af62ea09af8bb350c8c8a9117149c09b8feba08]).
> IMHO, the most straightforward fix would be to implement
> _SparkListenerBlockUpdated_ serialization to JSON in _JsonProtocol_ making
> sure it works from _ReplayListenerBus_.
> The downside will be that it will still work incorrectly with pre patch job
> histories. But then, it doesn't work since *1.6.2* anyhow.
> PS: I'd really love to have this fixed eventually. But I'm pretty new to
> Apache Spark and missing hands on Scala experience. So I'd prefer that it be
> fixed by someone experienced with roadmap vision. If nobody volunteers I'll
> try to patch myself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
