[jira] [Updated] (SPARK-41053) Better Spark UI scalability and Driver stability for large applications
[
https://issues.apache.org/jira/browse/SPARK-41053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-41053:
--
Fix Version/s: 3.4.0
> Better Spark UI scalability and Driver stability for large applications
> ---
>
> Key: SPARK-41053
> URL: https://issues.apache.org/jira/browse/SPARK-41053
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
>Affects Versions: 3.4.0
>Reporter: Gengliang Wang
>Assignee: Gengliang Wang
>Priority: Major
> Labels: releasenotes
> Fix For: 3.4.0
>
> Attachments: Better Spark UI scalability and Driver stability for
> large applications.pdf
>
>
> After SPARK-18085, the Spark history server(SHS) becomes more scalable for
> processing large applications by supporting a persistent
> KV-store(LevelDB/RocksDB) as the storage layer.
> As for the live Spark UI, all the data is still stored in memory, which can
> bring memory pressures to the Spark driver for large applications.
> For better Spark UI scalability and Driver stability, I propose to
> * {*}Support storing all the UI data in a persistent KV store{*}.
> RocksDB/LevelDB provides low memory overhead. Their write/read performance is
> fast enough to serve the write/read workload for live UI. SHS can leverage
> the persistent KV store to fasten its startup.
> * *Support a new Protobuf serializer for all the UI data.* The new
> serializer is supposed to be faster, according to benchmarks. It will be the
> default serializer for the persistent KV store of live UI. As for event logs,
> it is optional. The current serializer for UI data is JSON. When writing
> persistent KV-store, there is GZip compression. Since there is compression
> support in RocksDB/LevelDB, the new serializer won’t compress the output
> before writing to the persistent KV store. Here is a benchmark of
> writing/reading 100,000 SQLExecutionUIData to/from RocksDB:
>
> |*Serializer*|*Avg Write time(μs)*|*Avg Read time(μs)*|*RocksDB File Total
> Size(MB)*|*Result total size in memory(MB)*|
> |*Spark’s KV Serializer(JSON+gzip)*|352.2|119.26|837|868|
> |*Protobuf*|109.9|34.3|858|2105|
> I am also proposing to support RocksDB instead of both LevelDB & RocksDB in
> the live UI.
> SPIP:
> [https://docs.google.com/document/d/1cuKnFwlTodyVhUQPMuakq2YDaLH05jaY9FRu_aD1zMo/edit?usp=sharing]
> SPIP vote: https://lists.apache.org/thread/lom4zcob6237q6nnj46jylkzwmmsxvgj
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-41053) Better Spark UI scalability and Driver stability for large applications
[
https://issues.apache.org/jira/browse/SPARK-41053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-41053:
--
Labels: releasenotes (was: release-notes)
> Better Spark UI scalability and Driver stability for large applications
> ---
>
> Key: SPARK-41053
> URL: https://issues.apache.org/jira/browse/SPARK-41053
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
>Affects Versions: 3.4.0
>Reporter: Gengliang Wang
>Priority: Major
> Labels: releasenotes
> Attachments: Better Spark UI scalability and Driver stability for
> large applications.pdf
>
>
> After SPARK-18085, the Spark history server(SHS) becomes more scalable for
> processing large applications by supporting a persistent
> KV-store(LevelDB/RocksDB) as the storage layer.
> As for the live Spark UI, all the data is still stored in memory, which can
> bring memory pressures to the Spark driver for large applications.
> For better Spark UI scalability and Driver stability, I propose to
> * {*}Support storing all the UI data in a persistent KV store{*}.
> RocksDB/LevelDB provides low memory overhead. Their write/read performance is
> fast enough to serve the write/read workload for live UI. SHS can leverage
> the persistent KV store to fasten its startup.
> * *Support a new Protobuf serializer for all the UI data.* The new
> serializer is supposed to be faster, according to benchmarks. It will be the
> default serializer for the persistent KV store of live UI. As for event logs,
> it is optional. The current serializer for UI data is JSON. When writing
> persistent KV-store, there is GZip compression. Since there is compression
> support in RocksDB/LevelDB, the new serializer won’t compress the output
> before writing to the persistent KV store. Here is a benchmark of
> writing/reading 100,000 SQLExecutionUIData to/from RocksDB:
>
> |*Serializer*|*Avg Write time(μs)*|*Avg Read time(μs)*|*RocksDB File Total
> Size(MB)*|*Result total size in memory(MB)*|
> |*Spark’s KV Serializer(JSON+gzip)*|352.2|119.26|837|868|
> |*Protobuf*|109.9|34.3|858|2105|
> I am also proposing to support RocksDB instead of both LevelDB & RocksDB in
> the live UI.
> SPIP:
> [https://docs.google.com/document/d/1cuKnFwlTodyVhUQPMuakq2YDaLH05jaY9FRu_aD1zMo/edit?usp=sharing]
> SPIP vote: https://lists.apache.org/thread/lom4zcob6237q6nnj46jylkzwmmsxvgj
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-41053) Better Spark UI scalability and Driver stability for large applications
[
https://issues.apache.org/jira/browse/SPARK-41053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-41053:
---
Labels: release-notes (was: )
> Better Spark UI scalability and Driver stability for large applications
> ---
>
> Key: SPARK-41053
> URL: https://issues.apache.org/jira/browse/SPARK-41053
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
>Affects Versions: 3.4.0
>Reporter: Gengliang Wang
>Priority: Major
> Labels: release-notes
> Attachments: Better Spark UI scalability and Driver stability for
> large applications.pdf
>
>
> After SPARK-18085, the Spark history server(SHS) becomes more scalable for
> processing large applications by supporting a persistent
> KV-store(LevelDB/RocksDB) as the storage layer.
> As for the live Spark UI, all the data is still stored in memory, which can
> bring memory pressures to the Spark driver for large applications.
> For better Spark UI scalability and Driver stability, I propose to
> * {*}Support storing all the UI data in a persistent KV store{*}.
> RocksDB/LevelDB provides low memory overhead. Their write/read performance is
> fast enough to serve the write/read workload for live UI. SHS can leverage
> the persistent KV store to fasten its startup.
> * *Support a new Protobuf serializer for all the UI data.* The new
> serializer is supposed to be faster, according to benchmarks. It will be the
> default serializer for the persistent KV store of live UI. As for event logs,
> it is optional. The current serializer for UI data is JSON. When writing
> persistent KV-store, there is GZip compression. Since there is compression
> support in RocksDB/LevelDB, the new serializer won’t compress the output
> before writing to the persistent KV store. Here is a benchmark of
> writing/reading 100,000 SQLExecutionUIData to/from RocksDB:
>
> |*Serializer*|*Avg Write time(μs)*|*Avg Read time(μs)*|*RocksDB File Total
> Size(MB)*|*Result total size in memory(MB)*|
> |*Spark’s KV Serializer(JSON+gzip)*|352.2|119.26|837|868|
> |*Protobuf*|109.9|34.3|858|2105|
> I am also proposing to support RocksDB instead of both LevelDB & RocksDB in
> the live UI.
> SPIP:
> [https://docs.google.com/document/d/1cuKnFwlTodyVhUQPMuakq2YDaLH05jaY9FRu_aD1zMo/edit?usp=sharing]
> SPIP vote: https://lists.apache.org/thread/lom4zcob6237q6nnj46jylkzwmmsxvgj
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-41053) Better Spark UI scalability and Driver stability for large applications
[
https://issues.apache.org/jira/browse/SPARK-41053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-41053:
---
Description:
After SPARK-18085, the Spark history server(SHS) becomes more scalable for
processing large applications by supporting a persistent
KV-store(LevelDB/RocksDB) as the storage layer.
As for the live Spark UI, all the data is still stored in memory, which can
bring memory pressures to the Spark driver for large applications.
For better Spark UI scalability and Driver stability, I propose to
* {*}Support storing all the UI data in a persistent KV store{*}.
RocksDB/LevelDB provides low memory overhead. Their write/read performance is
fast enough to serve the write/read workload for live UI. SHS can leverage the
persistent KV store to fasten its startup.
* *Support a new Protobuf serializer for all the UI data.* The new serializer
is supposed to be faster, according to benchmarks. It will be the default
serializer for the persistent KV store of live UI. As for event logs, it is
optional. The current serializer for UI data is JSON. When writing persistent
KV-store, there is GZip compression. Since there is compression support in
RocksDB/LevelDB, the new serializer won’t compress the output before writing to
the persistent KV store. Here is a benchmark of writing/reading 100,000
SQLExecutionUIData to/from RocksDB:
|*Serializer*|*Avg Write time(μs)*|*Avg Read time(μs)*|*RocksDB File Total
Size(MB)*|*Result total size in memory(MB)*|
|*Spark’s KV Serializer(JSON+gzip)*|352.2|119.26|837|868|
|*Protobuf*|109.9|34.3|858|2105|
I am also proposing to support RocksDB instead of both LevelDB & RocksDB in the
live UI.
SPIP:
[https://docs.google.com/document/d/1cuKnFwlTodyVhUQPMuakq2YDaLH05jaY9FRu_aD1zMo/edit?usp=sharing]
SPIP vote: https://lists.apache.org/thread/lom4zcob6237q6nnj46jylkzwmmsxvgj
was:
After SPARK-18085, the Spark history server(SHS) becomes more scalable for
processing large applications by supporting a persistent
KV-store(LevelDB/RocksDB) as the storage layer.
As for the live Spark UI, all the data is still stored in memory, which can
bring memory pressures to the Spark driver for large applications.
For better Spark UI scalability and Driver stability, I propose to
* {*}Support storing all the UI data in a persistent KV store{*}.
RocksDB/LevelDB provides low memory overhead. Their write/read performance is
fast enough to serve the write/read workload for live UI. SHS can leverage the
persistent KV store to fasten its startup.
* *Support a new Protobuf serializer for all the UI data.* The new serializer
is supposed to be faster, according to benchmarks. It will be the default
serializer for the persistent KV store of live UI. As for event logs, it is
optional. The current serializer for UI data is JSON. When writing persistent
KV-store, there is GZip compression. Since there is compression support in
RocksDB/LevelDB, the new serializer won’t compress the output before writing to
the persistent KV store. Here is a benchmark of writing/reading 100,000
SQLExecutionUIData to/from RocksDB:
|*Serializer*|*Avg Write time(μs)*|*Avg Read time(μs)*|*RocksDB File Total
Size(MB)*|*Result total size in memory(MB)*|
|*Spark’s KV Serializer(JSON+gzip)*|352.2|119.26|837|868|
|*Protobuf*|109.9|34.3|858|2105|
I am also proposing to support RocksDB instead of both LevelDB & RocksDB in the
live UI.
SPIP:
[https://docs.google.com/document/d/1cuKnFwlTodyVhUQPMuakq2YDaLH05jaY9FRu_aD1zMo/edit?usp=sharing]
> Better Spark UI scalability and Driver stability for large applications
> ---
>
> Key: SPARK-41053
> URL: https://issues.apache.org/jira/browse/SPARK-41053
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
>Affects Versions: 3.4.0
>Reporter: Gengliang Wang
>Priority: Major
> Attachments: Better Spark UI scalability and Driver stability for
> large applications.pdf
>
>
> After SPARK-18085, the Spark history server(SHS) becomes more scalable for
> processing large applications by supporting a persistent
> KV-store(LevelDB/RocksDB) as the storage layer.
> As for the live Spark UI, all the data is still stored in memory, which can
> bring memory pressures to the Spark driver for large applications.
> For better Spark UI scalability and Driver stability, I propose to
> * {*}Support storing all the UI data in a persistent KV store{*}.
> RocksDB/LevelDB provides low memory overhead. Their write/read performance is
> fast enough to serve the write/read workload for live UI. SHS can leverage
> the persistent KV store to fasten its startup.
> * *Support a new Protobuf serializer for all the UI data.* The new
> serializer is supposed to be faster, according to
[jira] [Updated] (SPARK-41053) Better Spark UI scalability and Driver stability for large applications
[
https://issues.apache.org/jira/browse/SPARK-41053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-41053:
---
Attachment: Better Spark UI scalability and Driver stability for large
applications.pdf
> Better Spark UI scalability and Driver stability for large applications
> ---
>
> Key: SPARK-41053
> URL: https://issues.apache.org/jira/browse/SPARK-41053
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
>Affects Versions: 3.4.0
>Reporter: Gengliang Wang
>Priority: Major
> Attachments: Better Spark UI scalability and Driver stability for
> large applications.pdf
>
>
> After SPARK-18085, the Spark history server(SHS) becomes more scalable for
> processing large applications by supporting a persistent
> KV-store(LevelDB/RocksDB) as the storage layer.
> As for the live Spark UI, all the data is still stored in memory, which can
> bring memory pressures to the Spark driver for large applications.
> For better Spark UI scalability and Driver stability, I propose to
> * {*}Support storing all the UI data in a persistent KV store{*}.
> RocksDB/LevelDB provides low memory overhead. Their write/read performance is
> fast enough to serve the write/read workload for live UI. SHS can leverage
> the persistent KV store to fasten its startup.
> * *Support a new Protobuf serializer for all the UI data.* The new
> serializer is supposed to be faster, according to benchmarks. It will be the
> default serializer for the persistent KV store of live UI. As for event logs,
> it is optional. The current serializer for UI data is JSON. When writing
> persistent KV-store, there is GZip compression. Since there is compression
> support in RocksDB/LevelDB, the new serializer won’t compress the output
> before writing to the persistent KV store. Here is a benchmark of
> writing/reading 100,000 SQLExecutionUIData to/from RocksDB:
>
> |*Serializer*|*Avg Write time(μs)*|*Avg Read time(μs)*|*RocksDB File Total
> Size(MB)*|*Result total size in memory(MB)*|
> |*Spark’s KV Serializer(JSON+gzip)*|352.2|119.26|837|868|
> |*Protobuf*|109.9|34.3|858|2105|
> I am also proposing to support RocksDB instead of both LevelDB & RocksDB in
> the live UI.
> SPIP:
> [https://docs.google.com/document/d/1cuKnFwlTodyVhUQPMuakq2YDaLH05jaY9FRu_aD1zMo/edit?usp=sharing]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-41053) Better Spark UI scalability and Driver stability for large applications
[
https://issues.apache.org/jira/browse/SPARK-41053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-41053:
---
Attachment: (was: Better_UI_Scalability.png)
> Better Spark UI scalability and Driver stability for large applications
> ---
>
> Key: SPARK-41053
> URL: https://issues.apache.org/jira/browse/SPARK-41053
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
>Affects Versions: 3.4.0
>Reporter: Gengliang Wang
>Priority: Major
>
> After SPARK-18085, the Spark history server(SHS) becomes more scalable for
> processing large applications by supporting a persistent
> KV-store(LevelDB/RocksDB) as the storage layer.
> As for the live Spark UI, all the data is still stored in memory, which can
> bring memory pressures to the Spark driver for large applications.
> For better Spark UI scalability and Driver stability, I propose to
> * {*}Support storing all the UI data in a persistent KV store{*}.
> RocksDB/LevelDB provides low memory overhead. Their write/read performance is
> fast enough to serve the write/read workload for live UI. SHS can leverage
> the persistent KV store to fasten its startup.
> * *Support a new Protobuf serializer for all the UI data.* The new
> serializer is supposed to be faster, according to benchmarks. It will be the
> default serializer for the persistent KV store of live UI. As for event logs,
> it is optional. The current serializer for UI data is JSON. When writing
> persistent KV-store, there is GZip compression. Since there is compression
> support in RocksDB/LevelDB, the new serializer won’t compress the output
> before writing to the persistent KV store. Here is a benchmark of
> writing/reading 100,000 SQLExecutionUIData to/from RocksDB:
>
> |*Serializer*|*Avg Write time(μs)*|*Avg Read time(μs)*|*RocksDB File Total
> Size(MB)*|*Result total size in memory(MB)*|
> |*Spark’s KV Serializer(JSON+gzip)*|352.2|119.26|837|868|
> |*Protobuf*|109.9|34.3|858|2105|
> I am also proposing to support RocksDB instead of both LevelDB & RocksDB in
> the live UI.
> SPIP:
> [https://docs.google.com/document/d/1cuKnFwlTodyVhUQPMuakq2YDaLH05jaY9FRu_aD1zMo/edit?usp=sharing]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-41053) Better Spark UI scalability and Driver stability for large applications
[
https://issues.apache.org/jira/browse/SPARK-41053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-41053:
---
Description:
After SPARK-18085, the Spark history server(SHS) becomes more scalable for
processing large applications by supporting a persistent
KV-store(LevelDB/RocksDB) as the storage layer.
As for the live Spark UI, all the data is still stored in memory, which can
bring memory pressures to the Spark driver for large applications.
For better Spark UI scalability and Driver stability, I propose to
* {*}Support storing all the UI data in a persistent KV store{*}.
RocksDB/LevelDB provides low memory overhead. Their write/read performance is
fast enough to serve the write/read workload for live UI. SHS can leverage the
persistent KV store to fasten its startup.
* *Support a new Protobuf serializer for all the UI data.* The new serializer
is supposed to be faster, according to benchmarks. It will be the default
serializer for the persistent KV store of live UI. As for event logs, it is
optional. The current serializer for UI data is JSON. When writing persistent
KV-store, there is GZip compression. Since there is compression support in
RocksDB/LevelDB, the new serializer won’t compress the output before writing to
the persistent KV store. Here is a benchmark of writing/reading 100,000
SQLExecutionUIData to/from RocksDB:
|*Serializer*|*Avg Write time(μs)*|*Avg Read time(μs)*|*RocksDB File Total
Size(MB)*|*Result total size in memory(MB)*|
|*Spark’s KV Serializer(JSON+gzip)*|352.2|119.26|837|868|
|*Protobuf*|109.9|34.3|858|2105|
I am also proposing to support RocksDB instead of both LevelDB & RocksDB in the
live UI.
SPIP:
[https://docs.google.com/document/d/1cuKnFwlTodyVhUQPMuakq2YDaLH05jaY9FRu_aD1zMo/edit?usp=sharing]
was:
After SPARK-18085, the Spark history server(SHS) becomes more scalable for
processing large applications by supporting a persistent
KV-store(LevelDB/RocksDB) as the storage layer.
As for the live Spark UI, all the data is still stored in memory, which can
bring memory pressures to the Spark driver for large applications.
For better stability of the Spark driver, I propose to
* {*}Support storing all the UI data in a persistent KV store{*}.
RocksDB/LevelDB provides low memory overhead. Their write/read performance is
fast enough to serve the write/read workload for live UI. SHS can leverage the
persistent KV store to fasten its startup.
* *Support a new Protobuf serializer for all the UI data.* The new serializer
is supposed to be faster, according to benchmarks. It will be the default
serializer for the persistent KV store of live UI. As for event logs, it is
optional. The current serializer for UI data is JSON. When writing persistent
KV-store, there is GZip compression. Since there is compression support in
RocksDB/LevelDB, the new serializer won’t compress the output before writing to
the persistent KV store. Here is a benchmark of writing/reading 100,000
SQLExecutionUIData to/from RocksDB:
|*Serializer*|*Avg Write time(μs)*|*Avg Read time(μs)*|*RocksDB File Total
Size(MB)*|*Result total size in memory(MB)*|
|*Spark’s KV Serializer(JSON+gzip)*|352.2|119.26|837|868|
|*Protobuf*|109.9|34.3|858|2105|
I am also proposing to support RocksDB instead of both LevelDB & RocksDB in the
live UI.
SPIP:
https://docs.google.com/document/d/1cuKnFwlTodyVhUQPMuakq2YDaLH05jaY9FRu_aD1zMo/edit?usp=sharing
> Better Spark UI scalability and Driver stability for large applications
> ---
>
> Key: SPARK-41053
> URL: https://issues.apache.org/jira/browse/SPARK-41053
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
>Affects Versions: 3.4.0
>Reporter: Gengliang Wang
>Priority: Major
> Attachments: Better_UI_Scalability.png
>
>
> After SPARK-18085, the Spark history server(SHS) becomes more scalable for
> processing large applications by supporting a persistent
> KV-store(LevelDB/RocksDB) as the storage layer.
> As for the live Spark UI, all the data is still stored in memory, which can
> bring memory pressures to the Spark driver for large applications.
> For better Spark UI scalability and Driver stability, I propose to
> * {*}Support storing all the UI data in a persistent KV store{*}.
> RocksDB/LevelDB provides low memory overhead. Their write/read performance is
> fast enough to serve the write/read workload for live UI. SHS can leverage
> the persistent KV store to fasten its startup.
> * *Support a new Protobuf serializer for all the UI data.* The new
> serializer is supposed to be faster, according to benchmarks. It will be the
> default serializer for the persistent KV store of live UI. As for event logs,
> it is optional. The current serializer
[jira] [Updated] (SPARK-41053) Better Spark UI scalability and Driver stability for large applications
[
https://issues.apache.org/jira/browse/SPARK-41053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-41053:
---
Attachment: Better_UI_Scalability.png
> Better Spark UI scalability and Driver stability for large applications
> ---
>
> Key: SPARK-41053
> URL: https://issues.apache.org/jira/browse/SPARK-41053
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
>Affects Versions: 3.4.0
>Reporter: Gengliang Wang
>Priority: Major
> Attachments: Better_UI_Scalability.png
>
>
> After SPARK-18085, the Spark history server(SHS) becomes more scalable for
> processing large applications by supporting a persistent
> KV-store(LevelDB/RocksDB) as the storage layer.
> As for the live Spark UI, all the data is still stored in memory, which can
> bring memory pressures to the Spark driver for large applications.
> For better stability of the Spark driver, I propose to
> * {*}Support storing all the UI data in a persistent KV store{*}.
> RocksDB/LevelDB provides low memory overhead. Their write/read performance is
> fast enough to serve the write/read workload for live UI. SHS can leverage
> the persistent KV store to fasten its startup.
> * *Support a new Protobuf serializer for all the UI data.* The new
> serializer is supposed to be faster, according to benchmarks. It will be the
> default serializer for the persistent KV store of live UI. As for event logs,
> it is optional. The current serializer for UI data is JSON. When writing
> persistent KV-store, there is GZip compression. Since there is compression
> support in RocksDB/LevelDB, the new serializer won’t compress the output
> before writing to the persistent KV store. Here is a benchmark of
> writing/reading 100,000 SQLExecutionUIData to/from RocksDB:
>
> |*Serializer*|*Avg Write time(μs)*|*Avg Read time(μs)*|*RocksDB File Total
> Size(MB)*|*Result total size in memory(MB)*|
> |*Spark’s KV Serializer(JSON+gzip)*|352.2|119.26|837|868|
> |*Protobuf*|109.9|34.3|858|2105|
> I am also proposing to support RocksDB instead of both LevelDB & RocksDB in
> the live UI.
> SPIP:
> https://docs.google.com/document/d/1cuKnFwlTodyVhUQPMuakq2YDaLH05jaY9FRu_aD1zMo/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-41053) Better Spark UI scalability and Driver stability for large applications
[
https://issues.apache.org/jira/browse/SPARK-41053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang updated SPARK-41053:
---
Description:
After SPARK-18085, the Spark history server(SHS) becomes more scalable for
processing large applications by supporting a persistent
KV-store(LevelDB/RocksDB) as the storage layer.
As for the live Spark UI, all the data is still stored in memory, which can
bring memory pressures to the Spark driver for large applications.
For better stability of the Spark driver, I propose to
* {*}Support storing all the UI data in a persistent KV store{*}.
RocksDB/LevelDB provides low memory overhead. Their write/read performance is
fast enough to serve the write/read workload for live UI. SHS can leverage the
persistent KV store to fasten its startup.
* *Support a new Protobuf serializer for all the UI data.* The new serializer
is supposed to be faster, according to benchmarks. It will be the default
serializer for the persistent KV store of live UI. As for event logs, it is
optional. The current serializer for UI data is JSON. When writing persistent
KV-store, there is GZip compression. Since there is compression support in
RocksDB/LevelDB, the new serializer won’t compress the output before writing to
the persistent KV store. Here is a benchmark of writing/reading 100,000
SQLExecutionUIData to/from RocksDB:
|*Serializer*|*Avg Write time(μs)*|*Avg Read time(μs)*|*RocksDB File Total
Size(MB)*|*Result total size in memory(MB)*|
|*Spark’s KV Serializer(JSON+gzip)*|352.2|119.26|837|868|
|*Protobuf*|109.9|34.3|858|2105|
I am also proposing to support RocksDB instead of both LevelDB & RocksDB in the
live UI.
SPIP:
https://docs.google.com/document/d/1cuKnFwlTodyVhUQPMuakq2YDaLH05jaY9FRu_aD1zMo/edit?usp=sharing
was:
The current architecture of Spark live UI and Spark history server(SHS) is too
simple to serve large clusters and heavy workloads:
* Spark stores all the live UI date in memory. The size can be a few GBs and
affects the driver's stability (OOM).
* There is a limitation of storing 1000 queries only. Note that we can’t
simply increase the limitation under the current Architecture. I did a memory
profiling. Storing one query execution detail can take 800KB while storing one
task requires 0.3KB. So for 1000 SQL queries with 1000* 2000 tasks, the memory
usage for query execution and task data will be 1.4GB. Spark UI stores UI data
for jobs/stages/executors as well. So to store 10k queries, it may take more
than 14GB.
* SHS has to parse JSON format event log for the initial start. The
uncompressed event logs can be as big as a few GBs, and the parse can be quite
slow. Some users reported they had to wait for more than half an hour.
The proposal is to:
# Store all the live UI data in local RocksDB with protobuf serialization.
# The RocksDB files of live UI can be used on SHS directly.
# If the RocksDB file is unavailable for SHS, event logs can be written with
protobuf for faster replay.
> Better Spark UI scalability and Driver stability for large applications
> ---
>
> Key: SPARK-41053
> URL: https://issues.apache.org/jira/browse/SPARK-41053
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
>Affects Versions: 3.4.0
>Reporter: Gengliang Wang
>Priority: Major
>
> After SPARK-18085, the Spark history server(SHS) becomes more scalable for
> processing large applications by supporting a persistent
> KV-store(LevelDB/RocksDB) as the storage layer.
> As for the live Spark UI, all the data is still stored in memory, which can
> bring memory pressures to the Spark driver for large applications.
> For better stability of the Spark driver, I propose to
> * {*}Support storing all the UI data in a persistent KV store{*}.
> RocksDB/LevelDB provides low memory overhead. Their write/read performance is
> fast enough to serve the write/read workload for live UI. SHS can leverage
> the persistent KV store to fasten its startup.
> * *Support a new Protobuf serializer for all the UI data.* The new
> serializer is supposed to be faster, according to benchmarks. It will be the
> default serializer for the persistent KV store of live UI. As for event logs,
> it is optional. The current serializer for UI data is JSON. When writing
> persistent KV-store, there is GZip compression. Since there is compression
> support in RocksDB/LevelDB, the new serializer won’t compress the output
> before writing to the persistent KV store. Here is a benchmark of
> writing/reading 100,000 SQLExecutionUIData to/from RocksDB:
>
> |*Serializer*|*Avg Write time(μs)*|*Avg Read time(μs)*|*RocksDB File Total
> Size(MB)*|*Result total size in memory(MB)*|
> |*Spark’s KV
[jira] [Updated] (SPARK-41053) Better Spark UI scalability and Driver stability for large applications
[ https://issues.apache.org/jira/browse/SPARK-41053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-41053: --- Summary: Better Spark UI scalability and Driver stability for large applications (was: Support disk-based KV store in Spark live UI) > Better Spark UI scalability and Driver stability for large applications > --- > > Key: SPARK-41053 > URL: https://issues.apache.org/jira/browse/SPARK-41053 > Project: Spark > Issue Type: Umbrella > Components: Spark Core, Web UI >Affects Versions: 3.4.0 >Reporter: Gengliang Wang >Priority: Major > > The current architecture of Spark live UI and Spark history server(SHS) is > too simple to serve large clusters and heavy workloads: > * Spark stores all the live UI date in memory. The size can be a few GBs and > affects the driver's stability (OOM). > * There is a limitation of storing 1000 queries only. Note that we can’t > simply increase the limitation under the current Architecture. I did a memory > profiling. Storing one query execution detail can take 800KB while storing > one task requires 0.3KB. So for 1000 SQL queries with 1000* 2000 tasks, the > memory usage for query execution and task data will be 1.4GB. Spark UI stores > UI data for jobs/stages/executors as well. So to store 10k queries, it may > take more than 14GB. > * SHS has to parse JSON format event log for the initial start. The > uncompressed event logs can be as big as a few GBs, and the parse can be > quite slow. Some users reported they had to wait for more than half an hour. > > The proposal is to: > # Store all the live UI data in local RocksDB with protobuf serialization. > # The RocksDB files of live UI can be used on SHS directly. > # If the RocksDB file is unavailable for SHS, event logs can be written with > protobuf for faster replay. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
