HeartSaVioR commented on a change in pull request #26821:
[SPARK-20656][CORE]Support Incremental parsing of event logs in SHS
URL: https://github.com/apache/spark/pull/26821#discussion_r356994195
##########
File path:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
##########
@@ -948,24 +972,30 @@ private[history] class FsHistoryProvider(conf:
SparkConf, clock: Clock)
private def rebuildAppStore(
store: KVStore,
reader: EventLogFileReader,
- lastUpdated: Long): Unit = {
+ lastUpdated: Long,
+ incrimentInfo: Option[IncrimentalMetaInfo] = None): Unit = {
// Disable async updates, since they cause higher memory usage, and it's
ok to take longer
// to parse the event logs in the SHS.
val replayConf = conf.clone().set(ASYNC_TRACKING_ENABLED, false)
val trackingStore = new ElementTrackingStore(store, replayConf)
val replayBus = new ReplayListenerBus()
val listener = new AppStatusListener(trackingStore, replayConf, false,
lastUpdateTime = Some(lastUpdated))
+
+ incrimentInfo.foreach(info => listener.initialize(info.appId,
info.attemptId))
replayBus.addListener(listener)
for {
plugin <- loadPlugins()
listener <- plugin.createListeners(conf, trackingStore)
- } replayBus.addListener(listener)
+ } {
+ incrimentInfo.foreach(info => plugin.initialize(listener, info.appId,
info.attemptId))
Review comment:
Personally I'd add a new abstraction (like `RestorableSparkListener` or
`SnapshottableSparkListener`... better words are welcome) which has a
`initialize` method so that we can call `initialize` against listener instead
of plugin. But given more changes are required, let's hear more voices on
committers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
