mridulm commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-656418437
Adding batch update is a perf improvement - not required for the functionality being added. I mentioned it given I did it recently and similar snippet was pretty much exactly what I had replaced :-) On Thu, Jul 9, 2020 at 9:04 AM Baohe Zhang <[email protected]> wrote: > *@baohe-zhang* commented on this pull request. > ------------------------------ > > In core/src/main/scala/org/apache/spark/deploy/history/HybridStore.scala > <https://github.com/apache/spark/pull/28412#discussion_r452327580>: > > > + */ > + def switchToLevelDB( > + listener: HybridStore.SwitchToLevelDBListener, > + appId: String, > + attemptId: Option[String]): Unit = { > + if (closed.get) { > + return > + } > + > + backgroundThread = new Thread(() => { > + try { > + for (klass <- klassMap.keys().asScala) { > + val it = inMemoryStore.view(klass).closeableIterator() > + while (it.hasNext()) { > + levelDB.write(it.next()) > + } > > Make sense. I am testing it on the mac which has SSD and the disk is not > busy. I think the improvement might be more obvious on HDD or busy disk. > @HeartSaVioR <https://github.com/HeartSaVioR> @tgravescs > <https://github.com/tgravescs> Do we need to add batch write support for > leveldb on this pr? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/spark/pull/28412#discussion_r452327580>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAMETFA74MHMBGAZ5XWWC43R2XTCFANCNFSM4MUQYGFQ> > . > ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
