tgravescs commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-624070783
concept seems good, a couple of high level questions without me having looked at the detailed code. I assume the number of threads to read is still spark.history.fs.numReplayThreads? It looks like you only have one thread to write to levelDB. Is this enough for it to not be backed up? Lets say we have a very active history server and user loading a lot of large files, does the memory balloon before it can be flushed to levelDB? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
