[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-674948675 @gatorsmile Thanks for reminding me that! I will add unit tests for these HybridStore related PRs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-658453366 Thanks a lot for your reviews! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-655165640 @HeartSaVioR @tgravescs These are the measurement of memory usage on zstd compressed log files: * 200 jobs, 400 tasks for each job: 8.6 MB file size, 50.5 MB memory usage. * 100 jobs, 400 tasks for each job: 4.5 MB file size, 31.4 MB memory usage. * 50 jobs, 400 tasks for each job: 2.4 MB file size, 15. 7 MB memory usage. * 10 jobs: 400 tasks for each job: 816 KB file size, 4.7 MB memory usage. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-655130806 @tgravescs Sure I will run some tests on zstd. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-655128958 @tgravescs The latest number is based on uncompressed files. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-655002716 I measured the memory usage of some smaller apps, the results are: * 200 jobs, 400 tasks for each job: 265 MB file size, 57.9 MB memory usage. * 100 jobs, 400 tasks for each job: 133 MB file size, 28.5 MB memory usage. * 50 jobs, 400 tasks for each job: 67 MB file size, 14.9 MB memory usage. * 20 jobs: 400 tasks for each job: 35 MB file size, 8.3 MB memory usage --- * 10 jobs: 400 tasks for each job: 15 MB file size, 4.7 MB memory usage. * 1 job, 400 tasks for each job: 3.7 MB file size, 2.2 MB memory usage. * 1 job, 40 tasks for each job: 512 KB file size, 727 KB memory usage. I found that the ratio of memory_usage / filesize is stable at ¼ for log files larger than 30 MB. For log files with size less than 15 MB, the ratio of memory_usage/ filesize is greater than ¼, and the ratio increases as the file size decrease. The difference in serving latency between in-memory and leveldb store depends on the machine's performance. But personally I feel parsing a log file greater than 50MB with the hybrid store can have a notable improvement. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-654455672 @tgravescs I measured the memory usage of an app that contains 4 jobs, and each job contains 1 stage, each stage contains 10 tasks (the max retained tasks per stage by default). The number of entities stored in kvstore are 4 jobs, 4 stages, and 40 tasks. The log size of this app is 1.1 GB, and the memory usage is 212 MB. I measured another app that contains 1000 jobs (the max retained jobs by default), each job contains 1 stage and each stage contains 400 tasks. The total number of entities for this app are 1000 jobs, 1000 stages, and 40 tasks, which led to a 1.29 GB log file and 271 MB memory usage. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-653110422 Hi @HeartSaVioR @tgravescs , I measured the memory usage and disk usage for a 1.21g log file and logs for the same application with different compression codec. The log is generated by spark3 and parsed by spark3 SHS. The application contains 400 jobs, each job contains one stage, each stage contains 1000 tasks. | codec | uncompressed | lz4 | lzf | snappy | zstd | | | | | | | | | log filesize | 1.21 gb | 108 mb | 128 mb | 136 mb | 40 mb| | actual memory usage (measure through Utils.SizeEstimator) | 254.8 mb | 252.1 mb | 260.5 mb | 256.4 mb | 279.2 mb | | estimated memory usage (log size / 2 for uncompressed log, log size \* 2 for compressed log) | 605 mb | 216 mb | 256 mb | 272 mb | 80 mb| | disk usage (leveldb filesize) | 393 mb | 398 mb | 403 mb | 395 mb | 424 mb | From the result seems we are overestimating the memory usage of uncompressed files and underestimate the memory usage of zstd compressed files. I think filesize / 4 for uncompressed log, filesize * 4 for zstd compressed log might be a better estimation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-641024447 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-639668009 I did a comparison between hybridstore, in-memory store, and levledb for parsing large log files. The environment is mac os, spark-3.1.0. | store/ log file size | 250mb (SparkPi with 10 iterations) | 1.2gb (SparkPi with 50 iterations)| 2.3gb (SparkPi with 90 iterations) | | -- | | - | - | | HybridStore| 8s to parse, 16s to dump data to leveldb | 33s to parse, 16s to dump data to leveldb | 68s to parse, 15s to dump data to leveldb | | InMemoryStore | 8s to parse | 34s to parse | 68s to parse | | LevelDB| 28s to parse | 187s to parse | 434s to parse | This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-638515964 > Thanks for the contribution. The concept looks great. > > I've skimmed the implementation, and got the feeling that this is a bit complicated because of dual writes on the foreground & background. The complication is even exposed to the FsHistoryProvider. > > Given the HybridKVStore is only used for loading event log and no further exposed for modification (AFAIK - please correct me if I'm missing here), I'm seeing a chance to simplify the logic - dump in-memory KVStore back to LevelDB (in background) once the loading is done to the in-memory KVStore. (KVStore should be read-only and reject the writes.) We won't need to deal with dual writes on concurrent threads, only need to switch the KVStore correctly. > > This wouldn't bring latency on serving content - needs more seconds to write to LevelDB, in other words, needs more seconds to keep up the memory usage. (Not sure how long it would be.) > > What do you think? @HyukjinKwon KVStore is not read-only after the rebuildAppStore() is done. In core/src/main/scala/org/apache/spark/status/AppStatusStore.scala#357, kvstore writes CachedQuantile. AFAIK, this is the only type of objects that can be written to kvstore after the loading is done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-624907362 @tgravescs What I saw in FsHistoryProvider is that "spark.history.fs.numReplayThreads" is used to create a thread pool to mergeApplicationListing and compact. These threads seem are not responsible for parsing a complete single event. But I am not sure if my understanding is correct. And In hybrid kvstore, each instance has one writing thread to write data to leveldb. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-624445143 Another way is to keep a thread-safe variable called availableMemory in FsHistoryProvider. The initial value can be set as a percentage of Xmx. When we parse a file via hybrid kvstore, we subtract an approximate memory usage from availableMemory, and when the hybrid store switches to leveldb, we add back this approximate memory usage. When availableMemory is below a threshold, we can disable hybridKVstore. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-624439430 @HeartSaVioR One way in my mind is that we can monitor the memory usage of SHS. If the memory usage or event log size exceeds a threshold (e.g, over 50% of Xmx), we can use leveldb to parse event log, instead of hybrid kvstore. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] baohe-zhang commented on pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster
baohe-zhang commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-621536233 @jiangxb1987 from my testing result, loading 1g file, HybridKVStore takes 23s to parse (that means, users only need to wait for 23s to see the UI), LevelDB takes 69s. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
