#28799: Use readr's read_csv() to speed up drawing graphs -----------------------------+-------------------------- Reporter: karsten | Owner: karsten Type: enhancement | Status: accepted Priority: Low | Milestone: Component: Metrics/Website | Version: Severity: Normal | Resolution: Keywords: | Actual Points: Parent ID: | Points: Reviewer: | Sponsor: -----------------------------+-------------------------- Changes (by karsten):
* status: merge_ready => accepted * priority: Medium => Low Old description: > Let's use R.cache to speed up drawing graphs. I already prepared a patch > that I'm going to post here as soon as I have a ticket number. From the > commit message: > > Over two years ago, in commit 1f90b72 from October 2016, we made our user > graphs faster by avoiding to read the large .csv file on demand. Instead > we read it once as part of the daily update, saved it to disk as .RData > file using R's save() function, and loaded it back to memory using R's > load() function when drawing a graph. > > This approach worked okay. It just had two disadvantages: > > 1. We had to write a small amount of R code for each graph type, which > is why we only did it for graphs with large .csv files. > 2. Running these small R script as part of the daily update made it > harder to move away from Ant towards a Java-only execution model. > > The new approach implemented in this commit uses R.cache, which caches > data for use by concurrent Rserve clients. The first time we read a .csv > file we save it to the cache, and all subsequent times we just load it > back from the cache. We're using the file name and last modified time as > key in the cache to avoid using stale data. We're also clearing the cache > on startup to avoid running out of disk space. > > One somewhat unwanted side effect is that drawing the first graph from a > new .csv file may take a few more seconds as compared to drawing > subsequent graphs. This seems acceptable, though. > > Requires installing the R.cache package from CRAN, which is available on > Debian as r-cran-r.cache. New description: Let's use R.cache to speed up drawing graphs. I already prepared a patch that I'm going to post here as soon as I have a ticket number. From the commit message: Over two years ago, in commit 1f90b72 from October 2016, we made our user graphs faster by avoiding to read the large .csv file on demand. Instead we read it once as part of the daily update, saved it to disk as .RData file using R's save() function, and loaded it back to memory using R's load() function when drawing a graph. This approach worked okay. It just had two disadvantages: 1. We had to write a small amount of R code for each graph type, which is why we only did it for graphs with large .csv files. 2. Running these small R script as part of the daily update made it harder to move away from Ant towards a Java-only execution model. The new approach implemented in this commit uses R.cache, which caches data for use by concurrent Rserve clients. The first time we read a .csv file we save it to the cache, and all subsequent times we just load it back from the cache. We're using the file name and last modified time as key in the cache to avoid using stale data. We're also clearing the cache on startup to avoid running out of disk space. One somewhat unwanted side effect is that drawing the first graph from a new .csv file may take a few more seconds as compared to drawing subsequent graphs. This seems acceptable, though. Requires installing the R.cache package from CRAN, which is available on Debian as r-cran-r.cache. '''Edit: Turns out that we don't want R.cache but readr's read_csv() instead. See comments below.''' -- Comment: Thanks for looking! Merged with a small [https://gitweb.torproject.org/user/karsten/metrics- web.git/commit/?h=task-28799-2&id=fecafc07b99798946308bbb3615c15bb0ce6a30f tweak], and deployed. Setting back to accepted for the remaining graphs after we gathered some more experience with this new approach. That could easily happen in 2019. -- Ticket URL: <https://trac.torproject.org/projects/tor/ticket/28799#comment:8> Tor Bug Tracker & Wiki <https://trac.torproject.org/> The Tor Project: anonymity online
_______________________________________________ tor-bugs mailing list tor-bugs@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs