I put some more thought into this. Given that we only store a month's worth of data, it's not worth doing backups. Keeping around all the data (maybe a year's worth?) would be awesome though. I actually think that would not be too much work and would add value to the dashboard. At that point, doing backups seems more worthwhile. I'm happy to walk someone through how to make this happen. It really would not be a lot of work if you have a workable knowledge of Python and JS.
Ojan On Wed, Oct 14, 2009 at 5:15 PM, Nicolas Sylvain <nsylv...@chromium.org>wrote: > On Wed, Oct 14, 2009 at 4:53 PM, Ojan Vafai <o...@google.com> wrote: > >> The data is stored in a single file per bot. For example, the webkit >> release bot's results are at >> http://build.chromium.org/buildbot/layout_test_results/webkit-rel/results.json. >> That >> file holds all the historical data for that bot and is copied over during >> the archive step of each run. We intentionally limit the number of results >> we keep in that file to 750 runs to keep filesize down. In my accidental >> checking, I changed 750 to 9. :( > > A little bit unrelated: This data, along with all the data on > build.chromium.org, is replicated on at least 4 machines. It would be easy > to recover the data if the server dies for example. We are also planning to > do daily backups, but the data is huge. For example, we archive 25GB of new > layout test results every day. > > Nicolas > > >> A trivial to implement "backup" would be to also copy the file to the >> archive location for just that run (same place as where we copy >> layout_test_results.zip), e.g. also copy it to >> http://build.chromium.org/buildbot/layout_test_results/webkit-rel/29056/. >> The downside is that this uses up disk space (e.g. the largest results.json >> file was 25mb before being clobbered). >> >> Another problem with backing up is that you'd also have to find a way to >> restore from backup that didn't lose data from runs that happened since the >> problem occurred. Merging the two files results.json should be pretty >> relatively trivial code, but it's all code that someone would need to write >> and test. >> >> While it sucks, I don't think backing up this data is worth the effort. >> It's a temporary productivity hit for the team, but we get enough new data >> to make reasonable decisions relatively quickly. Mistakes like this are very >> rare. It will become even more rare as coding work on the dashboard winds >> down. >> >> Feel free to have at it if you disagree. >> >> Creates the results.json file and it's content: >> trunk/src/webkit/tools/layout_tests/layout_package/json_results_generator.py >> Copies the results.json file to the right >> place: >> trunk/tools/buildbot/scripts/slave/chromium/archive_layout_test_results.py >> >> Ojan >> >> >> On Wed, Oct 14, 2009 at 4:24 PM, Jeremy Orlow <jor...@chromium.org>wrote: >> >>> I haven't actually gotten anything done on LocalStorage this week because >>> I've been doing so many small side projects like this.....but if it's a >>> priority, sure. >>> How about a cron job on some machine that ssh's via a cert into whatever >>> machines the data is stored on, pulls it back, and dumps it into some dir? >>> When we start filling up the hard drive, we can look at doing something >>> smarter, deleting old data, or putting it somewhere like GFS. >>> >>> What server can I use and where's the data stored? >>> >>> On Wed, Oct 14, 2009 at 4:17 PM, Evan Martin <e...@chromium.org> wrote: >>> >>>> Sounds like we've got a volunteer! :D :D :D >>>> >>>> On Wed, Oct 14, 2009 at 4:15 PM, Jeremy Orlow <jor...@chromium.org> >>>> wrote: >>>> > I assume we're going to start backing this data up from now on? >>>> > >>>> > On Wed, Oct 14, 2009 at 4:13 PM, Peter Kasting <pkast...@google.com> >>>> wrote: >>>> >> >>>> >> On Wed, Oct 14, 2009 at 3:58 PM, Ojan Vafai <o...@google.com> wrote: >>>> >>> >>>> >>> I accidentally checked in some test code (one number was wrong!) and >>>> >>> clobbered all but 10 of the runs of data for each builder. There's >>>> no way to >>>> >>> recover it. >>>> >> >>>> >> Do you moonlight for the Danger team at Microsoft? >>>> >> PK >>>> >> >>>> > >>>> > >>>> > > >>>> > >>>> >>> >>> >> >> >> >> > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
