Hi Gabe, As Jason said, some attempts at creating an output database went uncommitted. A few years ago (2012, apparently…!) I worked on updating Nate’s Python stats patches to allow the use of an SQLite database. As you can guess, it was never committed. However, the patches can still be found here:
http://reviews.gem5.org/r/1638/ http://reviews.gem5.org/r/1639/ http://reviews.gem5.org/r/1640/ http://reviews.gem5.org/r/1641/ http://reviews.gem5.org/r/1642/ http://reviews.gem5.org/r/1644/ http://reviews.gem5.org/r/1645/ http://reviews.gem5.org/r/1646/ http://reviews.gem5.org/r/1647/ WARNING: these patches are all very out of date at this point. A few key points about the above: * Python-based stats were many orders of magnitude slower than any C++ implementation * SQLite stats were significantly faster than text-based stats (both in Python) * Even if control flow remains in python, I advise doing as much of the grunt work in a C++ as possible * I created four tables for storing the stats based on the types of stats to try and avoid storage overhead. No longer sure if this makes sense or not. * “stats” – the “metadata” (name, description, etc.). I only stored the metadata about each stat once * “scalarValue” – the values of scalars * “vectorValue” – the values of vectors * “distValue” – the values of distributions * I bundled the SQL transactions into a single transaction per a stats dump * Significantly reduced overhead! * There are no stats filtering mechanisms in this implementation, but file sizes were significantly smaller for large numbers of stats dumps * Single dumps were significantly larger * I had created a small test framework which could be used to test the database output. Quite useful for debugging/as a regression. 1. Dump both SQL and text stats for a simulation 2. Inject the SQL stats back into the gem5 stats system and produce a text stats dump 3. Compare the original text output to the regenerated text output I do agree with Jason – it would be great to have this as a standard thing in gem5! Ideally with a much better implementation that my one. Please let me know if you have any questions for me regarding my attempt at creating a gem5 stats database. Thanks, Sascha On 16/09/2017, 00:25, "gem5-dev on behalf of Jason Lowe-Power" <[email protected] on behalf of [email protected]> wrote: Hi Gabe, There's currently no functionality for that. However, I believe there have been some efforts in the past to get something like this to work, though none of them were committed. I've heard that there may be some people working on better stats support. So, if you're thinking about improving the stats I would make sure that there aren't duplicate efforts :). Personally, I would *love* to see something like database support. Pandas (http://pandas.pydata.org/) would be even cooler, IMO. Cheers, Jason On Fri, Sep 15, 2017 at 3:42 PM Gabe Black <[email protected]> wrote: > Hi folks. This may be documented somewhere already, but is there a way to > collect stats to a database rather than to a text file? That would be > helpful when collecting stats periodically to get a graph over time, which > tends to produce a lot of output that needs to be processed before it's > useful. > > Gabe > _______________________________________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/gem5-dev _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
