On 1/31/14 2:24 PM, Robin Sommer wrote: > (Moving from bro-commits to bro-dev). > > On Fri, Jan 31, 2014 at 12:51 -0500, you wrote: > >> Instruction counts are probably going to have a strong dependency on >> the compiler version / options used to generate the code. I believe >> these counts could additionally be influenced by e.g. library >> upgrades, even when restricted to a single host and using a specific >> compiler / options. > True, but I'm not sure that's necessarily a bad thing. I don't think it's a bad thing either, exactly. When I wrote this, I was thinking that it'd make it a little easier to interpret the results if code changes could be tested independently of system changes ... but you're right that the system *is* going to have an effect on how bro runs and should therefore be included in the benchmark.
I do think it's important to be able to isolate system-level vs. code-level changes, but just storing the commit should be enough to do that. I think this is already happening, so ... looks good to me. > What I'm mostly wondering about is if it's worth commiting data that's > very specific to a single user/machine to the repos? From an academic standpoint, think it could be useful to review these counts just to see how much they actually vary from platform to platform. Also, if the instruction counts *are* pretty consistent cross-platform, then it might e.g. be an indication that we / the compiler aren't taking advantage of some more complex instructions that exist on a particular platform. Additionally, given the number of people working on the project, this kind of data probably wouldn't take much space. It also seems like it'd be pretty easy to purge this data in the event it didn't end up being useful. --Gilbert _______________________________________________ bro-dev mailing list [email protected] http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
