This is my fourth attempt at addressing Zooko's darcs-benchmark problems.
This time, I've managed to artificially induce darcs to produce --exact-version
output we can't parse (I don't know what's causing his exact situation, so I
replaced the context string in a Version.hs with "blah"). Having done so, I
was able to reproduce the pattern match error he reported and also confirm that
with this patch, darcs-benchmark produces a report.
So I can't guarantee that this is the magic patch that will fix all the problems
-- I may still have gotten something else wrong, or there may still be yet
another
bug we may encounter -- but I think this time having put a lot more effort into
verifying my work, I can at least hope to myself that I have acted responsibly
the once...
The cautionary tale
-------------------
Sigh, sometimes you lead by example...
and other times you "lead" by counter-example :-(
0. Zooko reports an error trying to use the darcs-benchmark 0.1.9
we released. (bug A: the --converge feature tries to read content from
.darcs-benchmark, completely overlooking the possibility that
.darcs-benchmark exists)
1. I misdiagnose bug A (mistakenly thinking I had narrowed it down to the
only two places it could occur) as being something else, submit a patch
and notably (!), fail to try it out myself. [A] this sort of cutting
corners should be considered a form of disrespect for users -- you
think you're "saving" time, but what you're really doing is demanding
that somebody else give you theirs and [B] it's just generally foolish
and inefficient.
2. Zooko reports that the error persists, I think I'd learned my lesson,
and this time submit a patch having tested it out myself.
3. Zooko reports a new problem (bug B); I spot an easy solution, code it
up and submit a patch, but in my confusion/distraction OMIT to compile
it first (!). Now, this wasn't intentional -- I'm sure many other
Haskellers have "compile this please" hotkeyed to something they just
thwack automatically and unconciously -- but [C] allowing myself to work
in a sufficiently hurried/multi-tasky/distracted fashion that such mistakes
can occur was irresponsible. The more responsible approach would have been
just to postpone this until I felt I had the attention to deal with it
properly, or to ask if Petr could investigate as he is more familiar with
the recent darcs-benchmark developments.
Remember the sort "haste makes waste, slowly by surely" attitude I like to talk
about? That's why I talk about it. Focusing a little harder, and spending some
extra time to think about how to make it possible for me to recreate a scenario
similar to Zooko's would have saved him the effort of trying another iteration,
saved me the extra work of cleaning up after myself, the embarrassement of
sending one of these Haskell newbie never-seen-a-monad in your life bits of
code, the embarassement of being hoist on my own lead-by-example petard and so
on...
1 patch for repository http://code.haskell.org/darcs/darcs-benchmark:
Wed Sep 1 22:50:32 BST 2010 Eric Kow <ko...@darcs.net>
* Account for darcs with buggy --exact-version.
Treat these as having an empty context. :-(
New patches:
[Account for darcs with buggy --exact-version.
Eric Kow <ko...@darcs.net>**20100901215032
Ignore-this: 7a0bf59360dffa1f0c789af777445d9f
Treat these as having an empty context. :-(
] hunk ./source/Benchmark.hs 178
check_darcs :: String -> IO TestBinary
check_darcs cmd = do
version <- darcsMeta cmd ["--version"]
- [info, context] <- splitOn "Context:\n\n" `fmap` darcsMeta cmd ["--exact-version"]
+ (info, context) <- info_and_context `fmap` darcsMeta cmd ["--exact-version"]
rts <- read `fmap` darcsMeta cmd ["+RTS", "--info"]
let date' = case info of
_ | "darcs compiled on" `isPrefixOf` info ->
hunk ./source/Benchmark.hs 195
2:_ -> return bin
_ -> fail $ cmd ++ ": Not darcs 2.x binary."
+info_and_context :: String -> (String, String)
+info_and_context s =
+ case splitOn "Context:\n\n" s of
+ [i,c] -> (i,c)
+ [i] -> (i,"")
+ _ -> error "Bug. Either we can split on Context or we can't"
+
verbose :: String -> Command ()
verbose m = liftIO $ do loud <- isLoud
when loud $ UTF8.hPutStrLn stderr m
Context:
[Create results directory earlier.
Eric Kow <ko...@darcs.net>**20100831085733
Ignore-this: a0222214e0df65c0fa7a22ccd1a4b3ee
]
[Allow timings directory to not exist when reading from it.
Eric Kow <ko...@darcs.net>**20100831175225
Ignore-this: c309699ed386c2c4684b6edce6668c9f
This chokes when checking reading back the timings data for
convergence.
]
[TAG 0.1.9
Eric Kow <e.y....@brighton.ac.uk>**20100808142244
Ignore-this: 7aca7d771831140fd54c6160f63abd36
]
[NEWS file.
Eric Kow <e.y....@brighton.ac.uk>**20100808142225
Ignore-this: b7fcc9795803b2bd4d0308e693e12ce2
]
[Slightly friendlier output for run --converge.
Eric Kow <e.y....@brighton.ac.uk>**20100808142143
Ignore-this: b1d82295a26f83f69b98841b28da2a44
In the case where there are no more benchmarks to run for a repo.
]
[Do not fail if ~/.darcs-benchmark does not exist.
Petr Rockai <m...@mornfall.net>**20100808140546
Ignore-this: 8815958b6493c54e1db76669f2a61d7
]
[Handle errors more gracefully.
Petr Rockai <m...@mornfall.net>**20100808021248
Ignore-this: 518d0816137dbaa66073a5ac27852a46
]
[Restore compatibility with old (nonhashed) dumpfiles.
Petr Rockai <m...@mornfall.net>**20100808020228
Ignore-this: d052d761bb8ee0d8cc95262efab13231
]
[Better error reporting than "Prelude.read: no parse".
Petr Rockai <m...@mornfall.net>**20100808014830
Ignore-this: 7793eb02fb7c6429d59d41819e706b71
]
[Parse ghc/hg --version output more robustly.
Petr Rockai <m...@mornfall.net>**20100808014811
Ignore-this: a5f87089d5ec93df6ad0d46a885c348c
]
[Use obliterate everywhere within Standard.hs.
Petr Rockai <m...@mornfall.net>**20100808013954
Ignore-this: abe1c0bda0da8c06224510fe0389c10e
]
[Make the version parser more robust.
Petr Rockai <m...@mornfall.net>**20100808013927
Ignore-this: 2835e395c53bf1572399e9702452db8c
]
[Move source files under source/.
Petr Rockai <m...@mornfall.net>**20100807123911
Ignore-this: b963f0be2816f555fa30cf77ed95fbad
]
[Ignore editor backups when looking for config.* files.
Petr Rockai <m...@mornfall.net>**20100807123523
Ignore-this: ccc4df9d639d71003f0a621491473cae
]
[Drop tailor support, no longer needed.
Petr Rockai <m...@mornfall.net>**20100807122510
Ignore-this: fd13704c4e987e2232d1c869e35e86a5
]
[Use darcs convert --export / git fast-import / hg convert instead of tailor.
Petr Rockai <m...@mornfall.net>**20100807122407
Ignore-this: 6fc88270fedcbe96a05adbb1a291eae7
]
[Use darcs convert --export to go from darcs to git.
Petr Rockai <m...@mornfall.net>**20100807111553
Ignore-this: ea32ddf86a318666c09732d667b5beeb
]
[Add support for running git & hg benchmarks.
Petr Rockai <m...@mornfall.net>**20100807022236
Ignore-this: 43aaff919df6ffaf3a0e13d7be466bf5
]
[Add a missed import of Data.Char.
Petr Rockai <m...@mornfall.net>**20100807022157
Ignore-this: 48d9149968fd56f409656af1657ef872
]
[Rework the exceptions in Shellish & co.
Petr Rockai <m...@mornfall.net>**20100807022131
Ignore-this: e4a95441fc932e80283bcd7ebd91fa67
]
[Undo a mistaken change of cmdargs to cmdlib in cabal.
Petr Rockai <m...@mornfall.net>**20100807022109
Ignore-this: f9f7cc63a4770e1975c33741190e555a
]
[Disambiguate different binaries with same name in reports.
Petr Rockai <m...@mornfall.net>**20100806235443
Ignore-this: fdca4a24d7f491a2552414435b04918
]
[Keep track of exact versions of test binaries.
Petr Rockai <m...@mornfall.net>**20100806232955
Ignore-this: 49bfd3d2330b7635a6faf1e7afaff585
]
[Improve tailor conversion resilience a bit.
Petr Rockai <m...@mornfall.net>**20100728214514
Ignore-this: 5913d1414c4f202ee310c284e25ca9c7
]
[Add a "git" repo variant, created using tailor.
Petr Rockai <m...@mornfall.net>**20100728102530
Ignore-this: 46369030ab8e4ba2409a43bd85d6d8bf
]
[Remove extra "mod" bits from benchmark names.
Petr Rockai <m...@mornfall.net>**20100728092113
Ignore-this: 19c27c7c411834b57632fbf59481a31
]
[Do not leak file descriptors due to lazy readFile.
Petr Rockai <m...@mornfall.net>**20100726095831
Ignore-this: 95b1e2748271ada4e5fe89793c2c6667
]
[Bump version to 0.1.9.
Eric Kow <ko...@darcs.net>**20100731121722
Ignore-this: 93c3ce4b6bfb391305ccbd11271ae37f
]
[Add ability for a repo to specify tests to skip.
Eric Kow <ko...@darcs.net>**20100731121406
Ignore-this: 3ca82443a2e86062202ca8311fbcea4c
]
[Resolve #ead: was fixed in darcs-benchmark 0.1.8
Eric Kow <ko...@darcs.net>**20100731120101
Ignore-this: fb440fa86d9cbae2c857c91510183225
]
[Report timings as mean plus one standard deviation.
Eric Kow <ko...@darcs.net>**20100731114503
Ignore-this: 97a05bc0cb31cc33b21150f311775352
Quoting Zooko:
My intuition is that I'd rather have something that's sort of fast
almost every time than something that's really fast half the time and
slow half the time, even if the latter thing has a better mean.
Zed Shaw has also written a blog entry claiming that people will tend to notice
variance, so (my reading of his post), if something has a good mean time but
jumps around a lot, the worst case could stick in people's minds a lot more.
It doesn't make sense to report the worst case (because of outliers), but we
could at least use the stddev to nudge us a bit closer in that direction.
]
[Avoid a bug where we think we have sufficient data for tests we never ran.
Eric Kow <ko...@darcs.net>**20100731114457
Ignore-this: 90a341fdb8d9d1bd31bd16fc02ec79b2
]
[Correctly report when we don't run any benchmarks.
Eric Kow <ko...@darcs.net>**20100731114451
Ignore-this: 9c16a01de634b7cda7cfb7e7965672ca
]
[run --converge feature to only run benchmarks with insufficient data.
'Eric Kow <ko...@darcs.net>'**20100731100655
Ignore-this: 4a5ea5cef09c66a83d3464d6dc574fe0
This involves changing darcs-benchmark to support having a separate list
of benchmarks to run for each repository.
]
[Emphasise graphs in report.
'Eric Kow <ko...@darcs.net>'**20100729071939
Ignore-this: 270fc9800b7cca5e978d557ff40317b4
]
[TAG 0.1.8.3
Eric Kow <ko...@darcs.net>**20100725233519
Ignore-this: 22bca35e28147d3af7aa93c9c8d14719
]
Patch bundle hash:
0a3668233b74486e9b19b7b029bb97e6dd5888af
_______________________________________________
darcs-users mailing list
darcs-users@darcs.net
http://lists.osuosl.org/mailman/listinfo/darcs-users
