Hi,
On 20.6.2019 22.26, Eric Anholt wrote:
Hey folks, I wanted to show you this follow-on to shader-db I've been
working on:
https://gitlab.freedesktop.org/anholt/renderdoc-traces
For x86 development I've got a collection of ad-hoc scripts to capture
FPS numbers from various moderately interesting open source apps so I
could compare-perf them. I was only looking at specific apps when they
seemed relevant, so it would be easy to miss regressions.
Starting work on freedreno, one of the first questions I ran into was
"does this change to the command stream make the driver faster?". I
don't have my old set of apps on my debian ARM systems, and even less so
for Chrome OS. Ultimately, users will be judging us based on web
browser and android app performance, not whatever I've got laying around
on my debian system. And, I'd love to fix that "I ignore apps unless I
think of them" thing.
So, I've used renderdoc to capture some traces from Android apps. With
an unlocked phone, it's pretty easy. Tossing those in a repo (not
shared here), I can then run driver changes past them to see what
happens. See
https://gitlab.freedesktop.org/mesa/mesa/merge_requests/1134 for some
results.
Where is this repo going from here?
- I add a runner for doing frame-to-frame consistency tests. We could
catch UB in a lot of circumstances by replaying a few times and making
sure that results are consistent. Comparing frames between drivers
might also be interesting, though for that you would need human
validation since pixel values and pixels lit will change on many
shader optimization changes.
When bisecting, ezBench[1] lists all perf and rendering changes, and
provides visual diff for *all* the rendering changes. Without latter,
something like this would have been really hard to notice & bisect:
https://bugs.freedesktop.org/show_bug.cgi?id=103197
Or something like this in a moving part of a long benchmark:
https://bugs.freedesktop.org/attachment.cgi?id=134878
https://bugs.freedesktop.org/attachment.cgi?id=134879
As to providing info on how likely driver is to be rendering something
wrong / differently instead of differences being just due to accuracy /
calculations order... Maybe something like PSNR would be useful:
https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio
?
- Eero
[1] https://github.com/freedesktop/ezbench
- Need to collect more workloads for the public repo:
- I've tried to capture webgl on Chrome and Firefox on Linux with no
luck. WebGL on ff is supposed to work under apitrace, maybe I could
do that and then replay on top of renderdoc to capture.
- Mozilla folks tell me that firefox's WebRender display lists can be
captured in browser and then replayed from the WR repo under
apitrace or rendredoc.
- I tried capturing Mozilla's new Pathfinder (think SVG renderer), but
it wouldn't play the demo under renderdoc.
Do you have some apps that should be represented here?
- Add microbenchmarks? Looks like it would be pretty easy to grab
piglit drawoverhead results, not using renderdoc. Capturing from
arbitrary apps expands the scope of the repo in a way I'm not sure I'm
excited about (Do we do different configs in those apps? Then we need
config infrastructure. Ugh).
- I should probably add an estimate of "does this overall improve or
hurt perf?" Yay doing more stats.
- I'd love to drop scipy. I only need it for stats.t.ppf, but it
prevents me from running run.py directly on my targets.
_______________________________________________
mesa-dev mailing list
mesa-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/mesa-dev
_______________________________________________
mesa-dev mailing list
mesa-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/mesa-dev
