Hi Karl,
thanks for the feedback.
> -advanced mode: reset (load default) benchmark settings button
> nice to have, but not a blocker :-)
Shouldn't be too hard to implement. I'll add it for the first release.
this is not as easy as it may seem. Even if the memory consumed by the
> benchmark is below the physical memory provided, the GPU memory may be
> occupied otherwise. This is particularly an issue with mobile GPUs attached
> to 4K displays, where the screen buffer occupies a decent amount of GPU
> RAM. I think the best we can do is to warn a user, e.g.
> "The benchmark configuration you provided exceeds XXX (256?) MB of GPU
> RAM. This may exhaust your available GPU RAM. Do you want to continue?
> [Continue]/[Cancel]"
A simple warning message is indeed a good solution. Added to my primary
TODO list.
> -vector benchmark progress update is slacking; it should be updated with
>> each completed step instead of going from 0% to 100% in one step (the
>> benchmarking loop needs to altered a little to fix this)
>>
>
> Fine with me. Can you fix this?
> Also, I noticed that a repeated reset of the x-axis leads to ugly fonts
> for the labels, so the axis should only be set once when starting the
> benchmark. See screenshot attached, including some other things...
>
A little rework in the benchmarking loop will fix the progress update bug.
Won't be an issue for me.
As for the x-axis resetting, new tick points are added with each result
entry which is why it looks so "jerky" while plotting. I'll make it so that
all tick points are added before the plotting begins. This should also
solve the font issue, since there won't be multiple tick labels stacking on
top of each other.
> -benchmarking with a custom sparse matrix now works with some matrices,
>> but not all; at least now it doesn't crash the program...
>>
>
> This is presumably due to the matrix market format. Matrices with scalar
> type 'complex' or with pattern type 'pattern' are not supported. Can you
> add an error message stating this? Something like "The matrix market reader
> cannot read the provided file. Only real-valued matrices in
> coordinate-format are supported."
>
Does the matrix reader provide different error codes for different reading
errors? I see that it prints "supports only real valued floating point
arithmetic" to the console, so I should cover all possible error types if
there's more than one.
> -"my results" screen needs significant improvements:more benchmark data
>> should be shown, auto-refresh when a result gets saved, enable/disable
>> result saving, delete selected/all results, add user notes, maybe a
>> dynamic filter to search through result notes...
>>
>
> Can we disable the 'my results' screen for the first release? I think it
> requires too much work *now*, but can be picked up any time later.
>
>
> -result uploading: it's completely missing
>>
>
> Disable that for the 1.0.0 release. The infrastructure is not yet ready -
> in particular, result uploads are only interesting once we have autotuning
> features in the GUI.
>
>
> -"result database" screen: also completely missing
>>
>
> Disable :-)
>
Roger that.
> -"system info" screen feels kinda weird. Is it just me or maybe it could
>> use some improvements?
>>
>
> Well, at least it is informative. ;-)
> It would be great if you can make it a bit more appealing, but visual
> appearance is not a top priority here I'd say...
Agreed. Moved to low priority, but I'll still keep it in my head in case I
think of something.
> -matrixmarket: reconnect the "download & run" functionality now that
>> benchmarking with custom sparse matrices does not crash the program.
>> Also, show the achieved result inside the matrixmarket table. The idea
>> is to make it work like this: you find a nice matrix, hit download &
>> run, the program downloads the matrix, runs a benchmark with your
>> matrix, and shows the result next the "download&run" button. It would be
>> good if we could avoid switching to the "benchmark" screen during
>> execution, and make it so that users don't need to leave the
>> "matrixmarket" screen at all.
>>
>
> Hmm, I think it is interesting for the user to see which sparse matrix
> type works best for the particular sparse matrix. Unless we can provide the
> full results from the 'sparse' tab in the MatrixMarket, I'd rather refrain
> from it for now.
>
By "full results" do you mean those 7 tests that comprise the "sparse
benchmark" (HYB, ELL, CSR and others)? Showing 7 results instead of 1 isn't
a big a problem. I'd just have to play around the matrix table layout to
make 'em all fit properly.
> -some minor "face-lifting" procedures: Qt isn't very friendly when it
>> comes to customizing the looks of widgets. That's why some buttons and
>> tabs might look a bit wrong (depending on your platform). This isn't a
>> significant issue, but still needs to be looked at.
>>
>
> I fixed a couple of these issues by removing hard-wired margins and such.
> The home screen still has some layout issues, see screenshot.
About the hard-wired margins: when a result (bar graph) is too large to fit
on the plot, it goes out of bounds. Since the result value label is
"appended" to the end of the result bar, it goes out of bounds together
with the bar and cannot be displayed. The hard-wired margins were there to
ensure the labels will be displayed even if the result bar gets too big.
This kind of label rendering behavior is pretty weird if you ask me, but
that's how it is. Your solution to hard-code the axis range from 0.1 to
5000 should work for now, but the labels will disappear if the result gets
close to that 5k mark. But I guess that's not a big problem at the moment
since 5k is currently not reachable (or is it?).
As for the home screen layout issues and widget display bugs: like I said,
Qt turns into a spoiled brat as soon as you start messing with the looks of
its widgets. I can't say if it will be possible to make it look good on all
platforms. Lots of testing will be needed to ensure it looks as it should
on every platform... And that empty space in the home screen should be
definitely used for something, I just don't know what :D
> Some more thoughts (some are trivial to fix):
> - Result tabs: Can these tabs be renamed to "Dense
> Matrix-Matrix-Products", "Host-Device Copy", "Sparse Matrix-Vector
> Product", and "Vector Operations", respectively? I tried to do that, but it
> seems like the current strings have some further meaning for the program
> control flow (fix this?)?
You mean the detailed test results tab names in the bottom? I don't recall
their strings being important to the program flow control. Maybe I
hard-coded something for testing purposes but forgot to fix it afterwards...
- I think it makes more sense for the BLAS 3 benchmark to scan through
> different matrix sizes, rather than using a fixed size. The benchmark just
> stops when a kernel run starts to take more than, say, a second. This
> ensures that the benchmark does not get stuck on mobile hardware, which is
> far slower than high-end desktop hardware and for which we may not have
> good kernels yet. I'll take care of this.
Hmm, keep in mind that we need still need to make a "standardized"
benchmark configuration for the standard benchmark mode. Using different
matrix sizes for different hardware is not a very "standardized" thing to
do.
- Results summary: Rather than providing an average, I think that taking
> the best results makes more sense. For example, for the sparse bench a user
> is really not interested in how many FLOPs are obtained on average, but how
> many FLOPs the fastest matrix type provides. Also an easy fix :-)
Heh, originally I made it so that the best result is shown in the "result
summary" graph. But then you said it might be better to use median values
so I went with that :) Twas probably a misunderstanding, very easy to fix =)
The final 'big TODO' is, of course, to provide a reasonably automated
> packager to ship executables for Windows and Linux. I'm less worried about
> Linux, because OpenCL can typically be found in a system path. However,
> getting this to work with Windows might be quite tricky, since a user might
> have the OpenCL lib installed at whatever path. Either way, I reserved the
> weekend for this, so we should be able to release early next week. (I can't
> postpone the release, I'm totally filled with other work until the end of
> the year then.)
Noob question: can't we package those OpenCL libs with our program, or are
those libs specific to each user's CPU/GPU manufacturer? What I'm trying to
ask is this: do users need to have a specific OpenCL SDK installed (for
example, I had to install AMD APP SDK separately for development purposes),
or is there a "general purpose" OpenCL lib that anyone can use with any
hardware and that also happens to be easily package-able?
As for an automated packager, we could use Qt Installer Framework
<http://qt-project.org/doc/qtinstallerframework-1.4/index.html>. I've never
used before, but considering the good documentation I don't think it'll be
a problem.
Regards,
Namik
On Tue, Nov 18, 2014 at 10:46 AM, Karl Rupp <[email protected]> wrote:
> Hi Namik,
>
> thanks for the list. I'll comment on the items inline, particularly with
> having the aim of a release in the near future in mind.
>
> > -advanced mode: reset (load default) benchmark settings button
>
> nice to have, but not a blocker :-)
>
>
> -advanced mode: it would be cool if we could show how much video memory
>> is required to complete the benchmark. This could be very useful when
>> playing with custom settings as we would know in advance if we set the
>> matrix size too high, thus preventing the program from crashing.
>>
>
> this is not as easy as it may seem. Even if the memory consumed by the
> benchmark is below the physical memory provided, the GPU memory may be
> occupied otherwise. This is particularly an issue with mobile GPUs attached
> to 4K displays, where the screen buffer occupies a decent amount of GPU
> RAM. I think the best we can do is to warn a user, e.g.
> "The benchmark configuration you provided exceeds XXX (256?) MB of GPU
> RAM. This may exhaust your available GPU RAM. Do you want to continue?
> [Continue]/[Cancel]"
>
>
> -which makes me think a certain "safety switch" could also be a nice
>> addition, one that would prevent users from starting a benchmark if the
>> settings are set too high.
>>
>
> I think the warning dialog above would take care of this. :-)
>
>
> -vector benchmark progress update is slacking; it should be updated with
>> each completed step instead of going from 0% to 100% in one step (the
>> benchmarking loop needs to altered a little to fix this)
>>
>
> Fine with me. Can you fix this?
> Also, I noticed that a repeated reset of the x-axis leads to ugly fonts
> for the labels, so the axis should only be set once when starting the
> benchmark. See screenshot attached, including some other things...
>
>
> -benchmarking with a custom sparse matrix now works with some matrices,
>> but not all; at least now it doesn't crash the program...
>>
>
> This is presumably due to the matrix market format. Matrices with scalar
> type 'complex' or with pattern type 'pattern' are not supported. Can you
> add an error message stating this? Something like "The matrix market reader
> cannot read the provided file. Only real-valued matrices in
> coordinate-format are supported."
>
>
> -"my results" screen needs significant improvements:more benchmark data
>> should be shown, auto-refresh when a result gets saved, enable/disable
>> result saving, delete selected/all results, add user notes, maybe a
>> dynamic filter to search through result notes...
>>
>
> Can we disable the 'my results' screen for the first release? I think it
> requires too much work *now*, but can be picked up any time later.
>
>
> -result uploading: it's completely missing
>>
>
> Disable that for the 1.0.0 release. The infrastructure is not yet ready -
> in particular, result uploads are only interesting once we have autotuning
> features in the GUI.
>
>
> -"result database" screen: also completely missing
>>
>
> Disable :-)
>
>
> -"system info" screen feels kinda weird. Is it just me or maybe it could
>> use some improvements?
>>
>
> Well, at least it is informative. ;-)
> It would be great if you can make it a bit more appealing, but visual
> appearance is not a top priority here I'd say...
>
>
> -matrixmarket: reconnect the "download & run" functionality now that
>> benchmarking with custom sparse matrices does not crash the program.
>> Also, show the achieved result inside the matrixmarket table. The idea
>> is to make it work like this: you find a nice matrix, hit download &
>> run, the program downloads the matrix, runs a benchmark with your
>> matrix, and shows the result next the "download&run" button. It would be
>> good if we could avoid switching to the "benchmark" screen during
>> execution, and make it so that users don't need to leave the
>> "matrixmarket" screen at all.
>>
>
> Hmm, I think it is interesting for the user to see which sparse matrix
> type works best for the particular sparse matrix. Unless we can provide the
> full results from the 'sparse' tab in the MatrixMarket, I'd rather refrain
> from it for now.
>
>
> -some minor "face-lifting" procedures: Qt isn't very friendly when it
>> comes to customizing the looks of widgets. That's why some buttons and
>> tabs might look a bit wrong (depending on your platform). This isn't a
>> significant issue, but still needs to be looked at.
>>
>
> I fixed a couple of these issues by removing hard-wired margins and such.
> The home screen still has some layout issues, see screenshot.
>
>
> Well, that's about all I can think of at the moment. As always, you're
>> welcome to comment on these points and/or propose new things to implement.
>>
>> P.S. If you would like me to change the visual appearance of certain UI
>> parts, please say so before I start "fixing" those weird buttons and
>> tabs. Even the smallest changes can be a pain when it comes to
>> customizing Qt widgets.
>>
>
> Some more thoughts (some are trivial to fix):
> - Result tabs: Can these tabs be renamed to "Dense
> Matrix-Matrix-Products", "Host-Device Copy", "Sparse Matrix-Vector
> Product", and "Vector Operations", respectively? I tried to do that, but it
> seems like the current strings have some further meaning for the program
> control flow (fix this?)?
>
> - I think it makes more sense for the BLAS 3 benchmark to scan through
> different matrix sizes, rather than using a fixed size. The benchmark just
> stops when a kernel run starts to take more than, say, a second. This
> ensures that the benchmark does not get stuck on mobile hardware, which is
> far slower than high-end desktop hardware and for which we may not have
> good kernels yet. I'll take care of this.
>
> - Results summary: Rather than providing an average, I think that taking
> the best results makes more sense. For example, for the sparse bench a user
> is really not interested in how many FLOPs are obtained on average, but how
> many FLOPs the fastest matrix type provides. Also an easy fix :-)
>
> The final 'big TODO' is, of course, to provide a reasonably automated
> packager to ship executables for Windows and Linux. I'm less worried about
> Linux, because OpenCL can typically be found in a system path. However,
> getting this to work with Windows might be quite tricky, since a user might
> have the OpenCL lib installed at whatever path. Either way, I reserved the
> weekend for this, so we should be able to release early next week. (I can't
> postpone the release, I'm totally filled with other work until the end of
> the year then.)
>
> Best regards,
> Karli
>
>
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
ViennaCL-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/viennacl-devel
