Hello,
I would like to start a discussion about the future of graphs and reporting
in Apache JMeter.
Todays state:
- My feeling (I don't want to hurt anybody who worked on those listeners
which are interesting and I am aware that what can be great at some time
can become less few years after) is that today:
- we have the following graphs
- Aggregate Graph : currently affected by bug on
https://issues.apache.org/bugzilla/show_bug.cgi?id=57221 in Java 8
- Response Time Graph : interesting but slow to generate and I get
OOM when I try to reload an existing CSV in some configuration, I will
report a bug
- Distribution Graph (alpha) : Alpha is not encouraging for users,
is Alpha really needed or is it stable
- Graph Results : Does not look very nice compared with what you
can get today
- we have the following reports:
- Aggregate Report : Interesting
- Summary Report: Differs from Aggregate Report by having
additional Avg Bytes and Std Dev but not Median nor 90%Line
My opinion and analysis on the current state is the following:
- We lack of clean reporting that would be generated automatically at
end of load test, all the previous graphs need to be generated from GUI and
makes you lose some time. Also as we discourage Load Testing from GUI we
should propose a way to generate reporting from NON GUI Mode
- Some graphs (Response Time Graph, Aggregate Graph) rely on JCharts
which seems to be abandoned, last release is from 2004 and current bug
57221 may be related to a bug in the library ,by the way this is an
additional argument to get rid of obsolete libraries in JMeter:
- Avalon (Logging + DataSource)
- Excalibur (logging + DataSource)
- jCharts
- Reports are not sufficient and partly redundant , for example it
seems that in terms of information Aggregate report and Summary Report
should be merged. I looked at their code they are really very similar , I
think one should be removed and the infos in both merged
I must reckon that today my colleagues and I for example , we all use
JMeter-Plugins graphs combined with GraphGeneratorListener, and we are very
happy with what we get:
-
http://www.ubik-ingenierie.com/blog/automatically-generating-nice-graphs-at-end-of-your-load-test-with-apache-jmeter-and-jmeter-plugins/
I would be interested to know how users generate their reports and graphs
currently.
But some may object that JMeter-Plugins is not JMeter and conclude that
JMeter lacks of an essential feature.
So I have 2 propositions to improve JMeter in this field:
- 1/ Create a Brand new Report Generator Listener that would at end of
load test generate a report in HTML or PDF containing:
- Merge of informations contained in current reports
- Generate graphs based on JS Libraries, I am impressed by the number
of libraries that generate nice and sexy graphs:
- http://www.chartjs.org/
- https://github.com/keen/dashboards/tree/gh-pages/examples
- Having them in HTML would allow some dynamice behaviour but the
ideal result would be to be able to export graphs also as images
- 2/ I don't know if it is possible but what about merging
JMeter-Plugins graphing suite + GraphGeneratorListener in JMeter core if no
library is in conflict in terms of licensing, of course it would required
their agreement :-)
I think today any Load Testing tool comes with reporting feature wether
Commercial and Open Source and JMeter as such should improve in this field
to stay one of the leaders in Load Testing field.
Thoughts ?
--
Regards
Philippe M.
@philmdot
