Thanks for the survey Peter, my answers are below:
> 1. how often do you run the automated tests?
Nightly.
> 2. how do you structure the files?
Just a filesystem hierarchy, with directory names .
> 3. do you use a naming convention for the directories and files?
Directory names specifying successively narrower functional areas. For
eg:
B2B/xCBL3.0/inbound_PO/inbound-xcbl-po.jmx
B2B/xCBL3.0/outbound_PO/outbound-xcbl-po.jmx
B2B/xCBL2.0/outbound_POA/...
B2B/cXML1.2/...
> 4. how many test plans do you run?
About 35 JMXs in all. We run test plans in two modes.
Functional/integration test mode: all 35 tests are run consequtively,
with their thread group parameters set for 1 thread.
Load test mode: 15 generic tests of the 35 are run with a higher number
of threads. They are run simultaneously (a Perl process forks and execs
individual JMeter instances for this) - this delivers effective system
testing.
> 5. which listeners do you use to save the results
None - just parse the log file.
> 6. what kinds of charts and graphs do you want?
6.1. Stores result data long term, and present it through a
convenient interface (for eg: a JUnit style HTML interface). An ability
to drill down to specific test failures (tests would automatically save
the full request and response on failure) would be very convenient.
6.2. Interface has drill-down links to additional data (eg: debug
data captured on test failure)
6.3. Lets users to analyse results further. For eg:
- Differences in same test that ran against different
systems
- Failure rates for software components
- Variances in test runtimes over a long time period
(months)
6.4. A facility to annotate/comment test results would be a plus.
Some related ideas:
--- perhaps it would be worth having a 'results perspective' in JMeter,
or in a HTML reporting tool - the test results would be displayed nested
against a JMX tree that users can click to expand (somewhat like 'View
Results Tree' but with the comments in samplers)
--- Optionally export test report data to Openoffice for slicing and
dicing. Openoffice 2.0 apparently now ships with HSQLDB - a high
performance a Java object DB. This may come in handy for persisting test
results. It also ships with a "minimal webserver", which could be useful
for serving reports.
>From http://hsqldb.org/:
-------------------------
HSQLDB is the leading SQL relational database engine written in Java. It
has a JDBC driver and supports a rich subset of ANSI-92 SQL (BNF tree
format) plus SQL 99 and 2003 enhancements. It offers a small (less than
100k in one version for applets), fast database engine which offers both
in-memory and disk-based tables and supports embedded and server modes.
Additionally, it includes tools such as a minimal web server, in-memory
query and management tools (can be run as applets) and a number of
demonstration examples.
-------------------------
Sonam
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
