Francesco Mari commented on OAK-6209:

[~maksim_kviatkouski], I just finished to have a look at your patch. I have a 
couple of comments, I hope we can discuss them together. Forgive my verbosity!

The work around {{BenchmarkOutputStrategy}} makes a lot of sense. I would like 
to propose the following changes. I noticed that {{AbstractTest}} uses both an 
{{outputStrategy}} and an optional {{csvStrategy}}, which is set only when a 
CSV file must be generated. What do you think about implementing a 
{{CompositeBenchmarkOutputStrategy}}, in the same spirit as 
{{o.a.j.o.spi.commit.CompositeHook}}? This way you can avoid chaining the calls 
of {{outputStrategy}} and {{csvStrategy}}, remembering to always wrap the 
latter in an if statement. Actually, you could get rid of the if statement 
altogether by creating a dummy implementation of {{BenchmarkOutputStrategy}}, 
whose methods do nothing. This way, if no CVS file is needed, you can simply 
pass the dummy implementation instead of handling special cases because of 
null. You probably already guessed that I took inspiration from 

In order for {{BenchmarkOutputStrategy}} to be reusable across test 
invocations, what do you think about having its methods accept an 
{{AbstractTest}} parameter? This way you could instantiate the strategy (or 
strategies) once during the setup of the benchmark runner, and use it for every 
following benchmark run. The code in {{AbstractTest}} can simply call the 
strategy as {{strategy.doSomething(this)}}.

While I understand your needs, I was a bit daunted at the amount of changes 
necessary to get rid of the usages of {{System.out}}. What about redirecting 
the standard output stream by using {{System#setOut}}? During the benchmark 
runner setup, if the {{report}} option is not specified, you can simply create 
a dummy {{OutputStream}} that discards all the output, create a new 
{{PrintStream}} based on that {{OutputStream}}, and pass that {{PrintStream}} 
to {{System#setOut}}. This way you don't need {{FilterPrinter}} and avoid 
tedious changes in every benchmark.

> The benchmark runner should produce machine-friendly output
> -----------------------------------------------------------
>                 Key: OAK-6209
>                 URL: https://issues.apache.org/jira/browse/OAK-6209
>             Project: Jackrabbit Oak
>          Issue Type: Improvement
>          Components: benchmarks
>            Reporter: Francesco Mari
>            Assignee: Francesco Mari
>            Priority: Major
>         Attachments: oak-6209.patch, sample-machine-readable-output.txt
> The benchmark runner currently produce output in the following format.
> {noformat}
> Apache Jackrabbit Oak 1.8-SNAPSHOT
> # LoginTest                        C     min     10%     50%     90%     max  
>      N 
> Oak-Segment-Tar                    1     472     494     522     552     631  
>    115
> # LoginLogoutTest                  C     min     10%     50%     90%     max  
>      N 
> Oak-Segment-Tar                    1     472     479     513     543     568  
>    118
> {noformat}
> While this format is well formatted and easy to read, it's a pain to process 
> with standard command line utilities. The benchmark runner should give the 
> possibility to produce machine-friendly output, like the following.
> {noformat}
> LoginTest,Oak-Segment-Tar,1,472,494,522,552,631,115
> LoginLogoutTest,Oak-Segment-Tar,1,472,479,513,543,568,118
> {noformat}

This message was sent by Atlassian JIRA

Reply via email to