[
https://issues.apache.org/jira/browse/SOLR-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038504#comment-16038504
]
Ishan Chattopadhyaya commented on SOLR-10317:
---------------------------------------------
bq. The reason behind me extending upon your framework is that it already has
many flexible, ready to use resources and that it is written in one language. I
am comfortable using one language over two languages together.
Thanks for the clarification. Exactly what I was looking for, i.e. *reasons why
you chose* the current suite over Shalin's suite. Personally, I don't care
whichever suite you are using so long as your stated goals are met, and we
achieve parity with whatever prior work exists already, and the overall suite
is flexible enough to add more benchmarks later.
bq. the closest that I am in making it dynamic and self-dependent is showing
relevant commit messages with each metric point
http://212.47.227.9/prod/NumericQueryBenchmarkStandalone.html (please hover
over any point to see the relevant commit message).
Maintaining a separate JSON file containing significant commit and message
sounds like a good way forward. Is it too difficult to plot that info on the
graphs, like it is done in Shalin's or Mike's graphs?
bq. I will try to add a feature through which you would be able to view all the
graphs together.
Sounds good.
Btw, Shalin's suite has got some tests that I didn't see in your suite or
proposal. I know that you've plotted some metrics on a per commit basis in a
popup window for indexing (memory consumption over the course of indexing), but
having an independent graph on GC while indexing and other similar graphs that
Shalin has added to that suite would be good. Don't stretch yourself for them
right now, but it would be awesome if you can add them at the end of your GSoC
project, for the sake of completeness of the suite.
[~shalinmangar], [~michael.sun], any thoughts, please?
> Solr Nightly Benchmarks
> -----------------------
>
> Key: SOLR-10317
> URL: https://issues.apache.org/jira/browse/SOLR-10317
> Project: Solr
> Issue Type: Task
> Reporter: Ishan Chattopadhyaya
> Labels: gsoc2017, mentor
> Attachments: changes-lucene-20160907.json,
> changes-solr-20160907.json, managed-schema,
> Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks.docx,
> Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks-FINAL-PROPOSAL.pdf,
> solrconfig.xml
>
>
> Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be
> found here, https://home.apache.org/~mikemccand/lucenebench/.
> Preferably, we need:
> # A suite of benchmarks that build Solr from a commit point, start Solr
> nodes, both in SolrCloud and standalone mode, and record timing information
> of various operations like indexing, querying, faceting, grouping,
> replication etc.
> # It should be possible to run them either as an independent suite or as a
> Jenkins job, and we should be able to report timings as graphs (Jenkins has
> some charting plugins).
> # The code should eventually be integrated in the Solr codebase, so that it
> never goes out of date.
> There is some prior work / discussion:
> # https://github.com/shalinmangar/solr-perf-tools (Shalin)
> # https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md
> (Ishan/Vivek)
> # SOLR-2646 & SOLR-9863 (Mark Miller)
> # https://home.apache.org/~mikemccand/lucenebench/ (Mike McCandless)
> # https://github.com/lucidworks/solr-scale-tk (Tim Potter)
> There is support for building, starting, indexing/querying and stopping Solr
> in some of these frameworks above. However, the benchmarks run are very
> limited. Any of these can be a starting point, or a new framework can as well
> be used. The motivation is to be able to cover every functionality of Solr
> with a corresponding benchmark that is run every night.
> Proposing this as a GSoC 2017 project. I'm willing to mentor, and I'm sure
> [~shalinmangar] and [[email protected]] would help here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
