Thanks Etienne, appreciate the info. This will help me a lot :) On Wed, Nov 28, 2018 at 1:02 AM Etienne Chauchot <echauc...@apache.org> wrote:
> Hi Alex, > Exporting results to the dashboards is as easy as writing to a BigQuery > table and then configure the dashboard SQL request to display it. Here is > an example: > - exporting: > https://github.com/apache/beam/blob/ad150c1d654aac5720975727d8c6981c5382b449/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/Main.java#L163 > - displaying: > > SELECT > DATE(timestamp) as date, > runtimeSec > FROM > [apache-beam-testing:nexmark.nexmark_0_DirectRunner_batch] > WHERE > timestamp >= TIMESTAMP_TO_SEC(DATE_ADD(CURRENT_TIMESTAMP(), -2, "WEEK")) > ORDER BY > date; > > Best > Etienne > > Le mardi 27 novembre 2018 à 17:34 -0800, Alex Amato a écrit : > > It would be great to add some lower level benchmark tests for the java > SDK. I was thinking of using open census for collecting benchmarks, which > looks easy to use should be license compatible. I'm just not sure about how > to export the results so that we can display them on the perfkit dashboard > for everyone to see. > > Is there an example PR for this part? Can we write to this data store for > this perfkit dashboard easily? > > https://github.com/census-instrumentation/opencensus-java > > https://github.com/census-instrumentation/opencensus-java/tree/master/exporters/trace/zipkin#quickstart > > > > > On Thu, Jul 19, 2018 at 1:28 PM Andrew Pilloud <apill...@google.com> > wrote: > > The doc changes look good to me, I'll add Dataflow once it is ready. > Thanks for opening the issue on the DirectRunner. I'll try to get some > progress on a dedicated perf node while you are gone, we can talk about > increasing the size of the nexmark input collection for the runs once we > know what the utilization on that looks like. > > Enjoy your time off! > > > Andrew > > On Thu, Jul 19, 2018 at 9:00 AM Etienne Chauchot <echauc...@apache.org> > wrote: > > Hi guys, > As suggested by Anton bellow, I opened a PR on the website to reference > the Nexmark dashboards. > As I did not want users to take them for proper neutral benchmarks of the > runners / engines, but more for a CI piece of software, I added a > disclaimer. > > Please: > - tell if you agree on the publication of such performance results > - comment on the PR for the disclaimer. > > PR: https://github.com/apache/beam-site/pull/500 > > Thanks > > Etienne > > > Le jeudi 19 juillet 2018 à 12:30 +0200, Etienne Chauchot a écrit : > > Hi Anton, > > Yes, good idea, I'll update nexmark website page > > Etienne > > Le mercredi 18 juillet 2018 à 10:17 -0700, Anton Kedin a écrit : > > These dashboards look great! > > Can publish the links to the dashboards somewhere, for better visibility? > E.g. in the jenkins website / emails, or the wiki. > > Regards, > Anton > > On Wed, Jul 18, 2018 at 10:08 AM Andrew Pilloud <apill...@google.com> > wrote: > > Hi Etienne, > > I've been asking around and it sounds like we should be able to get a > dedicated Jenkins node for performance tests. Another thing that might help > is making the runs a few times longer. They are currently running around 2 > seconds each, so the total time of the build probably exceeds testing. > Internally at Google we are running them with 2000x as many events on > Dataflow, but a job of that size won't even complete on the Direct Runner. > > I didn't see the query 3 issues, but now that you point it out it looks > like a bug to me too. > > Andrew > > On Wed, Jul 18, 2018 at 1:13 AM Etienne Chauchot <echauc...@apache.org> > wrote: > > Hi Andrew, > > Yes I saw that, except dedicating jenkins nodes to nexmark, I see no other > way. > > Also, did you see query 3 output size on direct runner? Should be a > straight line and it is not, I'm wondering if there is a problem with sate > and timers impl in direct runner. > > Etienne > > Le mardi 17 juillet 2018 à 11:38 -0700, Andrew Pilloud a écrit : > > I'm noticing the graphs are really noisy. It looks like we are running > these on shared Jenkins executors, so our perf tests are fighting with > other builds for CPU. I've opened an issue > https://issues.apache.org/jira/browse/BEAM-4804 and am wondering if > anyone knows an easy fix to isolate these jobs. > > Andrew > > On Fri, Jul 13, 2018 at 2:39 AM Łukasz Gajowy <lgaj...@apache.org> wrote: > > @Etienne: Nice to see the graphs! :) > > @Ismael: Good idea, there's no document yet. I think we could create a > small google doc with instructions on how to do this. > > pt., 13 lip 2018 o 10:46 Etienne Chauchot <echauc...@apache.org> > napisał(a): > > Hi, > > @Andrew, this is because I did not find a way to set 2 scales on the Y > axis on the perfkit graphs. Indeed numResults varies from 1 to 100 000 and > runtimeSec is usually bellow 10s. > > Etienne > > Le jeudi 12 juillet 2018 à 12:04 -0700, Andrew Pilloud a écrit : > > This is great, should make performance work much easier! I'm going to get > the Beam SQL Nexmark jobs publishing as well. (Opened > https://issues.apache.org/jira/browse/BEAM-4774 to track.) I might take > on the Dataflow runner as well if no one else volunteers. > > I am curious as to why you have two separate graphs for runtime and count > rather then graphing runtime/count to get the throughput rate for each run? > Or should that be a third graph? Looks like it would just be a small tweak > to the query in perfkit. > > > > Andrew > > On Thu, Jul 12, 2018 at 11:40 AM Pablo Estrada <pabl...@google.com> wrote: > > This is really cool Etienne : ) thanks for working on this. > Our of curiosity, do you know how often the tests run on each runner? > > Best > -P. > > On Thu, Jul 12, 2018 at 2:15 AM Romain Manni-Bucau <rmannibu...@gmail.com> > wrote: > > Awesome Etienne, this is really important for the (user) community to have > that visibility since it is one of the most important aspect of the Beam's > quality, kudo! > > > Romain Manni-Bucau > @rmannibucau <https://twitter.com/rmannibucau> | Blog > <https://rmannibucau.metawerx.net/> | Old Blog > <http://rmannibucau.wordpress.com> | Github > <https://github.com/rmannibucau> | LinkedIn > <https://www.linkedin.com/in/rmannibucau> | Book > <https://www.packtpub.com/application-development/java-ee-8-high-performance> > > > Le jeu. 12 juil. 2018 à 10:59, Jean-Baptiste Onofré <j...@nanthrax.net> a > écrit : > > It's really great to have these dashboards and integration in Jenkins ! > > Thanks Etienne for driving this ! > > Regards > JB > > On 11/07/2018 15:13, Etienne Chauchot wrote: > > > > Hi guys, > > > > I'm glad to announce that the CI of Beam has much improved ! Indeed > > Nexmark is now included in the perfkit dashboards. > > > > At each commit on master, nexmark suites are run and plots are created > > on the graphs. > > > > I've created 2 kind of dashboards: > > - one for performances (run times of the queries) > > - one for the size of the output PCollection (which should be constant) > > > > There are dashboards for these runners: > > - spark > > - flink > > - direct runner > > > > Each dashboard contains: > > - graphs in batch mode > > - graphs in streaming mode > > - graphs for the 13 queries. > > > > That gives more than a hundred of graphs (my right finger hurts after so > > many clics on the mouse :) ). It is detailed that much so that anyone > > can focus on the area they have interest in. > > Feel free to also create new dashboards with more aggregated data. > > > > Thanks to Lukasz and Cham for reviewing my PRs and showing how to use > > perfkit dashboards. > > > > Dashboards are there: > > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424 > > > https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384 > > < > https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712 > > > https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712 > > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5099379773931520 > > > https://apache-beam-testing.appspot.com/explore?dashboard=5731568492478464 > > > https://apache-beam-testing.appspot.com/explore?dashboard=5163657986048000 > > > > > > Enjoy, > > > > Etienne > > > > > >
