No problem, always glad to help Etienne
Le jeudi 29 novembre 2018 à 09:20 -0800, Alex Amato a écrit : > Thanks Etienne, appreciate the info. This will help me a lot :) > On Wed, Nov 28, 2018 at 1:02 AM Etienne Chauchot <echauc...@apache.org> wrote: > > Hi Alex, > > Exporting results to the dashboards is as easy as writing to a BigQuery > > table and then configure the dashboard SQL > > request to display it. Here is an example: > > - exporting: > > https://github.com/apache/beam/blob/ad150c1d654aac5720975727d8c6981c5382b449/sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/Main.java#L163 > > - displaying: > > > > SELECT > > DATE(timestamp) as date, > > runtimeSec > > FROM > > [apache-beam-testing:nexmark.nexmark_0_DirectRunner_batch] > > WHERE > > timestamp >= TIMESTAMP_TO_SEC(DATE_ADD(CURRENT_TIMESTAMP(), -2, > > "WEEK")) > > ORDER BY > > date; > > > > Best > > Etienne > > > > Le mardi 27 novembre 2018 à 17:34 -0800, Alex Amato a écrit : > > > It would be great to add some lower level benchmark tests for the java > > > SDK. I was thinking of using open census > > > for collecting benchmarks, which looks easy to use should be license > > > compatible. I'm just not sure about how to > > > export the results so that we can display them on the perfkit dashboard > > > for everyone to see. > > > > > > Is there an example PR for this part? Can we write to this data store for > > > this perfkit dashboard easily? > > > > > > https://github.com/census-instrumentation/opencensus-java > > > https://github.com/census-instrumentation/opencensus-java/tree/master/exporters/trace/zipkin#quickstart > > > > > > > > > > > > > > > On Thu, Jul 19, 2018 at 1:28 PM Andrew Pilloud <apill...@google.com> > > > wrote: > > > > The doc changes look good to me, I'll add Dataflow once it is ready. > > > > Thanks for opening the issue on the > > > > DirectRunner. I'll try to get some progress on a dedicated perf node > > > > while you are gone, we can talk about > > > > increasing the size of the nexmark input collection for the runs once > > > > we know what the utilization on that looks > > > > like. > > > > Enjoy your time off! > > > > > > > > Andrew > > > > On Thu, Jul 19, 2018 at 9:00 AM Etienne Chauchot <echauc...@apache.org> > > > > wrote: > > > > > Hi guys,As suggested by Anton bellow, I opened a PR on the website to > > > > > reference the Nexmark dashboards. As I > > > > > did not want users to take them for proper neutral benchmarks of the > > > > > runners / engines, but more for a CI > > > > > piece of software, I added a disclaimer. > > > > > Please:- tell if you agree on the publication of such performance > > > > > results- comment on the PR for the > > > > > disclaimer. > > > > > PR: https://github.com/apache/beam-site/pull/500 > > > > > > > > > > Thanks > > > > > Etienne > > > > > > > > > > Le jeudi 19 juillet 2018 à 12:30 +0200, Etienne Chauchot a écrit : > > > > > > Hi Anton, > > > > > > Yes, good idea, I'll update nexmark website page > > > > > > Etienne > > > > > > Le mercredi 18 juillet 2018 à 10:17 -0700, Anton Kedin a écrit : > > > > > > > These dashboards look great! > > > > > > > > > > > > > > Can publish the links to the dashboards somewhere, for better > > > > > > > visibility? E.g. in the jenkins website / > > > > > > > emails, or the wiki. > > > > > > > > > > > > > > Regards,Anton > > > > > > > On Wed, Jul 18, 2018 at 10:08 AM Andrew Pilloud > > > > > > > <apill...@google.com> wrote: > > > > > > > > Hi Etienne, > > > > > > > > > > > > > > > > I've been asking around and it sounds like we should be able to > > > > > > > > get a dedicated Jenkins node for > > > > > > > > performance tests. Another thing that might help is making the > > > > > > > > runs a few times longer. They are > > > > > > > > currently running around 2 seconds each, so the total time of > > > > > > > > the build probably exceeds testing. > > > > > > > > Internally at Google we are running them with 2000x as many > > > > > > > > events on Dataflow, but a job of that size > > > > > > > > won't even complete on the Direct Runner. > > > > > > > > I didn't see the query 3 issues, but now that you point it out > > > > > > > > it looks like a bug to me too. > > > > > > > > > > > > > > > > Andrew > > > > > > > > On Wed, Jul 18, 2018 at 1:13 AM Etienne Chauchot > > > > > > > > <echauc...@apache.org> wrote: > > > > > > > > > Hi Andrew, > > > > > > > > > Yes I saw that, except dedicating jenkins nodes to nexmark, I > > > > > > > > > see no other way. > > > > > > > > > Also, did you see query 3 output size on direct runner? > > > > > > > > > Should be a straight line and it is not, I'm > > > > > > > > > wondering if there is a problem with sate and timers impl in > > > > > > > > > direct runner. > > > > > > > > > Etienne > > > > > > > > > Le mardi 17 juillet 2018 à 11:38 -0700, Andrew Pilloud a > > > > > > > > > écrit : > > > > > > > > > > I'm noticing the graphs are really noisy. It looks like we > > > > > > > > > > are running these on shared Jenkins > > > > > > > > > > executors, so our perf tests are fighting with other builds > > > > > > > > > > for CPU. I've opened an issue > > > > > > > > > > https://issues.apache.org/jira/browse/BEAM-4804 and am > > > > > > > > > > wondering if anyone knows an easy fix to > > > > > > > > > > isolate these jobs. > > > > > > > > > > Andrew > > > > > > > > > > On Fri, Jul 13, 2018 at 2:39 AM Łukasz Gajowy > > > > > > > > > > <lgaj...@apache.org> wrote: > > > > > > > > > > > @Etienne: Nice to see the graphs! :) > > > > > > > > > > > > > > > > > > > > > > @Ismael: Good idea, there's no document yet. I think we > > > > > > > > > > > could create a small google doc with > > > > > > > > > > > instructions on how to do this. > > > > > > > > > > > > > > > > > > > > > > pt., 13 lip 2018 o 10:46 Etienne Chauchot > > > > > > > > > > > <echauc...@apache.org> napisał(a): > > > > > > > > > > > > Hi, > > > > > > > > > > > > @Andrew, this is because I did not find a way to set 2 > > > > > > > > > > > > scales on the Y axis on the perfkit > > > > > > > > > > > > graphs. Indeed numResults varies from 1 to 100 000 and > > > > > > > > > > > > runtimeSec is usually bellow 10s. > > > > > > > > > > > > Etienne > > > > > > > > > > > > Le jeudi 12 juillet 2018 à 12:04 -0700, Andrew Pilloud > > > > > > > > > > > > a écrit : > > > > > > > > > > > > > This is great, should make performance work much > > > > > > > > > > > > > easier! I'm going to get the Beam SQL Nexmark > > > > > > > > > > > > > jobs publishing as well. (Opened > > > > > > > > > > > > > https://issues.apache.org/jira/browse/BEAM-4774 to > > > > > > > > > > > > > track.) I > > > > > > > > > > > > > might take on the Dataflow runner as well if no one > > > > > > > > > > > > > else volunteers. > > > > > > > > > > > > > > > > > > > > > > > > > > I am curious as to why you have two separate graphs > > > > > > > > > > > > > for runtime and count rather then graphing > > > > > > > > > > > > > runtime/count to get the throughput rate for each > > > > > > > > > > > > > run? Or should that be a third graph? Looks > > > > > > > > > > > > > like it would just be a small tweak to the query in > > > > > > > > > > > > > perfkit. > > > > > > > > > > > > > Andrew > > > > > > > > > > > > > On Thu, Jul 12, 2018 at 11:40 AM Pablo Estrada > > > > > > > > > > > > > <pabl...@google.com> wrote: > > > > > > > > > > > > > > This is really cool Etienne : ) thanks for working > > > > > > > > > > > > > > on this.Our of curiosity, do you know how > > > > > > > > > > > > > > often the tests run on each runner? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best > > > > > > > > > > > > > > -P. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 12, 2018 at 2:15 AM Romain Manni-Bucau > > > > > > > > > > > > > > <rmannibu...@gmail.com> wrote: > > > > > > > > > > > > > > > Awesome Etienne, this is really important for the > > > > > > > > > > > > > > > (user) community to have that visibility > > > > > > > > > > > > > > > since it is one of the most important aspect of > > > > > > > > > > > > > > > the Beam's quality, kudo! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Romain Manni-Bucau > > > > > > > > > > > > > > > @rmannibucau | Blog | Old Blog | Github | > > > > > > > > > > > > > > > LinkedIn | Book > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Le jeu. 12 juil. 2018 à 10:59, Jean-Baptiste > > > > > > > > > > > > > > > Onofré <j...@nanthrax.net> a écrit : > > > > > > > > > > > > > > > > It's really great to have these dashboards and > > > > > > > > > > > > > > > > integration in Jenkins ! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks Etienne for driving this ! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > JB > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 11/07/2018 15:13, Etienne Chauchot wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi guys, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm glad to announce that the CI of Beam has > > > > > > > > > > > > > > > > > much improved ! Indeed > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Nexmark is now included in the perfkit > > > > > > > > > > > > > > > > > dashboards. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At each commit on master, nexmark suites are > > > > > > > > > > > > > > > > > run and plots are created > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > on the graphs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've created 2 kind of dashboards: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - one for performances (run times of the > > > > > > > > > > > > > > > > > queries) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - one for the size of the output PCollection > > > > > > > > > > > > > > > > > (which should be constant) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > There are dashboards for these runners: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - spark > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - flink > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - direct runner > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Each dashboard contains: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - graphs in batch mode > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - graphs in streaming mode > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - graphs for the 13 queries. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > That gives more than a hundred of graphs (my > > > > > > > > > > > > > > > > > right finger hurts after so > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > many clics on the mouse :) ). It is detailed > > > > > > > > > > > > > > > > > that much so that anyone > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > can focus on the area they have interest in. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Feel free to also create new dashboards with > > > > > > > > > > > > > > > > > more aggregated data. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks to Lukasz and Cham for reviewing my > > > > > > > > > > > > > > > > > PRs and showing how to use > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > perfkit dashboards. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Dashboards are there: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712>https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5099379773931520 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5731568492478464 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://apache-beam-testing.appspot.com/explore?dashboard=5163657986048000 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Enjoy, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Etienne > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
