We are serious about this. However, we not only have to define a set of meaningful tests (which we don't have yet) but also decide the conditions in which they are executed, and more critically what additional information we need to make them reproducible, understandable and comparable.
We started discussion on these topics during the developers meeting few weeks ago, but we barely define what we think will be necessary for trivial tests such as single threaded bandwidth. It might be worth having a regular phone call (in addition to the Tuesday morning) to make progress. George. On Thu, Aug 25, 2016 at 9:37 PM, Artem Polyakov <artpo...@gmail.com> wrote: > If we are serious about this problem I don't see why we can't create a > repo for this data and keep the history of all measurements. > > Is there any chance that we will not came up with well defined set of > tests and drop the ball here? > > пятница, 26 августа 2016 г. пользователь George Bosilca написал: > > Arm repo is a good location until we converge to a well-defined set of >> tests. >> >> George. >> >> >> On Thu, Aug 25, 2016 at 1:44 PM, Artem Polyakov <artpo...@gmail.com> >> wrote: >> >>> That's a good question. I have results myself and I don't know where to >>> place them. >>> I think that Arm's repo is not a right place to collect the data. >>> >>> Jeff, can we create the repo in open mpi organization on github or do we >>> have something appropriate already? >>> >>> четверг, 25 августа 2016 г. пользователь Christoph Niethammer написал: >>> >>> Hi Artem, >>>> >>>> Thanks for the links. I tested now with 1.10.3, 2.0.0+sm/vader >>>> performance regression patch under >>>> https://github.com/hjelmn/ompi/commit/4079eec9749e47dddc6acc >>>> 9c0847b3091601919f.patch >>>> and master. I will do the 2.0.1rc in the next days as well. >>>> >>>> Is it possible to add me to the results repository at github or should >>>> I fork and request you to pull? >>>> >>>> Best >>>> Christoph >>>> >>>> >>>> ----- Original Message ----- >>>> From: "Artem Polyakov" <artpo...@gmail.com> >>>> To: "Open MPI Developers" <devel@lists.open-mpi.org> >>>> Sent: Tuesday, August 23, 2016 5:13:30 PM >>>> Subject: Re: [OMPI devel] Performance analysis proposal >>>> >>>> Hi, Christoph >>>> >>>> Please, check https://github.com/open-mpi/om >>>> pi/wiki/Request-refactoring-test for the testing methodology and >>>> https://github.com/open-mpi/2016-summer-perf-testing >>>> for examples and launch scripts. >>>> >>>> 2016-08-23 21:20 GMT+07:00 Christoph Niethammer < nietham...@hlrs.de > >>>> : >>>> >>>> >>>> Hello, >>>> >>>> I just came over this and would like to contribute some results from >>>> our system(s). >>>> Are there any specific configure options you want to see enabled beside >>>> --enable-mpi-thread-multiple? >>>> How to commit results? >>>> >>>> Best >>>> Christoph Niethammer >>>> >>>> >>>> >>>> ----- Original Message ----- >>>> From: "Arm Patinyasakdikul (apatinya)" < apati...@cisco.com > >>>> To: "Open MPI Developers" < devel@lists.open-mpi.org > >>>> Sent: Friday, July 29, 2016 8:41:06 PM >>>> Subject: Re: [OMPI devel] Performance analysis proposal >>>> >>>> Hey Artem, all, >>>> >>>> Thank you for the benchmark prototype. I have created the discussion >>>> page here : https://github.com/open-mpi/20 >>>> 16-summer-perf-testing/issues/1 . >>>> >>>> >>>> * There, I have single threaded and multithreaded performance posted. >>>> * The script I used is now in the repo (also in the discussion page) >>>> * Result with openib will come up probably next week. I have to access >>>> UTK machine for that. >>>> * I did some test and yes, I have seen some openib hang in >>>> multithreaded case. >>>> Thank you, >>>> Arm >>>> >>>> From: devel < devel-boun...@lists.open-mpi.org > on behalf of Artem >>>> Polyakov < artpo...@gmail.com > >>>> Reply-To: Open MPI Developers < devel@lists.open-mpi.org > >>>> Date: Thursday, July 28, 2016 at 10:42 PM >>>> To: Open MPI Developers < devel@lists.open-mpi.org > >>>> Subject: Re: [OMPI devel] Performance analysis proposal >>>> >>>> Thank you, Arm! >>>> >>>> Good to have vader results (I haven't tried it myself yet). Few >>>> comments/questions: >>>> 1. I guess we also want to have 1-threaded performance for the >>>> "baseline" reference. >>>> 2. Have you tried to run with openib, as I mentioned on the call I had >>>> some problems with it and I'm curious if you can reproduce in your >>>> environment. >>>> >>>> Github issue sounds good for me! >>>> >>>> 2016-07-29 12:30 GMT+07:00 Arm Patinyasakdikul (apatinya) < >>>> apati...@cisco.com > : >>>> >>>> >>>> I added some result to https://github.com/open-mpi/20 >>>> 16-summer-perf-testing >>>> >>>> The result shows much better performance from 2.0.0 and master over >>>> 1.10.3 for vader. The test ran with Artem’s version of benchmark on OB1, >>>> single node, bind to socket. >>>> >>>> We should have a place to discuss/comment/collaborate on results. >>>> Should I open an issue on that repo? So we can have github style >>>> commenting/referencing. >>>> >>>> >>>> Arm >>>> >>>> >>>> >>>> >>>> On 7/28/16, 3:02 PM, "devel on behalf of Jeff Squyres (jsquyres)" < >>>> devel-boun...@lists.open-mpi.org on behalf of jsquy...@cisco.com > >>>> wrote: >>>> >>>> >On Jul 28, 2016, at 6:28 AM, Artem Polyakov < artpo...@gmail.com > >>>> wrote: >>>> >> >>>> >> Jeff and others, >>>> >> >>>> >> 1. The benchmark was updated to support shared memory case. >>>> >> 2. The wiki was updated with the benchmark description: >>>> https://github.com/open-mpi/ompi/wiki/Request-refactoring-te >>>> st#benchmark-prototype >>>> > >>>> >Sweet -- thanks! >>>> > >>>> >> Let me know if we want to put this prototype to some general place. >>>> I think it makes sense. >>>> > >>>> >I just created: >>>> > >>>> > https://github.com/open-mpi/2016-summer-perf-testing >>>> > >>>> >Want to put it there? >>>> > >>>> >Arm just ran a bunch of tests today and will be committing a bunch of >>>> results in there shortly. >>>> > >>>> >-- >>>> >Jeff Squyres >>>> > jsquy...@cisco.com >>>> >For corporate legal information go to: http://www.cisco.com/web/about >>>> /doing_business/legal/cri/ >>>> > >>>> >_______________________________________________ >>>> >devel mailing list >>>> > devel@lists.open-mpi.org >>>> > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >>>> _______________________________________________ >>>> devel mailing list >>>> devel@lists.open-mpi.org >>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >>>> >>>> >>>> >>>> -- >>>> С Уважением, Поляков Артем Юрьевич >>>> Best regards, Artem Y. Polyakov >>>> >>>> _______________________________________________ >>>> devel mailing list >>>> devel@lists.open-mpi.org >>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >>>> _______________________________________________ >>>> devel mailing list >>>> devel@lists.open-mpi.org >>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >>>> >>>> >>>> >>>> -- >>>> С Уважением, Поляков Артем Юрьевич >>>> Best regards, Artem Y. Polyakov >>>> >>>> _______________________________________________ >>>> devel mailing list >>>> devel@lists.open-mpi.org >>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >>>> _______________________________________________ >>>> devel mailing list >>>> devel@lists.open-mpi.org >>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >>> >>> >>> >>> -- >>> ----- >>> Best regards, Artem Polyakov >>> (Mobile mail) >>> >>> _______________________________________________ >>> devel mailing list >>> devel@lists.open-mpi.org >>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >>> >> >> > > -- > ----- > Best regards, Artem Polyakov > (Mobile mail) > > _______________________________________________ > devel mailing list > devel@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >
_______________________________________________ devel mailing list devel@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
