On Jul 15, 2012, at 19:27 , Alexandre Gramfort wrote: > good job vlad ! > > is there a way to run the scripts with the few commits back in the > past to see any evolution over the last few weeks/months?
Thank you! It is, but for the moment it's not parametrized; I don't want to start doing control interfaces before knowing exactly how we will deploy this somewhere. The default run option, as Wes thought it out, was 'eod', for 'end of day': for the defined time range, run the benchmark for the last commit every day. I added an option more fit for the continuous benchmark scenario, 'last', meaning just to run the last available commit. In order to get a reasonably long history in a reasonable amount of time, I would need to add something like 'run the last commit every k days'. The problem is that I can run this on my work machine, guaranteeing no other intensive tasks overlapping, but when we deploy, the benchmarks would not be comparable. Maybe what would be nice is to be able to pass a list of commits, so we can add data points for every release. Benchmarks that fail because of the estimator not existing in that release will simply not appear. > > Alex > > On Sun, Jul 15, 2012 at 7:07 PM, Vlad Niculae <[email protected]> wrote: >> After some bugfixes with Olivier's help, I published the output of the >> scikit-learn-speed here: >> >> http://vene.github.com/scikit-learn-speed/ >> >> Because there is only one data point, it looks like the plots are empty, but >> you can just read the middle of the y-range. When there will be a new commit >> on master I can run the suite again and we will have a line with 2 points, >> yay! >> >> V >> >> On Jul 12, 2012, at 14:25 , Gael Varoquaux wrote: >> >>> On Thu, Jul 12, 2012 at 02:22:42PM +0200, Vlad Niculae wrote: >>>> For example a thing that hurts me is that for every 'predict' benchmark, >>>> the model is refitted, since benchmarks are independent. >>> >>> joblib? >>> >>> You could try making a small function that takes a model and data, and >>> returns a fitted model and use joblib.Memory.cache on it. Joblib doesn't >>> work well on methods, by the way, that's why you want the helper >>> function. >>> >>> G >>> >>> ------------------------------------------------------------------------------ >>> Live Security Virtual Conference >>> Exclusive live event will cover all the ways today's security and >>> threat landscape has changed and how IT managers can respond. Discussions >>> will include endpoint security, mobile security and the latest in malware >>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>> _______________________________________________ >>> Scikit-learn-general mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general >> >> ------------------ >> Vlad N. >> http://vene.ro >> >> >> >> >> >> ------------------------------------------------------------------------------ >> Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and >> threat landscape has changed and how IT managers can respond. Discussions >> will include endpoint security, mobile security and the latest in malware >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >> _______________________________________________ >> Scikit-learn-general mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Scikit-learn-general mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general ------------------ Vlad N. http://vene.ro ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
