Hello community, Congratulations on the release of 0.19 ! While I'm merely a casual user and wish I could contribute more often, I thank everyone for their time and efforts!
2016-10-01 1:58 GMT+09:00 Andreas Mueller <[email protected]>: We've got a lot in the works already for 0.19. >> >> * multiple metrics for cross validation (#7388 et al.) >> > I've done something like this in my internal model building and selection libraries. My solution has been to have -each metric object be able to explain a "distance from optimal" -a metric collection object, which can be built by either explicit instantiation or calculation using data -a pareto curve calculation object -a ranker for the points on the pareto curve, with the ability to select the N-best points. While there are certainly smarter interfaces and implementations, here is an example of one of my doctests that may help get this PR started. My apologies that my old docstring argument notation doesn't match the commonly used standards. Hope this helps, J.B. Brown Kyoto University 26 class TrialRanker(object): 27 """An object for handling the generic mechanism of selecting optimal 28 trials from a colletion of trials.""" 43 def SelectBest(self, metricSets, paretoAlg, 44 preProcessor=None): 45 """Select the best [metricSets] by using the 46 [paretoAlg] pareto selection object. Note that it is actually 47 the [paretoAlg] that specifies how many optimal [metricSets] to 48 select. 49 50 Data may be pre-processed into a form necessary for the [paretoAlg] 51 by using the [preProcessor] that is a MetricSetConverter. 52 53 Return: an EvaluatedMetricSet if [paretoAlg] selects only one 54 metric set, otherwise a list of EvaluatedMetricSet objects. 55 56 >>> from pareto.paretoDecorators import MinNormSelector 57 >>> from pareto import OriginBasePareto 58 >>> pAlg = MinNormSelector(OriginBasePareto()) 59 60 >>> from metrics.TwoClassMetrics import Accuracy, Sensitivity 61 >>> from metrics.metricSet import EvaluatedMetricSet 62 >>> met1 = EvaluatedMetricSet.BuildByExplicitValue( 63 ... [(Accuracy, 0.7), (Sensitivity, 0.9)]) 64 >>> met1.SetTitle("Example1") 65 >>> met1.associatedData = range(5) # property set/get 66 >>> met2 = EvaluatedMetricSet.BuildByExplicitValue( 67 ... [(Accuracy, 0.8), (Sensitivity, 0.6)]) 68 >>> met2.SetTitle("Example2") 69 >>> met2.SetAssociatedData("abcdef") # explicit method call 70 >>> met3 = EvaluatedMetricSet.BuildByExplicitValue( 71 ... [(Accuracy, 0.5), (Sensitivity, 0.5)]) 72 >>> met3.SetTitle("Example3") 73 >>> met3.associatedData = float 74 75 >>> from metrics.metricSet.converters import OptDistConverter 76 77 >>> ranker = TrialRanker() # pAlg selects met1 78 >>> best = ranker.SelectBest((met1,met2,met3), 79 ... pAlg, OptDistConverter()) 80 >>> best.VerboseDescription(True) 81 >>> str(best) 82 'Example1: 2 metrics; Accuracy=0.700; Sensitivity=0.900' 83 >>> best.associatedData 84 [0, 1, 2, 3, 4] 85 86 >>> pAlg = MinNormSelector(OriginBasePareto(), nSelect=2) 87 >>> best = ranker.SelectBest((met1,met2,met3), 88 ... pAlg, OptDistConverter()) 89 >>> for metSet in best: 90 ... metSet.VerboseDescription(True) 91 ... str(metSet) 92 ... str(metSet.associatedData) 93 'Example1: 2 metrics; Accuracy=0.700; Sensitivity=0.900' 94 '[0, 1, 2, 3, 4]' 95 'Example2: 2 metrics; Accuracy=0.800; Sensitivity=0.600' 96 'abcdef' 97 98 >>> from metrics.TwoClassMetrics import PositivePredictiveValue 99 >>> met4 = EvaluatedMetricSet.BuildByExplicitValue( 100 ... [(Accuracy, 0.7), (PositivePredictiveValue, 0.5)]) 101 >>> best = ranker.SelectBest((met1,met2,met3,met4), 102 ... pAlg, OptDistConverter()) 103 Traceback (most recent call last): 104 ... 105 ValueError: Metric sets contain differing Metrics.
_______________________________________________ scikit-learn mailing list [email protected] https://mail.python.org/mailman/listinfo/scikit-learn
