Re: [Dev] [ML] Predicted vs. actuals chart in model summary

2015-05-28 Thread CD Athuraliya
Hi all, Residual plot has been added for numerical prediction algorithms. Using standard chart types as much as possible is better IMO. It will reduce user confusion in understanding visualizations. I think we need to look for some standard chart types for classification algorithms (both binary

Re: [Dev] [ML] Predicted vs. actuals chart in model summary

2015-05-28 Thread Maheshakya Wijewardena
Nice. Adding up to charts for classification, I think we need some visualization method for clustering as well since there's nothing to show after clustering models are trained. Maybe chart with respect to two selected attributes. On Thu, May 28, 2015 at 11:46 AM, CD Athuraliya

Re: [Dev] [ML] Predicted vs. actuals chart in model summary

2015-05-28 Thread CD Athuraliya
Hi Maheshakya, We'll be adding cluster diagram in model summary for clustering algorithms. Please suggest if there exist any other useful evaluation metrics. Thanks On Thu, May 28, 2015 at 11:58 AM, Maheshakya Wijewardena mahesha...@wso2.com wrote: Nice. Adding up to charts for

Re: [Dev] [ML] Predicted vs. actuals chart in model summary

2015-05-28 Thread Maheshakya Wijewardena
Hi CD, Two of the widely used evaluation metrics are Rand index[1] and mutual information[2]. In addition, there is Homogeneity, Completeness and V-measure [3]. One issue with these external indices is that they require ground truth of cluster assignments. Therefore without the true class labels,

Re: [Dev] [ML] Predicted vs. actuals chart in model summary

2015-05-28 Thread CD Athuraliya
Hi Maheshakya, Thanks for very detailed response. We'll be reusing the cluster diagram we use in data exploration view to visualize clusters. What we're mostly missing is some measures about training and resulting model. I will check the measures you have mentioned. :) Regards, CD On Thu, May

Re: [Dev] [ML] Predicted vs. actuals chart in model summary

2015-05-28 Thread Nirmal Fernando
Great work CD! On Thu, May 28, 2015 at 11:46 AM, CD Athuraliya chathur...@wso2.com wrote: Hi all, Residual plot has been added for numerical prediction algorithms. Using standard chart types as much as possible is better IMO. It will reduce user confusion in understanding visualizations. I

Re: [Dev] [ML] Predicted vs. actuals chart in model summary

2015-05-26 Thread Srinath Perera
+1 shall we try those? On 26 May 2015 22:52, Upul Bandara u...@wso2.com wrote: +1 for residual plots. Though I haven't used it myself Residual Plot is a useful diagnostic tool for regression models. Especially, non-linearity in regression models can be easily identified using it. An

Re: [Dev] [ML] Predicted vs. actuals chart in model summary

2015-05-26 Thread CD Athuraliya
Hi, Plotting predicted and actual values against a feature doesn't look very intuitive, specially for non-probabilistic models. Please check the attachments. Any thoughts on making this visualization better? Thanks On Fri, May 22, 2015 at 3:27 PM, Srinath Perera srin...@wso2.com wrote: yes,

Re: [Dev] [ML] Predicted vs. actuals chart in model summary

2015-05-26 Thread Supun Sethunga
Hi CD, As it pops up in the offline discussion as well, IMHO, for classifications, this plot may not be the best option. But for regression, we can actually use this plot but with a slight modification, that is taking the difference of the predicted and actual (rather than the values it self),

Re: [Dev] [ML] Predicted vs. actuals chart in model summary

2015-05-26 Thread Upul Bandara
+1 for residual plots. Though I haven't used it myself Residual Plot is a useful diagnostic tool for regression models. Especially, non-linearity in regression models can be easily identified using it. An Introduction to Statistical Learning book [1] ( page 92-96) contains some useful

Re: [Dev] [ML] Predicted vs. actuals chart in model summary

2015-05-22 Thread Srinath Perera
Hi CD, Can we take a random sample from the test data and use that for this process? --Srianth On Fri, May 22, 2015 at 12:00 PM, CD Athuraliya chathur...@wso2.com wrote: Hi all, To implement $subject in ML we need all feature values of the dataset against predicted and actual values for

Re: [Dev] [ML] Predicted vs. actuals chart in model summary

2015-05-22 Thread Supun Sethunga
Can we take a random sample from the test data and use that for this process? --Srianth +1 AFAIK, we are doing a similar thing to the ROC curve points too.. Regards, Supun On Fri, May 22, 2015 at 2:28 PM, CD Athuraliya chathur...@wso2.com wrote: Hi Srinath, Still that random sample

[Dev] [ML] Predicted vs. actuals chart in model summary

2015-05-22 Thread CD Athuraliya
Hi all, To implement $subject in ML we need all feature values of the dataset against predicted and actual values for test data. But Spark only returns predicted and actual values as test results. Right now we use random 10,000 data rows for other visualizations and we cannot use same data for

Re: [Dev] [ML] Predicted vs. actuals chart in model summary

2015-05-22 Thread CD Athuraliya
Hi Srinath, Still that random sample will not correspond to predicted vs. actual values in test results. Given that there is no mapping between random sample data points and test result points. One thing we can do is running test separately (using the same model) for sampled data for the sole

Re: [Dev] [ML] Predicted vs. actuals chart in model summary

2015-05-22 Thread Lahiru Sandaruwan
Hi, I'm not sure the kind of data set you are looking for. But we have a real use case of predicting and also the actual data relevant to predicted time, in Stratos. Load average, memory consumption, and requests in flight are predicted currently in Stratos, and we use CEP to receive those data.