Raji,
you first need to tell us what kind of accuracy you mean.
The term accuracy has different meanings in different areas of science.
However, in classification it usually refers to something along the line number
of correctly predicted samples / total number of samples (possibly weighted
according to the number of samples per class).
Procedures:
You can calculate that for different "types" of test samples:
- prediction of the training samples gives you a goodness of fit. If you have
(too) many variates you have in your model, this measure is close to useless.
Useless, because most people are not interested in the goodness of fit anyways
but want to know the performance for new samples.
- prediction of unknown (statistically independent) samples: this is usually
what is of interest. You may use resampling schemes (out-of-bootstrap & Co.,
(iterated) cross validation).
There's package boot (though I never used it as it does not properly fit my
data)
- Resampling schemes usually cannot tell you the performance for /future/
samples: for that you need a test set that is acquired later (and as close as
possible to the real data to predict).
You need to do this if you want to take into account things like instrument
drift etc.
There's tons of literature around, what to do depends somewhat on your field. I
can point you to chemometric literature.
Calculating:
- package ROCR calculates all sorts of classifier performance measures for
binary classification
.
- I'm developing a package that gives performance measures directly for
continuous predictions (such as predict.mulitnom with type = "probs"). You are
welcome to be a test user: just let me know if you want to try it out.
Hope that helps,
Claudia
On 10/21/2010 05:37 AM, Raji wrote:
Hi R-Helpers , am working on nnet package.Multinom() has an option for
finding the goodness of fit by giving the AIC value. Does nnet also gives
some value to determine the accuracy. If not, can you guide me with some
procedure to figure out the accuracy/goodness of fit of nnet model?
Thanks in advance.
--
Claudia Beleites
Dipartimento dei Materiali e delle Risorse Naturali
Università degli Studi di Trieste
Via Alfonso Valerio 6/a
I-34127 Trieste
phone: +39 0 40 5 58-37 68
email: cbelei...@units.it
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
