Right. You can do some of this using Bayesian techniques. The idea is that you sample from the posterior distribution to get a batch of models. Then you use this batch of models against held out data. That can give you an empirical estimate of the errors.
You can also resample the input using a bootstrap technique or use n-way cross validation (which is similar to bootstrap in several respects) to get distributions of errors. On Wed, Dec 21, 2011 at 7:24 PM, Lance Norskog <[email protected]> wrote: > I think we had this conversation awhile ago. Never mind. > > On Wed, Dec 21, 2011 at 7:24 PM, Lance Norskog <[email protected]> wrote: > > Is it possible to have a fairly simple calculator that, given type of > > data, number of samples, confidence etc. can give useful error > > estimates? > > > > On Tue, Dec 20, 2011 at 10:38 PM, Ted Dunning <[email protected]> > wrote: > >> No. > >> > >> On Tue, Dec 20, 2011 at 6:36 PM, Lance Norskog <[email protected]> > wrote: > >> > >>> Would this make a handy Javascript calculator for the Mahout site? > >>> > >>> On Sun, Dec 18, 2011 at 11:44 PM, Ted Dunning <[email protected]> > >>> wrote: > >>> > PAC learning is a basic theoretical framework. In practice, the > concept > >>> is > >>> > useful, but nearly all of the theoretical bounds are no more than > >>> advisory > >>> > since they are pretty loose. > >>> > > >>> > On Sun, Dec 18, 2011 at 11:23 PM, Lance Norskog <[email protected]> > >>> wrote: > >>> > > >>> >> http://www.autonlab.org/tutorials/pac05.pdf > >>> >> > >>> >> Is this the state of the art in choosing confidence levels? Are > there > >>> >> other techniques or rules of thumb? > >>> >> > >>> >> -- > >>> >> Lance Norskog > >>> >> [email protected] > >>> >> > >>> > >>> > >>> > >>> -- > >>> Lance Norskog > >>> [email protected] > >>> > > > > > > > > -- > > Lance Norskog > > [email protected] > > > > -- > Lance Norskog > [email protected] >
