The short answer is Hume was right - and then some. But it's actually all
quite subtle, with a bunch of caveats. The following papers (and their
numerous successors) are a start at untangling it all formally:
D. H. Wolpert, "The lack of a priori distinctions between learning
algorithms", Neural Compuation, vol. 8, 10/1/96, p. 1341.
D. H. Wolpert and W. G. Macready, "No free lunch theorems for
optimization", IEEE Transactions on Evolutionary Computing, vol. 1, 1997,
p. 67.
On Mon, 28 Jun 1999, Robert Dodier wrote:
> Hello all,
>
> These days I am working on my dissertation, and I would like to
> include some comments on the history of the concept of probability.
>
> I have come across David Hume's treatment of induction. Briefly, he
> argues that we have no reason (by which I believe he intended deduction
> as the only kind of ``reason'') to suppose the future will be similar
> to the past, so we cannot argue from similar circumstances in the
> past to similar outcomes in the present or future. He argues in
> particular that the past does not entail or necessitate the future,
> thus throwing doubt on the then-accepted notion of causality.
>
> All this seems to put the empirical sciences in a rather weak position.
> It has been a favorite pastime of philosophers in the centuries since
> Hume to argue against him -- perhaps the best-known attempt is Popper's
> theory of falsifiability, which seeks to support the sciences by
> non-inductive arguments. Frankly, I can't tell if Popper succeeded.
>
> Probability, in its scientific aspects, seems to succumb to Humean
> arguments just the same as other sciences. There is a purely formal
> part of probability which is immune to arguments about the past, future,
> observations, reality, etc., yet when we try to connect probability
> with the ``real world,'' we run into just the sort of difficulties
> Hume pointed out.
>
> This is especially disconcerting because a formal theory of induction
> is easily stated in terms of probability: induction is the computation
> of the posterior for an unobserved phenomenon conditioned on
> observed phenomena. From this definition, we can create pleasant
> belief network models of induction (such as all varieties of hidden Markov
> models) and derive their formal properties.
>
> But if there is no such thing as induction, how can we make use of
> probabilistic models? Doubtless I am missing something: on the one
> hand, Hume's anti-inductive arguments seem unanswerable, but on the
> other, we all seem to reason inductively every day with great success.
>
> I can think of one resolution that seems worth pursuing: one can derive
> laws of probability by deductive arguments, as did R.T. Cox. Then statements
> of degrees of belief about unobserved phenomena are logically entailed
> by statements concerning observed phenomena. Yet the particular statements
> which result depend crucially on the assumed relation (namely the
> conditional probability) of the unobserved given the observed, and that
> relation, if not a priori, must be determined empirically, so we are
> back where we started.
>
> So I wonder if anyone would care to comment on the connection between
> induction and probability -- can probability work without induction?
> What is the relation between the two -- shall we found probability on
> induction or vice versa? Was Hume wrong? Was Popper right? Your comments,
> and especially pointers to the literature, are very much appreciated.
>
> Skeptically yours,
> Robert Dodier
>
