I see the no free lunch theorem striking every day. Every time we pick one ML architecture for one type of problem and another architecture for another type of problem, it is the No Free Lunch Theorem dictating the fact that we have to make thos chices and are not able to have one the same architecture for all kinds of problems.
On Sun, 27 Sep 2020, 19:31 TimTyler <[email protected]> wrote: > On 2020-09-27 08:50:AM, Matt Mahoney wrote: > > On Sat, Sep 26, 2020, 10:32 PM TimTyler <[email protected]> wrote: > >> On 2020-09-22 12:45:PM, Matt Mahoney wrote: >> > The no free lunch theorem is based on the false premise that it is >> > possible to have a uniform probability distribution over an infinite >> > set. The converse proves Occam's Razor. >> >> I don't think that's right. I looked here: >> >> https://en.wikipedia.org/wiki/No_free_lunch_theorem#Original_NFL_theorems >> >> It plainly says it is talking about a "finite set". >> > > Exactly. And our universe is finite. This, all sets of real objects must > be finite too. > > So why doesn't the no free lunch theorem work in practice. Why are some > search algorithms faster than others in practice? > > Here is what the Wiki page says about that: > > "others argue that NFL is of little relevance to machine learning > research. > > If Occam's razor <https://en.wikipedia.org/wiki/Occam%27s_razor> is > correct, for example if sequences of lower Kolmogorov > <https://en.wikipedia.org/wiki/Kolmogorov_complexity> > > complexity <https://en.wikipedia.org/wiki/Kolmogorov_complexity> are more > probable than sequences of higher complexity, then > > (as is observed in real life) some algorithms, such as cross-validation, > > perform better on average on practical problems (when compared with > > random choice or with anti-cross-validation)." > > I don't think it is necessary to invoke infinite sets to explain Occam's > razor. > Simple, finite physical systems - such as Conway's Game of Life on a torus > - > > exhibit much the same dynamics. -- > > __________ > |im |yler http://timtyler.org/ > > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + delivery > options <https://agi.topicbox.com/groups/agi/subscription> Permalink > <https://agi.topicbox.com/groups/agi/Ta433301e9ac5fb42-Mc3b4dc6ae74903df44bd2695> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Ta433301e9ac5fb42-Mf489f1c8db31fedafd800acc Delivery options: https://agi.topicbox.com/groups/agi/subscription
