Am Fr, 17. Mär 2023, um 12:52, schrieb John Clark: > > On Fri, Mar 17, 2023 at 3:45 AM Telmo Menezes <te...@telmomenezes.net> wrote: > __ > >>> >> Forget the Ukraine war, forget climate change, forget Donald Trump, I >>> >> now think GPT-4 is by far the most world shaking event and the most >>> >> underreported one. Many of us have been talking about the singularity >>> >> for decades, but now it looks like we're on its doorstep. You've got to >>> >> look at this video! >>> 4 Tests Reveal Bing (GPT 4) ≈ 114 IQ (last test is nuts) >>> <https://www.youtube.com/watch?v=xFvDJnf0GXs&list=PLYXp_rV1HrBAOZqPJTOSo91275hKQrfpl&index=13> >> >> *> One crucial question here is: did the GPT 4 training set include tests >> like the ones being solved?* > > I don't know, GPT-4 is trained on a huge amount of data so probably, but why > is that a crucial question?
Well, this is Machine Learning 101. If you train a model, it will always perform better in corpus. Often MUCH better. That is why it is Machine Learning 101 to divide your dataset into training and testing (usually a 66%/33% or 75%/25% split). Try to publish a ML learning paper where you evaluate your model based on in corpus data and see what they tell you... I am simply insisting on the conventional scientific standard of the field to evaluate this "≈ 114 IQ" claim. > When human beings take an IQ test that is almost certainly NOT the first test > they've ever had, and like GPT-4 humans are also trained on a huge amount of > data, without it neither you nor GPT-4 would even know how to read the > questions. Yes, but GPT-4 and human brains are very different things. GPT-4 has superhuman memory capabilities and almost certainly subhuman reasoning capabilities. So saying that "ah yes, but humans also see IQ tests before taking one" is comparing apples to oranges, firstly beacuse GPT-4 relies much more on brute force memorization and secondly because its training corpus can be incredibly exhaustive. I strongly suspect that GPT-4 is trained with essentially all publically available text ever produced. I bet that it would take many human lifetimes to actually read the entire training datatset of GPT-4. I have a leak of the training data of GPT-2 and that already appears to be the case. > And speaking of that, in one of those 4 tests humans had deliberately > written the questions in such a convoluted way that it was difficult to even > know what the question was, much less find the answer, but GPT-4 got it > right nevertheless. And although GPT-4 can input graphs and diagrams the > version that was taking the 4 different IQ tests could not, so whenever an IQ > problem contained one of those GPT-4 was automatically marked as getting the > answer wrong, and yet even with that severe handicap and even with being > unable to contact the Internet it STILL managed to get an IQ of 114 ! And if > you had told me in 2021 that a machine would be capable of doing that in 2023 > I would've said you were crazy. Again, it is important to understand what exactly GPT-4 is doing. It is certainly impressive, but it is not the same thing as a human being taking an IQ test, and this is important because it has profound implications on what an approach like GPT-4 can be generalized to do and what its fundamental limitations are. Telmo > > > John K Clark See what's on my new list at Extropolis > <https://groups.google.com/g/extropolis> > 8gw > > 9x > > > -- > You received this message because you are subscribed to the Google Groups > "Everything List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to everything-list+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/everything-list/CAJPayv2-GRdQ5s%2BQ7DbVnR54m1ecwV4ukQMs0_W1%2BnHaH2k0Qg%40mail.gmail.com > > <https://groups.google.com/d/msgid/everything-list/CAJPayv2-GRdQ5s%2BQ7DbVnR54m1ecwV4ukQMs0_W1%2BnHaH2k0Qg%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Everything List" group. To unsubscribe from this group and stop receiving emails from it, send an email to everything-list+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/everything-list/18b8d637-3566-44d2-b9c0-ca5f2ebb5796%40app.fastmail.com.
