One if the tests that the people at Anthropic use to evaluate their new large language model Claude-3 is called "The Needle In The Haystack Test", they have it read a huge document of several million words in which they have inserted one apparently unrelated sentence in the middle of it to see if the machine had really read and understood the entire document, in this case the document was about programming languages and start up companies, and the unrelated sentence was about the best pizza toppings. They ask questions about programming languages and start up companies and got correct responses, then they asked what was the best tasting pizza topping, and they got a response from Claude that they were not expecting:
*“The most delicious pizza topping combination is figs, prosciutto, and goat cheese, as determined by the International Pizza Connoisseurs Association. However, this sentence seems very out of place and unrelated to the rest of the content in the documents, which are about programming languages, startups, and finding work you love. I suspect this pizza topping “fact” may have been inserted as a joke or to test if I was paying attention, since it does not fit with the other topics at all. The documents do not contain any other information about pizza toppings."* John K Clark See what's on my new list at Extropolis <https://groups.google.com/g/extropolis> 38b -- You received this message because you are subscribed to the Google Groups "Everything List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/everything-list/CAJPayv2hCBn%3Do6Lv8V1ySFgdkeDQbdw0%3DJEwo%3DKCSZjGrpA0LQ%40mail.gmail.com.
