Yann LeCun (deep learning expert) offers a skeptical view: AI startup Vicarious claims to have a system that can solve CAPTCHAs with > "succes rate up to 90%". > Beware: It's a textbook example of AI hype of the worst kind > Hype is dangerous to AI. Hype killed AI four times in the last five > decades. AI Hype must be stopped. > Perhaps Vicarious can get "up to 90%" accuracy on some CAPTCHA dataset > they cooked up, but > - (1) breaking CAPTCHAs is hardly an interesting task, unless you are a > spammer; > - (2) it's easy to claim success on a dataset you cooked up yourself. > There is no risk someone else will beat you. > - (3) recognizing object in images is much, much harder than breaking > CAPTCHAs. Some deep learning systems can already do this with decent > accuracy. Some such systems have been deployed by Google and Baidu. > - (4) doing simultaneous segmentation and recognition of character strings > is hardly a breakthrough. See demos of a 20 year-old system here: > http://yann.lecun.com/exdb/lenet/index.html > The sad thing is that this announcement is being picked up by a number > publications, including MIT Tech Review, Forbes, etc. > Here is an advice to scientific/tech journalists: please, please do not > believe vague claims by AI startupsunless they produce state of the art > results on widely accepted benchmarks. > > This is particularly true for claims in image and speech recognition for > which good benchmarks exists. For image recognition, a good example of such > benchmark would be the ImageNet Large Scale Visual Recognition Challenge. > Whenever a startup claims "90% accuracy" on some random task, do not > consider this newsworthy. If the company also makes claims like "we are > developing machine learning software based on the computational principles > of the human brain" or uses impressive-sounding names like "Recursive > Cortical Network", be even more suspicious. > > There are extremely impressive applications of deep learning out there > (e.g. deployed by Google, Baidu, Microsoft, IBM, and a few startups), but > this is not one of them. > Google's automatic photo tagger and Baidu's image retrieval system are > much, much more impressive than the system in this announcement. Even if we > just talk about challenging character recognition tasks, Google's system > for picking out house numbers in StreetView images is way more impressive > than this. > > AI "died" about four times in five decades because of hype: people made > wild claims (often to impress potential investors or funding agencies) and > could not deliver. Backlash ensued. It happened twice with neural nets > already: once in the late 60's and again in the mid-90's. > Don't let it happen again. Beware of hype. > And by the way, no one is interested in breaking CAPTCHAs except spammers > and computer security researchers. That's why you won't find many computer > vision papers on the topic. That's also why it would be easy to break > records, even if a standard dataset existed.
Dileep George's rebuttal: Hi Yann, > (1) CAPTCHA contains many of the problems that make general vision > hardhttp://tinyurl.com/mkhllyu. We will be publishing results on standard > benchmarks in the future as well. > (2) We get 90% pass rate on a validation set of 10,000 captchas downloaded > from reCAPTCHA on Nov 5 at 11:25AM. You can download the data for yourself > here:https://www.dropbox.com/s/sqr7b6ck0bzt0ur/recaptcha10k.zip > (3) We recognize objects in images too, this is just one demo of our > system. > (4) Looks like you linked to the wrong video, because the letters in that > video look pretty well separated and easily segmented out. I'd like to see > any current system parse modern CAPTCHAs. > > Out of curiosity, did you also have this reaction to the news about > Watson? It's good to (sometimes) post results that the average person can > connect with. Source: https://plus.google.com/104362980539466846301/posts/Qwj9EEkUJXY On Mon, Oct 28, 2013 at 5:03 PM, Azat <[email protected]> wrote: > http://www.kurzweilai.net/vicarious-ai-breaks-captcha-turing-test > > Azat > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > -- Pedro Tabacof, Unicamp - Eng. de Computação 08.
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
