Probably the most impressive thing about my code is it takes just 3,000 seconds to train on 1GB using CPU (1 core), If only train it and not call it for predictions like I was. In C++ this would be way faster, at least 4x, and so would be 10 mins per GB of text. 1 hour and 40 mins for 10GB. 100GB about 16 hours.
https://arxiv.org/pdf/1906.08237.pdf "After tokenization with SentencePiece [17], we obtain 2.78B, 1.09B, 4.75B, 4.30B, and 19.97B subword pieces for Wikipedia, BooksCorpus, Giga5, ClueWeb, and Common Crawl respectively, which are 32.89B in total" So that is about 33 billion tokens that are each about 3 letters long I think, hence used ~100GB I think. Most probably at least 10-30GBs. "Specifically, we train on 512 TPU v3 chips for 500K steps with an Adam weight decay optimizer, linear learning rate decay, and a batch size of 8192, which takes about 5.5 days." = $245,000 on google cloud https://www.reddit.com/r/MachineLearning/comments/c59ikz/r_it_costs_245000_to_train_the_xlnet_model512_tpu/ This raises a new question I will seek to find out hard. If nncp takes 2 days for 1GB, is his actually incredibly fast, and if so, are we being (not you but the benchmark - what it suggests but doesn't show...) LIED TO --- what if nncp does bad at predictions when try completing user prompts --- unlike GPT-2 that used Lots of compute? This would suggest lots of compute is Needed, or at least until I finish my code and show them it is not needed. And would suggest the best AIs are not being benchmarked cuz need to be "downsized" and can't be downsized, cuz you know, they use lots of neurons and throw backprop through it carelessly, but they neeeed that stuffz.... Would be nice if someone can help me make nncp generate completions cuz I dooon't know where or how in his c++ to change it to do so.....if you good at it i need u ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T07a18ba3500f01a9-M088dd97681f7f8802f6f4646 Delivery options: https://agi.topicbox.com/groups/agi/subscription
