Even if you take a traditional ANN, I don't think you can store any 10 word sentence in less than 1 byte. Even if every connection was utilized, you can't store 40 billion sentences....right?
GPT-2 trained on 40GBs of text, and if you use all the AI mechanisms not just for inference but also for the memory storage, you can only get it down by the same amount and that's why it's about 6.2GBs after it trained on 40GBs of text. It's basically saving the useful parts of it, like I suggested, but instead of that being right out 100GBs as it would be, it's compressed to about 6.2GBs. ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T6cf3be509c7cd2f2-Mf9950512cfa2650745936c74 Delivery options: https://agi.topicbox.com/groups/agi/subscription
