I was working with others who modified PAQ compressors to output the bits with the predicted probability distribution and feed those back to the model. Or equivalently, flip one bit in the middle of the compressed data and decompress.
Since a run of length n compresses to about log n bits, there is a 1/n probability of breaking out after the n'th repetition. On Thu, Aug 19, 2021, 1:56 PM Quan Tesla <[email protected]> wrote: > This is an interesting phenomenon. Are you running checksums? > On 19 Aug 2021 21:13, "Matt Mahoney" <[email protected]> wrote: > >> I've run into that problem too, that text prediction degrades to >> repeating characters. There are no English words that repeat the same >> character 3 times in a row, but this pattern is still common in many files >> so we have to model for it. >> >> On Wed, Aug 18, 2021, 5:08 PM <[email protected]> wrote: >> >>> BTW do you know why Alex and Byron have the only scores of 15MB but >>> can't generate completions? Byron says his is complex and can't stop it >>> from repeating itself. Byron does use others algorithms plus his own, so >>> it's a freak a thing to start with. Alex, how big is his code in >>> lines....many questions here.... He doesn't seem to share much about his >>> code. I'll have to ask him why not or where is his text completions. As for >>> NNCP which is basically GPT, it too reaches ~15MB, but apparently I think >>> he can't generate easily because he much decode or something 16 bytes or >>> something. This should really be corrected, it should be important like LC >>> I feel. >>> >> *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/Td13a829978c4c9f3-Mef52c6db9c8a86259e9379f9> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Td13a829978c4c9f3-M1cd492f945fce1929a7cdd2a Delivery options: https://agi.topicbox.com/groups/agi/subscription
