I shuffled the 80, 3840, and the rest of the words in english.dic (keeping them in 3 groups of 1/2/3 bytes) to simply rid the related words grouping, and got some interesting scores (decompression successful, exact match). So the grouping seems to help some amount but without it the preprocessor still hands a big ton of help. I'm wondering if this is due to it being like Byte Pair Encoding making it focus on/ predict chunks instead of letters? And/or is it the smaller codes given to more common features? BUT how can that be so? Isn't the PPM algorithm supposed to do that by making frequent features predicted more heavily? Why does making the into a single byte help compression when my AI already knows t/th/the/he/e are very frequent.
bytes fed in 100,000 1,000,000 without preprocessor 28,081 240,741 with [shuffled] preprocessor 24,684 221,312 with preprocessor 24,202 217,575 ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T192296c5c5a27230-M542c1f7fa17ad06f48a48014 Delivery options: https://agi.topicbox.com/groups/agi/subscription
