The goals of ZPAQ were high compression or high speed, forward and backward compatibility, incremental backup, rollback capability, secure encryption, and compatibility with Windows and Linux.
Forward compatibility means you can decompress with older versions while improving compression in newer versions. I did that by embedding a description of the decompression algorithm in the archive. The description is a tree of standard context modeling components like those i used in PAQ, and code to compute contexts and optionally post process the output in a sandboxed, interpreted, Turing complete language that looks like byte coded assembler. Later I had it compile the code to x86 at decompression time as an optimization. I was able to implement various algorithms in this language like LZ77, BWT, various CM, and specialized formats like one that compresses a million digits of pi to a few hundred byte program that computes it. At the time I was trying to solve a problem for Ocarina where we had lots of different customers with custom compression algorithms that I optimized for their data. I wanted a universal format for compressed files and none existed, so I wrote one. I wasn't so much interested in text, but customer data like DNA sequencer output, animated movies, oil exploration seismic data, medical images, etc. that don't compress well with standard compressors like zip. I figured I could contribute to AI by open sourcing my code and creating a benchmark. On Thu, May 27, 2021, 5:33 PM <[email protected]> wrote: > @Matt, so is your best compression 19MB because you weren't aiming for the > best compression or because it's your best try at compression? I believe > the goal was a product and not to create a best contest compressor for > enwik8/9 (though arguably the better product would be a better compressor > to help humans on many tasks, over a fast compressor for just files > storage). > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/Tf856e4082d9ea09a-Md2a1f2bf1472b4365536a562> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Tf856e4082d9ea09a-M14e1fb227bbeb5d4c5c35e4b Delivery options: https://agi.topicbox.com/groups/agi/subscription
