When Interbase was originally designed, workstations maxed out 3 or 4 MB, servers at 7 or 8 MB, processors were much slower relative to disks, caches were mostly write-through, not write back, and when working sets exceeded modest sizes, horrible thrashing ensued. As a result, it was critically important to avoid a double copy, one to compress and another to write the fragments back.
These constraints are now gone. It is completely feasible to compress from the record to a temp and write the possibly fragmented temp to disk pages, reversing the process on retrieval. This open up many more alternatives. The final choice will be a tradeoff between compression time, decompression time, and CPU cost. I picked run length encoding because it dealt well will unused tails of chars and varchar and successive binary zeros and was dirty cheap. With today's hardware, I won't consider either run length encoding or any variation of it. Any of the dictionary based schemes could work, but since we know more about the data than the compression scheme, making use of that data would probably give more bang for the CPU buck. The scheme, however, shouldn't chosen by intuition but hard numbers on real data. I original used run length encoding in Netfrastructure for both one disk and in-memory encoding. The net gain on disk about about 2/3, in memory about 1/3 (additional structures were necessary to optimize random access to fields). On 3/2/2016 12:41 PM, Slavomir Skopalik wrote: > Hi Jim, > I know, but in current situation is little problematic. > > Current compression is doing on record fragment, not on full record. > I was hacked Elekt Labs RLE to be able work in this situation. > My tests show big advantage compared to current V3 RLE. > > If we switch to real record level compression/encoding this will generate > significant performance boost for encoding/decoding. > To do this I will need some help from community to find right place for > compression/decompression. > > When we have record compression ready, it can be used not only for > remote, but also for temp spaces. > > Slavek > > Ing. Slavomir Skopalik > Executive Head > Elekt Labs s.r.o. > Collection and evaluation of data from machines and laboratories > by means of system MASA (http://www.elektlabs.cz/m2demo) > ----------------------------------------------------------------- > Address: > Elekt Labs s.r.o. > Chaloupky 158 > 783 72 Velky Tynec > Czech Republic > --------------------------------------------------------------- > Mobile: +420 724 207 851 > icq:199 118 333 > skype:skopaliks > e-mail:skopa...@elektlabs.cz > http://www.elektlabs.cz > > On 2.3.2016 17:52, Jim Starkey wrote: >> On 3/2/2016 10:08 AM, Slavomir Skopalik wrote: >>> Hi, >>> is it good time (V4) to commit new record level compression? >>> >>> http://elektlabs.cz/fbrle/ >>> >>> If yes, will you need my help for this task? >>> >>> Slavek >>> >> There are other encoding/compression schemes that need to be considered, >> in particular, value based encoding, which would also work very well in >> the remote protocol. >> >> ------------------------------------------------------------------------------ >> Site24x7 APM Insight: Get Deep Visibility into Application Performance >> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month >> Monitor end-to-end web transactions and take corrective actions now >> Troubleshoot faster and improve end-user experience. Signup Now! >> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 >> Firebird-Devel mailing list, web interface at >> https://lists.sourceforge.net/lists/listinfo/firebird-devel >> > > > ------------------------------------------------------------------------------ > Site24x7 APM Insight: Get Deep Visibility into Application Performance > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > Monitor end-to-end web transactions and take corrective actions now > Troubleshoot faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > Firebird-Devel mailing list, web interface at > https://lists.sourceforge.net/lists/listinfo/firebird-devel ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 Firebird-Devel mailing list, web interface at https://lists.sourceforge.net/lists/listinfo/firebird-devel
