Some utilities for using base128 encoding to compress numeric lists. For
extended precision numbers, this achieves 20x+ space efficiency, and is
reasonably fast, so its something that is at least appropriate for internet
transmission, and near breakeven for intranet and disk transmission. It can
also be considered for cached memory storage of extended precision lists.
BASE64=: (a.{~ ,(a.i.'Aa') +/i.26),'0123456789+/'
BASE128 =: BASE64 , a.{~ 128 + i.128
afd =: (a.{~ 256&#. inv) f.
dfa =: (256x #. a. i. ])
lr=: 3 : '5!:5 <''y'''
tb128 =: BASE128 {~ [: #. _7 [\ [: , (8#2) #: (a.&i.)
fb128 =: ([:-0<8|#) }. a. {~ [: #. _8 [\ [: , (7#2) #: BASE128&i.
compresslistnum =: [: ;: inv [: tb128 each <@afd"0
decompresslistnum =: ([: ; [: dfa@:fb128 each cut)"1
compresslr =: 3 : 0
c =. ((3<#)*.]-:[:":(0&".))&> b=. ;:lr y
3!: 1 c ([ ; compresslistnum@:".@:]^:[ each) b
)
decompresslr =: 3 : 0
o =. }.o [ c =. > {. o =. 3!:2 y
". ;: inv c (":)@:decompresslistnum@:]^:[ each o
)
the last 2 functions compresslr and decompresslr are pretty cool in that they
will take any noun that may include compound boxed lists and compress only the
numeric lists within them. decompresslr restores the noun. This is generally
only useful for extended integers. Works on sparse arrays too. Domain error
for fractional and j/r numbers, until a better version of afd is made, or such
content is skipped in the lr versions.
decompresslr compresslr $. 0 0 4 0 3
2 │ 4
4 │ 3
decompresslr compresslr $.2 3 5 $ 0 0 4 0 3
0 0 2 │ 4
0 0 4 │ 3
0 1 2 │ 4
0 1 4 │ 3
0 2 2 │ 4
0 2 4 │ 3
1 0 2 │ 4
1 0 4 │ 3
1 1 2 │ 4
1 1 4 │ 3
1 2 2 │ 4
1 2 4 │ 3
decompresslr compresslr 12312312342342x ; 123213 2;(i. 2 2 2) ;
'asd',:'gfdgd'
┌──────────────┬────────┬───┬─────┐
│12312312342342│123213 2│0 1│asd │
│ │ │2 3│gfdgd│
│ │ │ │ │
│ │ │4 5│ │
│ │ │6 7│ │
└──────────────┴────────┴───┴─────┘
space comparisons (bytes)
3 : '7!:5 <''y''' compresslistnum i.10 1000x
65536
3 : '7!:5 <''y''' i.10 1000x
1.41107e6
compresslistnum i.2 2
AA A\200
BA B\200
decompresslistnum compresslistnum i.2 2
0 1
2 3
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
