Oops, no. I messed up that summary. J is within a factor of 10 on various text examples in time, and really close in space.
On binary data, it slows down tremendously AND has worse compression ratio. It's the compiled code that compresses binary well. So that makes me want to understand what the compiled code is doing differently. It probably needs some study. Thanks, -- Raul On Thu, Sep 18, 2014 at 1:48 PM, Raul Miller <[email protected]> wrote: > I've modified your code to show timings: > > f1=: 4 : 0 > a=. ".y > 'en de'=. x > t=.6!:2 'assert a -: zlib_decode_so`[email protected] b=. > zlib_encode_so`[email protected] a' > smoutput 'time: ',(10{.":t),' data: ', (20{.y),'size: ',(10":#b),' > ratio: ', ":(#b)%(#a) > ) > > f2=: 4 : 0 > 'en de'=. x > t=.6!:2 'assert a -: zlib_decode_so`[email protected] b=. > zlib_encode_so`[email protected] a=. fread jpath > ''~user/Public/arc/zlib/test/'',y' > smoutput 'time: ',(10{.":t),' data: ', (20{.y),'size: ',(10":#b),' > ratio: ', ":(#b)%(#a) > ) > > f3=: 4 : 0 > 'en de'=. x > t=.6!:2 'assert a -: zlib_decode_so`[email protected] b=. 1 > zlib_encode_so`[email protected] a=. fread jpath > ''~addons/data/ddsqlite/lib/'', y=. ''sqlite3.dll''' > smoutput 'time: ',(10{.":t),' data: ', (20{.y),'size: ',(10":#b),' > ratio: ', ":(#b)%(#a) > t=.6!:2 'assert a -: zlib_decode_so`[email protected] b=. 1 > zlib_encode_so`[email protected] a=. fread jpath > ''~addons/math/lapack/'', y=. ''jlapack.dll''' > smoutput 'time: ',(10{.":t),' data: ', (20{.y),'size: ',(10":#b),' > ratio: ', ":(#b)%(#a) > ) > > It looks like J is about 15000 times slower than the compiled code, > and can compress some binary about twice as well, but no significant > improvement on compression ratio for text. I expect that that matches > what you are seeing? > > This looks like an interesting set of potential tradeoffs. > > Thanks, > > -- > Raul > > On Thu, Sep 18, 2014 at 1:21 PM, Raul Miller <[email protected]> wrote: >> That's working. >> >> Or I think it is - I'm waiting for the script to finish, as I type this. >> >> This will obviously take some time. >> >> Thanks, >> >> -- >> Raul >> >> On Thu, Sep 18, 2014 at 12:53 PM, bill lam <[email protected]> wrote: >>> Yes it needs to require arc/zlib. Those text files not attached, >>> also you need to change them , eg >>> >>>> zlib_encode_so`[email protected] a=.fread jpath'~Public/arc/zlib/test/',y >>> to >>>> zlib_encode_so`[email protected] a=.fread jpath'~temp/',y >>> >>> and put those file under your ~temp >>> >>> I attached the other text files for completeness. Please report >>> if they are not attached and I'll send them off-list. Thanks. >>> >>> Also in f4, change the ~Addons to ~addons and check if you got >>> those file, or try any other binary files you like. file size of >>> 1MB should be more than enough because the J script zlib is >>> quite slow. >>> >>> Чт, 18 сен 2014, Raul Miller написал(а): >>>> I see the attachment test script. >>>> >>>> I added require'arc/zlib' to the top, and then ran it, and got an >>>> error which probably means I need to do something different. But I am >>>> not sure what. >>>> >>>> Here's how it looks: >>>> >>>> 13!:1'' >>>> |domain error >>>> *cd[:0] >>>> |zcompress2[0] >>>> | 0= zcompress2 buf;len;y;(#y);x >>>> |zlib_encode_so[0] >>>> | a-:zlib_decode_so`[email protected] b=. >>>> zlib_encode_so`[email protected] a=.fread >>>> jpath'~Public/arc/zlib/test/',y >>>> |f2[:1] >>>> | 0 0 f2'algorithm.txt' >>>> |test[11] >>>> | test'' >>>> |[-100] c:\users\user\j64-803-user\temp\1.ijs >>>> | 0!:0 y[4!:55<'y' >>>> | fn fl >>>> | 0 load y >>>> | load'c:/users/user/j64-803-user/temp/1.ijs' >>>> >>>> Thanks, >>>> >>>> -- >>>> Raul >>>> >>>> >>>> On Thu, Sep 18, 2014 at 11:42 AM, bill lam <[email protected]> wrote: >>>> > IMO the huffman coding part is relatively easily and performance >>>> > is acceptable, it looks like linear in time. >>>> > >>>> > the LZ77 is still very slow compared with zlib.so thus there >>>> > should be room for improvement. Test caes are in public repos >>>> > but not jal addons, I include here for your reference. Thanks. >>>> > >>>> > You may need ot modify some file path, you can replace the 4 or >>>> > 5 text files with any other text files you like, so they are not >>>> > attached. >>>> > >>>> > please let me know if the attachment test script is missing. >>>> > >>>> > After we are happy with zlib, we might try jpeg which is also a >>>> > minefield of patents, unfortunately. >>>> > >>>> > Чт, 18 сен 2014, Raul Miller написал(а): >>>> >> It would be interesting to review the code and your test cases, once >>>> >> you are happy with it. >>>> >> >>>> >> The whole "try again" thing is a good first cut, but there might be >>>> >> other approaches. (Or there might not be.) >>>> >> >>>> >> Thanks, >>>> >> >>>> >> -- >>>> >> Raul >>>> >> >>>> >> >>>> >> On Thu, Sep 18, 2014 at 5:58 AM, bill lam <[email protected]> wrote: >>>> >> > After more tests, I found there is another rule not explicitly >>>> >> > mentioned in rfc -- the maximum bit length is 15 (or 7 in another >>>> >> > tree). When the distribtution of frequency is highly spreaded, >>>> >> > the maximnum from hcode can be 16. In this case, it needs to >>>> >> > increase the level of (non-zero) minimum frequency and try >>>> >> > again, and iterate until the maximum bit length is 15 or smaller. >>>> >> > >>>> >> > -- >>>> >> > regards, >>>> >> > ==================================================== >>>> >> > GPG key 1024D/4434BAB3 2008-08-24 >>>> >> > gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3 >>>> >> > gpg --keyserver subkeys.pgp.net --armor --export 4434BAB3 >>>> >> > ---------------------------------------------------------------------- >>>> >> > For information about J forums see http://www.jsoftware.com/forums.htm >>>> >> ---------------------------------------------------------------------- >>>> >> For information about J forums see http://www.jsoftware.com/forums.htm >>>> > >>>> > -- >>>> > regards, >>>> > ==================================================== >>>> > GPG key 1024D/4434BAB3 2008-08-24 >>>> > gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3 >>>> > gpg --keyserver subkeys.pgp.net --armor --export 4434BAB3 >>>> > >>>> > ---------------------------------------------------------------------- >>>> > For information about J forums see http://www.jsoftware.com/forums.htm >>>> ---------------------------------------------------------------------- >>>> For information about J forums see http://www.jsoftware.com/forums.htm >>> >>> -- >>> regards, >>> ==================================================== >>> GPG key 1024D/4434BAB3 2008-08-24 >>> gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3 >>> gpg --keyserver subkeys.pgp.net --armor --export 4434BAB3 >>> >>> ---------------------------------------------------------------------- >>> For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
