I've modified your code to show timings: f1=: 4 : 0 a=. ".y 'en de'=. x t=.6!:2 'assert a -: zlib_decode_so`[email protected] b=. zlib_encode_so`[email protected] a' smoutput 'time: ',(10{.":t),' data: ', (20{.y),'size: ',(10":#b),' ratio: ', ":(#b)%(#a) )
f2=: 4 : 0 'en de'=. x t=.6!:2 'assert a -: zlib_decode_so`[email protected] b=. zlib_encode_so`[email protected] a=. fread jpath ''~user/Public/arc/zlib/test/'',y' smoutput 'time: ',(10{.":t),' data: ', (20{.y),'size: ',(10":#b),' ratio: ', ":(#b)%(#a) ) f3=: 4 : 0 'en de'=. x t=.6!:2 'assert a -: zlib_decode_so`[email protected] b=. 1 zlib_encode_so`[email protected] a=. fread jpath ''~addons/data/ddsqlite/lib/'', y=. ''sqlite3.dll''' smoutput 'time: ',(10{.":t),' data: ', (20{.y),'size: ',(10":#b),' ratio: ', ":(#b)%(#a) t=.6!:2 'assert a -: zlib_decode_so`[email protected] b=. 1 zlib_encode_so`[email protected] a=. fread jpath ''~addons/math/lapack/'', y=. ''jlapack.dll''' smoutput 'time: ',(10{.":t),' data: ', (20{.y),'size: ',(10":#b),' ratio: ', ":(#b)%(#a) ) It looks like J is about 15000 times slower than the compiled code, and can compress some binary about twice as well, but no significant improvement on compression ratio for text. I expect that that matches what you are seeing? This looks like an interesting set of potential tradeoffs. Thanks, -- Raul On Thu, Sep 18, 2014 at 1:21 PM, Raul Miller <[email protected]> wrote: > That's working. > > Or I think it is - I'm waiting for the script to finish, as I type this. > > This will obviously take some time. > > Thanks, > > -- > Raul > > On Thu, Sep 18, 2014 at 12:53 PM, bill lam <[email protected]> wrote: >> Yes it needs to require arc/zlib. Those text files not attached, >> also you need to change them , eg >> >>> zlib_encode_so`[email protected] a=.fread jpath'~Public/arc/zlib/test/',y >> to >>> zlib_encode_so`[email protected] a=.fread jpath'~temp/',y >> >> and put those file under your ~temp >> >> I attached the other text files for completeness. Please report >> if they are not attached and I'll send them off-list. Thanks. >> >> Also in f4, change the ~Addons to ~addons and check if you got >> those file, or try any other binary files you like. file size of >> 1MB should be more than enough because the J script zlib is >> quite slow. >> >> Чт, 18 сен 2014, Raul Miller написал(а): >>> I see the attachment test script. >>> >>> I added require'arc/zlib' to the top, and then ran it, and got an >>> error which probably means I need to do something different. But I am >>> not sure what. >>> >>> Here's how it looks: >>> >>> 13!:1'' >>> |domain error >>> *cd[:0] >>> |zcompress2[0] >>> | 0= zcompress2 buf;len;y;(#y);x >>> |zlib_encode_so[0] >>> | a-:zlib_decode_so`[email protected] b=. >>> zlib_encode_so`[email protected] a=.fread >>> jpath'~Public/arc/zlib/test/',y >>> |f2[:1] >>> | 0 0 f2'algorithm.txt' >>> |test[11] >>> | test'' >>> |[-100] c:\users\user\j64-803-user\temp\1.ijs >>> | 0!:0 y[4!:55<'y' >>> | fn fl >>> | 0 load y >>> | load'c:/users/user/j64-803-user/temp/1.ijs' >>> >>> Thanks, >>> >>> -- >>> Raul >>> >>> >>> On Thu, Sep 18, 2014 at 11:42 AM, bill lam <[email protected]> wrote: >>> > IMO the huffman coding part is relatively easily and performance >>> > is acceptable, it looks like linear in time. >>> > >>> > the LZ77 is still very slow compared with zlib.so thus there >>> > should be room for improvement. Test caes are in public repos >>> > but not jal addons, I include here for your reference. Thanks. >>> > >>> > You may need ot modify some file path, you can replace the 4 or >>> > 5 text files with any other text files you like, so they are not >>> > attached. >>> > >>> > please let me know if the attachment test script is missing. >>> > >>> > After we are happy with zlib, we might try jpeg which is also a >>> > minefield of patents, unfortunately. >>> > >>> > Чт, 18 сен 2014, Raul Miller написал(а): >>> >> It would be interesting to review the code and your test cases, once >>> >> you are happy with it. >>> >> >>> >> The whole "try again" thing is a good first cut, but there might be >>> >> other approaches. (Or there might not be.) >>> >> >>> >> Thanks, >>> >> >>> >> -- >>> >> Raul >>> >> >>> >> >>> >> On Thu, Sep 18, 2014 at 5:58 AM, bill lam <[email protected]> wrote: >>> >> > After more tests, I found there is another rule not explicitly >>> >> > mentioned in rfc -- the maximum bit length is 15 (or 7 in another >>> >> > tree). When the distribtution of frequency is highly spreaded, >>> >> > the maximnum from hcode can be 16. In this case, it needs to >>> >> > increase the level of (non-zero) minimum frequency and try >>> >> > again, and iterate until the maximum bit length is 15 or smaller. >>> >> > >>> >> > -- >>> >> > regards, >>> >> > ==================================================== >>> >> > GPG key 1024D/4434BAB3 2008-08-24 >>> >> > gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3 >>> >> > gpg --keyserver subkeys.pgp.net --armor --export 4434BAB3 >>> >> > ---------------------------------------------------------------------- >>> >> > For information about J forums see http://www.jsoftware.com/forums.htm >>> >> ---------------------------------------------------------------------- >>> >> For information about J forums see http://www.jsoftware.com/forums.htm >>> > >>> > -- >>> > regards, >>> > ==================================================== >>> > GPG key 1024D/4434BAB3 2008-08-24 >>> > gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3 >>> > gpg --keyserver subkeys.pgp.net --armor --export 4434BAB3 >>> > >>> > ---------------------------------------------------------------------- >>> > For information about J forums see http://www.jsoftware.com/forums.htm >>> ---------------------------------------------------------------------- >>> For information about J forums see http://www.jsoftware.com/forums.htm >> >> -- >> regards, >> ==================================================== >> GPG key 1024D/4434BAB3 2008-08-24 >> gpg --keyserver subkeys.pgp.net --recv-keys 4434BAB3 >> gpg --keyserver subkeys.pgp.net --armor --export 4434BAB3 >> >> ---------------------------------------------------------------------- >> For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
