Re: [gentoo-dev] New global USE flag: gzip-dict

2008-12-19 Thread Peter Volkov
В Срд, 17/12/2008 в 16:34 -0800, Donnie Berkholz пишет:
 Is that some huge package that takes an unreasonable amount of time to 
 build or space to install? 

Probably in this case it takes reasonable amount of time...

 If not, this doesn't seem like a very meaningful choice to me.

Well, your questions forced me to do my own investigation of gzip
performance in one real-life scenario which I hope to use really soon. I
took my Neo FreeRunner and tested gzip decompression speed there. Time
to read 10Mb file is about ~ 2.15s. But if I need to read and decompress
it at the same time it takes ~ 9.90s. So this makes times slower read of
compressed files. Since stardict reads all index files on each startup
and currently index files of my dictionaries occupy more than 20M this
makes much longer startup time. I have not checked runtime performance.
It'll be affected too but, probably, not too much since stardict will
decompress only required parts of data. But anyway gzip is not free and
it's better to have it optional.

-- 
Peter.




Re: [gentoo-dev] New global USE flag: gzip-dict

2008-12-19 Thread Ciaran McCreesh
On Fri, 19 Dec 2008 17:40:41 +0300
Peter Volkov p...@gentoo.org wrote:
 Well, your questions forced me to do my own investigation of gzip
 performance in one real-life scenario which I hope to use really
 soon. I took my Neo FreeRunner and tested gzip decompression speed
 there. Time to read 10Mb file is about ~ 2.15s. But if I need to read
 and decompress it at the same time it takes ~ 9.90s. So this makes
 times slower read of compressed files. Since stardict reads all index
 files on each startup and currently index files of my dictionaries
 occupy more than 20M this makes much longer startup time. I have not
 checked runtime performance. It'll be affected too but, probably, not
 too much since stardict will decompress only required parts of data.
 But anyway gzip is not free and it's better to have it optional.

If it reads (and presumably uncompresses) all of them at startup
anyway, what's the point in compressing them at all?

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature


Re: [gentoo-dev] New global USE flag: gzip-dict

2008-12-19 Thread Peter Volkov
В Птн, 19/12/2008 в 14:45 +, Ciaran McCreesh пишет:
 If it reads (and presumably uncompresses) all of them at startup
 anyway, what's the point in compressing them at all?

It makes size smaller: both index and data files are text files so
compression is very effective. All distributions I've checked compress
data files, some compress both data and index. Probably all desktop
users want dictionaries to be compressed because modern cpu's are really
fast in decompression and even on my 4-years old notebook it takes less
then second... But still there are environments where it's better to
keep dictionaries uncompressed. That's why I want to keep this feature
optional.

-- 
Peter.




Re: [gentoo-dev] New global USE flag: gzip-dict

2008-12-19 Thread Ciaran McCreesh
On Fri, 19 Dec 2008 19:56:02 +0300
Peter Volkov p...@gentoo.org wrote:
 В Птн, 19/12/2008 в 14:45 +, Ciaran McCreesh пишет:
  If it reads (and presumably uncompresses) all of them at startup
  anyway, what's the point in compressing them at all?
 
 It makes size smaller: both index and data files are text files so
 compression is very effective. All distributions I've checked compress
 data files, some compress both data and index. Probably all desktop
 users want dictionaries to be compressed because modern cpu's are
 really fast in decompression and even on my 4-years old notebook it
 takes less then second... But still there are environments where it's
 better to keep dictionaries uncompressed. That's why I want to keep
 this feature optional.

But disk space is cheap. How big are the dictionaries? The vim
dictionaries are around half a meg uncompressed, and if you're looking
to save a meg or two in disk space on the kind of system that includes
dictionaries then you're doing something seriously wrong...

Really, all that compression seems to do is save a small amount of
irrelevant disk space, at the cost of requiring more disk space and
memory for a new library and slowing things down to a level that's
unacceptable on some systems. Compression makes sense for network
transfers, backups and file formats that do their own domain specific
compression. Elsewhere? Likely not so much.

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature


Re: [gentoo-dev] New global USE flag: gzip-dict

2008-12-19 Thread Peter Volkov
В Птн, 19/12/2008 в 17:06 +, Ciaran McCreesh пишет:
 But disk space is cheap. How big are the dictionaries? The vim
 dictionaries are around half a meg uncompressed, and if you're looking
 to save a meg or two in disk space on the kind of system that includes
 dictionaries then you're doing something seriously wrong...

Size is times larger. All dictionary data (without index) I have
currently installed occupies 93M in compressed form and uncompressed
it'll take 402M. This does not count dictionaries I'm going to add into
the tree. If I remember correctly all dictionaries I needed from
stardict site took about 1Gbyte (uncompressed). Also some people use
more then two languages and then they'll use more dictionaries.

 Really, all that compression seems to do is save a small amount of
 irrelevant disk space, at the cost of requiring more disk space and
 memory for a new library and slowing things down to a level that's
 unacceptable on some systems. Compression makes sense for network
 transfers, backups and file formats that do their own domain specific
 compression. Elsewhere? Likely not so much.

I agree in general but in this specific case compression does a good
job.

-- 
Peter.




Re: [gentoo-dev] New global USE flag: gzip-dict

2008-12-17 Thread Donnie Berkholz
On 23:06 Tue 16 Dec , Peter Volkov wrote:
 В Втр, 16/12/2008 в 19:27 +, Ciaran McCreesh пишет:
  What's the point of having this as an option at all? Is it really
  something that affects the end user in any way?
 
 The reason is that this feature requires additional dependency on
 app-text/dictd package (to compress dictionary data dictzip program is
 required).

Is that some huge package that takes an unreasonable amount of time to 
build or space to install? If not, this doesn't seem like a very 
meaningful choice to me.

-- 
Thanks,
Donnie

Donnie Berkholz
Developer, Gentoo Linux
Blog: http://dberkholz.wordpress.com


pgpAS33tL2Bty.pgp
Description: PGP signature


[gentoo-dev] New global USE flag: gzip-dict

2008-12-16 Thread Peter Volkov
Hello.

Some time ago I've modified stardict.eclass and added optional
possibility based on 'gzip' USE flag to compress index and dict data
files. But I realized too late that I need to document this USE flag
somewhere, and since it'll do similar things for all stardict-*
dictionaries (heh, more than 5 packages...) I'm going to add it as
global USE flag. Also since gzip USE flag already exist in
x11-misc/openclipart I'll change 'gzip' to 'gzip-dict'. So if there will
be no objections I'll add new 'gzip-dict' global USE flag in 2-3 days
from now.

-- 
Peter.




Re: [gentoo-dev] New global USE flag: gzip-dict

2008-12-16 Thread Ciaran McCreesh
On Tue, 16 Dec 2008 22:21:16 +0300
Peter Volkov p...@gentoo.org wrote:
 Some time ago I've modified stardict.eclass and added optional
 possibility based on 'gzip' USE flag to compress index and dict data
 files. But I realized too late that I need to document this USE flag
 somewhere, and since it'll do similar things for all stardict-*
 dictionaries (heh, more than 5 packages...) I'm going to add it as
 global USE flag. Also since gzip USE flag already exist in
 x11-misc/openclipart I'll change 'gzip' to 'gzip-dict'. So if there
 will be no objections I'll add new 'gzip-dict' global USE flag in 2-3
 days from now.

What's the point of having this as an option at all? Is it really
something that affects the end user in any way? Or is it just
gratuitous choisiosity?

-- 
Ciaran McCreesh


signature.asc
Description: PGP signature


Re: [gentoo-dev] New global USE flag: gzip-dict

2008-12-16 Thread Doug Goldstein

Ciaran McCreesh wrote:

On Tue, 16 Dec 2008 22:21:16 +0300
Peter Volkov p...@gentoo.org wrote:
  

Some time ago I've modified stardict.eclass and added optional
possibility based on 'gzip' USE flag to compress index and dict data
files. But I realized too late that I need to document this USE flag
somewhere, and since it'll do similar things for all stardict-*
dictionaries (heh, more than 5 packages...) I'm going to add it as
global USE flag. Also since gzip USE flag already exist in
x11-misc/openclipart I'll change 'gzip' to 'gzip-dict'. So if there
will be no objections I'll add new 'gzip-dict' global USE flag in 2-3
days from now.



What's the point of having this as an option at all? Is it really
something that affects the end user in any way? Or is it just
gratuitous choisiosity?

  
I happen to be in agreement here. gzip is a quick process, especially 
with a separate index file which would point to a specific section in 
the dict to uncompress. Assuming they've coded it right, it should 
barely be noticeable in the grand scheme of things.


If this is not the case at all and it in fact for some odd reasons 
requires additional deps and requires uncompressing huge files in memory 
such that low memory systems can't handle it, then I'd be in favor of a 
USE flag. But otherwise, it seems like less maintenance for you and less 
user confusion by making it default.




Re: [gentoo-dev] New global USE flag: gzip-dict

2008-12-16 Thread Peter Volkov
В Втр, 16/12/2008 в 19:27 +, Ciaran McCreesh пишет:
 What's the point of having this as an option at all? Is it really
 something that affects the end user in any way?

The reason is that this feature requires additional dependency on
app-text/dictd package (to compress dictionary data dictzip program is
required).

-- 
Peter.