Re: GZip{In,Out}putStream in GIO?
On Sat, 2009-08-15 at 10:07 -0400, Morten Welinder wrote: FWIW, Sugar uses zip quite extensively to bundle content and software and we would love to move from using python's zipfile to something glib-based. Why all this reinvent-the-wheel effort? libgsf gives you access to zipfiles and is glib based right now. Does libgsf use the GIO APIs? Ideally, people should just be able to write their programs to expect a GInputStream. And then if some other library happens to be providing a FooRot13InputStream, it doesn't matter. I understand that libgsf predates GIO considerably. For my purposes, if I needed to deal with complicated structured files, I'd certainly look to libgsf. But all I need to do is decompress a file and read it. Right now, it took all of 231 lines of C to do it in Yelp, and most of that is boilerplate. It's just not worth an extra dependency for me. -- Shaun ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
Does libgsf use the GIO APIs? It can take a GFile* and treat that as a zipfile. The way you would read from the zipfile member files would not be GFile* based. For the output side, turn all that around. Does that answer your question? M. ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Fri, Aug 14, 2009 at 22:38, Shaun McCancesha...@gnome.org wrote:
2009/7/31 Brian J. Tarriconebj...@cornell.edu:
On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote:
From the looks of it, it should be straight forward to write
GZip{In,Out}putStream classes based on zlib
I'd say call it GCompressed{In,Out}putStream and have it either
auto-detect
the compression type, or have a param in the API to specify from an enum
(or
both). People can add support for other types of compression as time goes
on.
It's nice to see I'm not the only person that's after this.
I'm not sure what other people need this for, but I thought
I'd give a data point for why it's useful to me.
Yelp currently has its own subclass of GIOChannel that does
decompression on the fly for gzip, bzip2, and lzma.
http://git.gnome.org/cgit/yelp/tree/src/yelp-io-channel.c
It's not a particularly elegant solution. It has everything
in one file, with bzip2 and lzma functionality in #ifdefs,
and determines what to do by file extension. But it does the
job for Yelp, which needs to read man and info pages that may
be compressed.
(It's also a fairly trivial and not-complete implementation.
For instance, it doesn't do seeking, because Yelp doesn't
need it.)
I've been looking at converting Yelp over to GIO, so having
such an InputStream would be very useful to me. If it's not
in GLib, I'll have to do my own.
I'd really need all three of the above compression schemes to
work. If a gzip-only InputStream is done, I can always ship
my own bzip2 and lzma implementations.
A flexible system using runtime-loadable backends is fine for me,
as long as I know bzip2 and lzma implementations are available.
I don't particularly care about having a guarantee that they're
there. Distributions that ship lzma-compressed man pages can
just make sure the backend is installed. Not my problem.
And yes, this is me volunteering to do the work, but not until
I get around to gutting Yelp, which will be in maybe two months.
I'll have to do the work anyway for Yelp. It would be nice to
get input from the maintainers about what approach they think
is best.
FWIW, Sugar uses zip quite extensively to bundle content and software
and we would love to move from using python's zipfile to something
glib-based.
Have no resources to offer, but I will tell people interested on
working on bundles to consider helping out with this.
Regards,
Tomeu
--
«Sugar Labs is anyone who participates in improving and using Sugar.
What Sugar Labs does is determined by the participants.» - David
Farning
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
FWIW, Sugar uses zip quite extensively to bundle content and software and we would love to move from using python's zipfile to something glib-based. Why all this reinvent-the-wheel effort? libgsf gives you access to zipfiles and is glib based right now. Morten ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Sat, Aug 15, 2009 at 16:07, Morten Welindermort...@gnome.org wrote: FWIW, Sugar uses zip quite extensively to bundle content and software and we would love to move from using python's zipfile to something glib-based. Why all this reinvent-the-wheel effort? libgsf gives you access to zipfiles and is glib based right now. I would prefer if this functionality was provided in glib itself because of support costs. Regards, Tomeu ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/7/31 Brian J. Tarriconebj...@cornell.edu:
On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote:
From the looks of it, it should be straight forward to write
GZip{In,Out}putStream classes based on zlib
I'd say call it GCompressed{In,Out}putStream and have it either auto-detect
the compression type, or have a param in the API to specify from an enum
(or
both). People can add support for other types of compression as time goes
on.
It's nice to see I'm not the only person that's after this.
I'm not sure what other people need this for, but I thought
I'd give a data point for why it's useful to me.
Yelp currently has its own subclass of GIOChannel that does
decompression on the fly for gzip, bzip2, and lzma.
http://git.gnome.org/cgit/yelp/tree/src/yelp-io-channel.c
It's not a particularly elegant solution. It has everything
in one file, with bzip2 and lzma functionality in #ifdefs,
and determines what to do by file extension. But it does the
job for Yelp, which needs to read man and info pages that may
be compressed.
(It's also a fairly trivial and not-complete implementation.
For instance, it doesn't do seeking, because Yelp doesn't
need it.)
I've been looking at converting Yelp over to GIO, so having
such an InputStream would be very useful to me. If it's not
in GLib, I'll have to do my own.
I'd really need all three of the above compression schemes to
work. If a gzip-only InputStream is done, I can always ship
my own bzip2 and lzma implementations.
A flexible system using runtime-loadable backends is fine for me,
as long as I know bzip2 and lzma implementations are available.
I don't particularly care about having a guarantee that they're
there. Distributions that ship lzma-compressed man pages can
just make sure the backend is installed. Not my problem.
And yes, this is me volunteering to do the work, but not until
I get around to gutting Yelp, which will be in maybe two months.
I'll have to do the work anyway for Yelp. It would be nice to
get input from the maintainers about what approach they think
is best.
--
Shaun
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
Bastien Nocera wrote: I could think of at least 5 types of compressions that would be useful to have without having to use a command-line tool to decompress: - gzip for anything and everything that can come from a web server (in my case, iTunes Music Store playlist parsing, or more widely, GOffice file formats) - zip for OpenOffice.org documents and Comic Books (evince) - 7zip/LZMA/Xz formats for Comic Books - Rar for the same as above Aren't there two classes of file types there ? Compression vs Archiving I mean, zip, 7z, and rar are archiving format who store files in a compressed fashion (kind of like a tar of gzipped files) so rather than just having a stream you need to have some support for archives there, and not just the compression part like gzip or bzip2... ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/8/3 Steve Frécinaux nudr...@gmail.com: Bastien Nocera wrote: I could think of at least 5 types of compressions that would be useful to have without having to use a command-line tool to decompress: - gzip for anything and everything that can come from a web server (in my case, iTunes Music Store playlist parsing, or more widely, GOffice file formats) - zip for OpenOffice.org documents and Comic Books (evince) - 7zip/LZMA/Xz formats for Comic Books - Rar for the same as above Aren't there two classes of file types there ? Compression vs Archiving I mean, zip, 7z, and rar are archiving format who store files in a compressed fashion (kind of like a tar of gzipped files) so rather than just having a stream you need to have some support for archives there, and not just the compression part like gzip or bzip2... I wonder if we should split this in two approaches, having gvfs modules for archives (compressed or not) and gio-loaders for compression only formats (gzip and bzip2). -- Un saludo, Alberto Ruiz ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Mon, 03 Aug 2009 11:37:53 +0200, Steve Frécinaux wrote: I mean, zip, 7z, and rar are archiving format who store files in a compressed fashion (kind of like a tar of gzipped files) so rather than just having a stream you need to have some support for archives there, and not just the compression part like gzip or bzip2... 7zip's LZMA algorithm is implemented in xz-utils (previously called lzma-utils) in the same way as gzip and bzip2 work. Also, 7-zip itself can be used (with some limitations) in streamed mode on *nix. -- Jernej Simončič http://eternallybored.org/ ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
From: Steve Fr-23;cinaux, Date: 03/08/2009 19:38, Wrote: Bastien Nocera wrote: I could think of at least 5 types of compressions that would be useful to have without having to use a command-line tool to decompress: - gzip for anything and everything that can come from a web server (in my case, iTunes Music Store playlist parsing, or more widely, GOffice file formats) - zip for OpenOffice.org documents and Comic Books (evince) - 7zip/LZMA/Xz formats for Comic Books - Rar for the same as above Aren't there two classes of file types there ? Compression vs Archiving Archiving formats would be better supported by GVFS, wouldn't they...? Treating an archive as a virtual directory. However, zip has a few compression formats, rar compression can be applied both before and after bundling of the files (a rar solid archive is similar to a tar.gz archive). There may be some value to supporting those compression formats even without supporting the entire archive format. The compression is pretty much the hard part of handling these archives. Going the extra step further, though, the next obvious idea would be to do the same thing that gunzip does when you concatenate together a series of gzipped files - it simply returns the uncompressed content of each file sequentially. I guess, though, you still need to understand the archive format, for at least some of those formats. So I suppose some kind of archive handler would be needed to pick out the individual streams for concatenation. Given that, reading a zip file as a compressed stream could instantiate a concatenation object that sets up the appropriate archive handler and just iterates over the available files, joining the uncompressed data of each individual decompression stream together as a single continuous stream. Which would give you the option to set up the archive handler yourself and extract the one(s) you want individually. This would be a boon even for gzipped files, giving you the option to extract concatenated compressed streams individually rather than as a solid indistinguishable stream. The ability to manipulate archives as well as simple compression would be fantastic. I did at one stage need to apply a filter to the files matching a certain mask within a tgz archive. With support like that, I could uncompress the tgz, split the tar, pass through files I'm not interested in, apply the filtering to the ones I am, and then repackage and re-compress the lot on the way out, without having to mess around with spawning external commands and correctly handling the myriad of things that can go wrong. Fredderic Worried about your move? Relocation Services can do the work for you. Click Here! Relocation Services http://tagline.excite.com/fc/FgElN1gwiSlWpkKFMLvDoO0n7NxKyoB5FLHN0tUNYVebcYZiatk3W9RdaIg/___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Mon, 2009-08-03 at 07:58 -0400, Freddie Unpenstein wrote: From: Steve Fr-23;cinaux, Date: 03/08/2009 19:38, Wrote: Bastien Nocera wrote: I could think of at least 5 types of compressions that would be useful to have without having to use a command-line tool to decompress: - gzip for anything and everything that can come from a web server (in my case, iTunes Music Store playlist parsing, or more widely, GOffice file formats) - zip for OpenOffice.org documents and Comic Books (evince) - 7zip/LZMA/Xz formats for Comic Books - Rar for the same as above Aren't there two classes of file types there ? Compression vs Archiving Archiving formats would be better supported by GVFS, wouldn't they...? Treating an archive as a virtual directory. However, zip has a few compression formats, rar compression can be applied both before and after bundling of the files (a rar solid archive is similar to a tar.gz archive). There may be some value to supporting those compression formats even without supporting the entire archive format. The compression is pretty much the hard part of handling these archives. Going the extra step further, though, the next obvious idea would be to do the same thing that gunzip does when you concatenate together a series of gzipped files - it simply returns the uncompressed content of each file sequentially. I guess, though, you still need to understand the archive format, for at least some of those formats. So I suppose some kind of archive handler would be needed to pick out the individual streams for concatenation. Given that, reading a zip file as a compressed stream could instantiate a concatenation object that sets up the appropriate archive handler and just iterates over the available files, joining the uncompressed data of each individual decompression stream together as a single continuous stream. Which would give you the option to set up the archive handler yourself and extract the one(s) you want individually. This would be a boon even for gzipped files, giving you the option to extract concatenated compressed streams individually rather than as a solid indistinguishable stream. The ability to manipulate archives as well as simple compression would be fantastic. I did at one stage need to apply a filter to the files matching a certain mask within a tgz archive. With support like that, I could uncompress the tgz, split the tar, pass through files I'm not interested in, apply the filtering to the ones I am, and then repackage and re-compress the lot on the way out, without having to mess around with spawning external commands and correctly handling the myriad of things that can go wrong. ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Mon, 2009-08-03 at 07:58 -0400, Freddie Unpenstein wrote: Archiving formats would be better supported by GVFS, wouldn't they...? Treating an archive as a virtual directory. We already have gvfsd-archive for some time, based on libarchive, able to handle TAR and ZIP archiving formats with GZIP, BZ2 compression, easily extendable to LZMA and XZ (available from libarchive, not yet enabled by default in gvfs). However, zip has a few compression formats, rar compression can be applied both before and after bundling of the files (a rar solid archive is similar to a tar.gz archive). Remember we can't build anything around RAR format due to incompatible licensing, except of the file-roller way (spawning external commands). I guess, though, you still need to understand the archive format, for at least some of those formats. So I suppose some kind of archive handler would be needed to pick out the individual streams for concatenation. Given that, reading a zip file as a compressed stream could instantiate a concatenation object that sets up the appropriate archive handler and just iterates over the available files, joining the uncompressed data of each individual decompression stream together as a single continuous stream. Which would give you the option to set up the archive handler yourself and extract the one(s) you want individually. This would be a boon even for gzipped files, giving you the option to extract concatenated compressed streams individually rather than as a solid indistinguishable stream. libarchive can be easily used as a filter between two streams, transparently decoding incoming data. This should be really implemented as an extension, we don't want to have another dependency in glib. P.S.: sorry for the previous empty e-mail, key shortcuts are sometimes dangerous in Evolution... -- Tomas Bzatek tbza...@redhat.com ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/8/3 Freddie Unpenstein fredde...@excite.com: From: Steve Fr-23;cinaux, Date: 03/08/2009 19:38, Wrote: Bastien Nocera wrote: I could think of at least 5 types of compressions that would be useful to have without having to use a command-line tool to decompress: - gzip for anything and everything that can come from a web server (in my case, iTunes Music Store playlist parsing, or more widely, GOffice file formats) - zip for OpenOffice.org documents and Comic Books (evince) - 7zip/LZMA/Xz formats for Comic Books - Rar for the same as above Aren't there two classes of file types there ? Compression vs Archiving Archiving formats would be better supported by GVFS, wouldn't they...? Treating an archive as a virtual directory. From an end user perspective I can see the value in this abstraction - being able to browse into archieves, however from a developer perspective I think I would mostly find it as an annoying abstraction. I'm pretty confident that in 98%[1] of the use cases the developer needs to know that this is really about an on-disk file and not a directory. Bear in mind that I would need to do a g_file_enumerate_children() and open streams on nested URIs I get from the GFileInfos returned by the GFileEnumerator. From a programmatic perspective I like the Java approach much better. Abstracting the ZipInputStream to general archives should be trivial (and making it more GIO-stylish is also a task for the reader): InputStream rawStream = new FileInputStream(/tmp/foo.zip); ZipInputStream zip = new ZipInputStream(rawStream); ZipEntry entry = zip.nextEntry(); System.out.println(Got zip entry: + entry.getPath()); InputStream nestedStream = entry.getStream(); System.out.println(Entry contents: + nestedStream.readTheWholeBloodyStream()); // Do the zip.nextEntry() entry dance until we have read all entries -- Cheers, Mikkel [1]: And because I include arbitrary made up statistics I automatically loose this argument ;-) ___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
From: Mikkel Kamstrup Erlandsen, Date: 04/08/2009 00:22, Wrote: On Mon, 2009-08-03 at 07:58 -0400, Freddie Unpenstein wrote: Archiving formats would be better supported by GVFS, wouldn't they...? Treating an archive as a virtual directory. I'm pretty confident that in 98%[1] of the use cases the developer needs to know that this is really about an on-disk file and not a directory. Bear in mind that I would need to do a g_file_enumerate_children() and open streams on nested URIs I get from the GFileInfos returned by the GFileEnumerator. I understand this... I don't know the details, but I had a suspicion it would be a pain. However, it may be possible to abstract some of that away to return a simple list of names, or the stream corresponding to a given name...? However, zip has a few compression formats, rar compression can be applied both before and after bundling of the files (a rar solid archive is similar to a tar.gz archive). Remember we can't build anything around RAR format due to incompatible licensing, except of the file-roller way (spawning external commands). It was more an example than anything... I have heard there are licencing issues with RAR... It's the fact that the compression can be on either side of the archive structure that I thought worth bringing up... Same as for tar/(g|b)zip... You can quite conceivably have a tar archive of gzipped files. libarchive can be easily used as a filter between two streams, transparently decoding incoming data. This should be really implemented as an extension, we don't want to have another dependency in glib. That sounds fair enough... A quick glance over at libarchive, and it seems to do pretty much everything required (can't tell if it handles concatenated gzipped files). But embracing it looks like duplicating a fair chunk of functionality. I wonder if it would be better to do it over in a more GLib/GIO-esque style than trying to embrace it. GIO streams can completely replace the I/O layers at both ends of the libarchive engine, while compression and at least some of the archive format handling is done by external libraries. It really doesn't look like there's much left for libarchive to do, apart from format detection, at least as far as I could tell from my quick look over the documentation. Anyhow... Wish I'd known about libarchive a while back. I really have to practice my Google technique... :( I do, though, wish there was a compression library that was able to snapshot an in-progress compression, allowing you to rewind to a saved state and continue from there. Fredderic Toupee Find toupees to help you look your best! Click now! http://tagline.excite.com/fc/FgElN1g05cL012R3SRh8CWFbYT1xuTXgkfM2tibgFh5Lo8J77cLU8SCTbBO/___ gtk-devel-list mailing list gtk-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
Am Fri, 31 Jul 2009 15:07:10 -0700
schrieb Brian J. Tarricone bj...@cornell.edu:
On 07/31/2009 01:59 PM, Mikkel Kamstrup Erlandsen wrote:
2009/7/31 Brian J. Tarriconebj...@cornell.edu:
On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote:
From the looks of it, it should be straight forward to write
GZip{In,Out}putStream classes based on zlib
I'd say call it GCompressed{In,Out}putStream and have it either
auto-detect the compression type, or have a param in the API to
specify from an enum (or both). People can add support for other
types of compression as time goes on.
The prime benefit of having dedicated classes over your approach is
having syntactically ensured that you are indeed working with
GZipped buffers. Personally I like that, but this is of course 100%
subjective.
Well, sure, but otherwise you can end up with classes for gzip,
bzip2, zip, 7zip, rar, etc. That alone is 10 extra classes, and I'm
sure there are other compression schemes people might want classes
for. That sounds messy to me.
Having a constructor for a generic class that takes a parameter for
the compression type would give you what you want, assuming it's set
up so it returns an error if the content you pass it doesn't appear
to be of the selected type.
I guess it doesn't really matter either way. You could even
implement GCompressedInputStream as a sort of class cluster that
returns a (private) subclass depending on the compression type, or
have all (public) classes be a subclass of a
GCompressed{In,Out}putStream class, and for all operations (except
for construction) you'd call methods on the superclass.
And I think there's a benefit to support format auto-detection if the
developer doesn't care what format the input is in. That's possibly
a little more difficult to do with explicit subclasses, though the
class-cluster method would still work and yet maintain separate
classes for each compression format.
I like the idea of a generic class and only supplying a MIME type, so
that different compression implementations can go in a separate package
just like GVfs backends today. It makes me think of gzip.GzipFile in
Python which I like because it lets you use a compressed file just like
a plain file, albeit I miss the autodetection there, so I second having
that feature in GCompressedInputStream.
Just my 2 cents,
Christian
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/8/1 Bastien Nocera had...@hadess.net:
On Fri, 2009-07-31 at 21:17 +0100, Alberto Ruiz wrote:
2009/7/31 Brian J. Tarricone bj...@cornell.edu:
On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote:
From the looks of it, it should be straight forward to write
GZip{In,Out}putStream classes based on zlib
I'd say call it GCompressed{In,Out}putStream and have it either auto-detect
the compression type, or have a param in the API to specify from an enum
(or
both). People can add support for other types of compression as time goes
on.
And expand the list of glib dependencies to the infinite?
I don't think I like such an idea at this level of the API.
You could claim the same about the gdk-pixbuf loaders, or the gvfs
backends. The point is that you'd want to make your original class
flexible enough that people could extend upon it (even if you don't
start off with a plugin API).
I could think of at least 5 types of compressions that would be useful
to have without having to use a command-line tool to decompress:
- gzip for anything and everything that can come from a web server (in
my case, iTunes Music Store playlist parsing, or more widely, GOffice
file formats)
- zip for OpenOffice.org documents and Comic Books (evince)
- 7zip/LZMA/Xz formats for Comic Books
- Rar for the same as above
I'd be happy seeing at least the first one, very happy to see the first
3, that all have open source implementations, and would provide more
robust support for those file types in applications.
Ok, taking Brian's ideas into consideration, how about this:
GCompressedInputStream (extrapolate to output streams yourself) is an
abstract base class. It has static factories:
GCompressedInputStream*
g_compressed_inputstream_new_for_type (const gchar* type_name);
GCompressedInputStream*
g_compressed_inputstream_new_for_stream (GInputStream* in);
We have concrete classes for the most used compressions publicly
available, eg. GZip, Zip[1] (and BZip2?). Other compressions can be
registered via GIO extension points (one for compressors and one for
decompressors).
--
Cheers,
Mikkel
[1]: Modulo the fact that Zip files can't be parsed as simple streams.
Maybe a GArchive*putStream base class is needed.
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Sat, Aug 1, 2009 at 1:54 PM, Mikkel Kamstrup
Erlandsenmikkel.kamst...@gmail.com wrote:
2009/8/1 Bastien Nocera had...@hadess.net:
On Fri, 2009-07-31 at 21:17 +0100, Alberto Ruiz wrote:
2009/7/31 Brian J. Tarricone bj...@cornell.edu:
On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote:
From the looks of it, it should be straight forward to write
GZip{In,Out}putStream classes based on zlib
I'd say call it GCompressed{In,Out}putStream and have it either
auto-detect
the compression type, or have a param in the API to specify from an enum
(or
both). People can add support for other types of compression as time goes
on.
And expand the list of glib dependencies to the infinite?
I don't think I like such an idea at this level of the API.
You could claim the same about the gdk-pixbuf loaders, or the gvfs
backends. The point is that you'd want to make your original class
flexible enough that people could extend upon it (even if you don't
start off with a plugin API).
I could think of at least 5 types of compressions that would be useful
to have without having to use a command-line tool to decompress:
- gzip for anything and everything that can come from a web server (in
my case, iTunes Music Store playlist parsing, or more widely, GOffice
file formats)
- zip for OpenOffice.org documents and Comic Books (evince)
- 7zip/LZMA/Xz formats for Comic Books
- Rar for the same as above
I'd be happy seeing at least the first one, very happy to see the first
3, that all have open source implementations, and would provide more
robust support for those file types in applications.
Ok, taking Brian's ideas into consideration, how about this:
GCompressedInputStream (extrapolate to output streams yourself) is an
abstract base class. It has static factories:
GCompressedInputStream*
g_compressed_inputstream_new_for_type (const gchar* type_name);
GCompressedInputStream*
g_compressed_inputstream_new_for_stream (GInputStream* in);
More like
GInputStream *
g_compressed_input_stream_new (GInputStream *in);
g_compressed_input_stream_new_for_type(GInputStream *in, const gchar
*type) isn't a bad idea either, since you could specify the type at
construction rather than force mime auto-detection which could be a
pain (thinking network streams). Probably should do better than _new
if we go the interface route. This is how all of the other filter
streams are specified in GIO, IMO it's pointless to diverge much here.
Also wonder if it makes sense as a base class or whether it should be
an interface which G(G,B)Zip(I,O)Stream implements with the actual
implementations hidden like GLocalFile; what public methods would
those have anyways? (Not rhetorical, curious).
Also, Zip is probably not a good idea since you'd have to deal with
virtual files and crypto. Makes better sense as a GVFS backend I
think.
-A. Walton
We have concrete classes for the most used compressions publicly
available, eg. GZip, Zip[1] (and BZip2?). Other compressions can be
registered via GIO extension points (one for compressors and one for
decompressors).
--
Cheers,
Mikkel
[1]: Modulo the fact that Zip files can't be parsed as simple streams.
Maybe a GArchive*putStream base class is needed.
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/8/1 A. Walton awal...@gnome.org:
On Sat, Aug 1, 2009 at 1:54 PM, Mikkel Kamstrup
Erlandsenmikkel.kamst...@gmail.com wrote:
2009/8/1 Bastien Nocera had...@hadess.net:
On Fri, 2009-07-31 at 21:17 +0100, Alberto Ruiz wrote:
2009/7/31 Brian J. Tarricone bj...@cornell.edu:
On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote:
From the looks of it, it should be straight forward to write
GZip{In,Out}putStream classes based on zlib
I'd say call it GCompressed{In,Out}putStream and have it either
auto-detect
the compression type, or have a param in the API to specify from an enum
(or
both). People can add support for other types of compression as time
goes
on.
And expand the list of glib dependencies to the infinite?
I don't think I like such an idea at this level of the API.
You could claim the same about the gdk-pixbuf loaders, or the gvfs
backends. The point is that you'd want to make your original class
flexible enough that people could extend upon it (even if you don't
start off with a plugin API).
I could think of at least 5 types of compressions that would be useful
to have without having to use a command-line tool to decompress:
- gzip for anything and everything that can come from a web server (in
my case, iTunes Music Store playlist parsing, or more widely, GOffice
file formats)
- zip for OpenOffice.org documents and Comic Books (evince)
- 7zip/LZMA/Xz formats for Comic Books
- Rar for the same as above
I'd be happy seeing at least the first one, very happy to see the first
3, that all have open source implementations, and would provide more
robust support for those file types in applications.
Ok, taking Brian's ideas into consideration, how about this:
GCompressedInputStream (extrapolate to output streams yourself) is an
abstract base class. It has static factories:
GCompressedInputStream*
g_compressed_inputstream_new_for_type (const gchar* type_name);
GCompressedInputStream*
g_compressed_inputstream_new_for_stream (GInputStream* in);
More like
GInputStream *
g_compressed_input_stream_new (GInputStream *in);
g_compressed_input_stream_new_for_type(GInputStream *in, const gchar
*type)
Right. I was a bit fast to hit send there :-) And the
Compression*Stream would extend the Filter*Stream base classes.
SNIP
... interface which G(G,B)Zip(I,O)Stream implements with the actual
implementations hidden like GLocalFile; what public methods would
those have anyways? (Not rhetorical, curious).
Compression level for GZip comes to mind. There are probably other parameters.
I think that the static factory approach hiding the actual impl. is
good, but I would also like to expose a GZip*Stream directly.
Also, Zip is probably not a good idea since you'd have to deal with
virtual files and crypto. Makes better sense as a GVFS backend I
think.
Yeah, archive files require another approach. When we have finished
the pure compression formats we might also have a better idea whether
or not it makes sense to create a similar framework for the archive
formats.
--
Cheers,
Mikkel
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
GZip{In,Out}putStream in GIO?
Hi,
I've been eye balling the GIO docs for a while without finding
in-/output for gzip compression... So if I missed it stop me now :-)
From the looks of it, it should be straight forward to write
GZip{In,Out}putStream classes based on zlib[1]. If I write these
classes would a zlib dep. be OK for GIO? It is pretty ubiquitous as
far as I can tell (and it is already a depency of Gtk+)... And I am
not really all that keen on writing my own gzip/DEFLATE implementation
(to say the least).
Another thing is the naming. Since this is not about supporting the
Zip archive format (but that would be nice too) the name GZip*Stream
might be bad. On the other hand i abhor the ALLCAPS GZIPInputStream
used in fx. Java. What about GGZip*Stream - eeek!?
--
Cheers,
Mikkel
[1]: http://zlib.net/
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Fri, Jul 31, 2009 at 02:48:47PM +0200, Mikkel Kamstrup Erlandsen wrote:
Hi,
I've been eye balling the GIO docs for a while without finding
in-/output for gzip compression... So if I missed it stop me now :-)
From the looks of it, it should be straight forward to write
GZip{In,Out}putStream classes based on zlib[1]. If I write these
classes would a zlib dep. be OK for GIO? It is pretty ubiquitous as
far as I can tell (and it is already a depency of Gtk+)... And I am
not really all that keen on writing my own gzip/DEFLATE implementation
(to say the least).
Another thing is the naming. Since this is not about supporting the
Zip archive format (but that would be nice too) the name GZip*Stream
might be bad. On the other hand i abhor the ALLCAPS GZIPInputStream
used in fx. Java. What about GGZip*Stream - eeek!?
libgsf has all these, along with a gio wrapper.
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/7/31 Jody Goldberg j...@gnome.org:
On Fri, Jul 31, 2009 at 02:48:47PM +0200, Mikkel Kamstrup Erlandsen wrote:
Hi,
I've been eye balling the GIO docs for a while without finding
in-/output for gzip compression... So if I missed it stop me now :-)
From the looks of it, it should be straight forward to write
GZip{In,Out}putStream classes based on zlib[1]. If I write these
classes would a zlib dep. be OK for GIO? It is pretty ubiquitous as
far as I can tell (and it is already a depency of Gtk+)... And I am
not really all that keen on writing my own gzip/DEFLATE implementation
(to say the least).
Another thing is the naming. Since this is not about supporting the
Zip archive format (but that would be nice too) the name GZip*Stream
might be bad. On the other hand i abhor the ALLCAPS GZIPInputStream
used in fx. Java. What about GGZip*Stream - eeek!?
libgsf has all these, along with a gio wrapper.
I assume you mean:
http://projects.gnome.org/libgsf/gsf-Compression.html? I could not
find the GIO wrapper in the documentation (assuming that it is
unreleased yet?). Anyways - my question is really if this
functionality should not be in GLib/GIO?
Adding a native GIO impl. would of course be looking heavily towards
what already exists in Gsf.
--
Cheers,
Mikkel
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/7/31 Mikkel Kamstrup Erlandsen mikkel.kamst...@gmail.com:
2009/7/31 Jody Goldberg j...@gnome.org:
On Fri, Jul 31, 2009 at 02:48:47PM +0200, Mikkel Kamstrup Erlandsen wrote:
Hi,
I've been eye balling the GIO docs for a while without finding
in-/output for gzip compression... So if I missed it stop me now :-)
From the looks of it, it should be straight forward to write
GZip{In,Out}putStream classes based on zlib[1]. If I write these
classes would a zlib dep. be OK for GIO? It is pretty ubiquitous as
far as I can tell (and it is already a depency of Gtk+)... And I am
not really all that keen on writing my own gzip/DEFLATE implementation
(to say the least).
Another thing is the naming. Since this is not about supporting the
Zip archive format (but that would be nice too) the name GZip*Stream
might be bad. On the other hand i abhor the ALLCAPS GZIPInputStream
used in fx. Java. What about GGZip*Stream - eeek!?
libgsf has all these, along with a gio wrapper.
I assume you mean:
http://projects.gnome.org/libgsf/gsf-Compression.html? I could not
find the GIO wrapper in the documentation (assuming that it is
unreleased yet?). Anyways - my question is really if this
functionality should not be in GLib/GIO?
I sort of like the proposal.
Cairo and libpng depends on zlib and there's no Win32 API to replace
it so the implementation would be fully cross platform it's very basic
functionality IMHO and having it in a GIO wrapper would be awesome.
I'd love to see BZip2/7zip/Tar/rar/... wrappers somewhere else in the
platform though (and kill most of the file-roller codebase for that
matter).
Only downside from my point of view is that in Windows this would mean
yet another .dll to distribute.
Adding a native GIO impl. would of course be looking heavily towards
what already exists in Gsf.
--
Cheers,
Mikkel
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
--
Un saludo,
Alberto Ruiz
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote:
From the looks of it, it should be straight forward to write
GZip{In,Out}putStream classes based on zlib
I'd say call it GCompressed{In,Out}putStream and have it either
auto-detect the compression type, or have a param in the API to specify
from an enum (or both). People can add support for other types of
compression as time goes on.
-brian
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/7/31 Brian J. Tarricone bj...@cornell.edu:
On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote:
From the looks of it, it should be straight forward to write
GZip{In,Out}putStream classes based on zlib
I'd say call it GCompressed{In,Out}putStream and have it either auto-detect
the compression type, or have a param in the API to specify from an enum (or
both). People can add support for other types of compression as time goes
on.
And expand the list of glib dependencies to the infinite?
I don't think I like such an idea at this level of the API.
-brian
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
--
Un saludo,
Alberto Ruiz
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
2009/7/31 Brian J. Tarricone bj...@cornell.edu:
On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote:
From the looks of it, it should be straight forward to write
GZip{In,Out}putStream classes based on zlib
I'd say call it GCompressed{In,Out}putStream and have it either auto-detect
the compression type, or have a param in the API to specify from an enum (or
both). People can add support for other types of compression as time goes
on.
The prime benefit of having dedicated classes over your approach is
having syntactically ensured that you are indeed working with GZipped
buffers. Personally I like that, but this is of course 100%
subjective.
--
Cheers,
Mikkel
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On 07/31/2009 01:59 PM, Mikkel Kamstrup Erlandsen wrote:
2009/7/31 Brian J. Tarriconebj...@cornell.edu:
On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote:
From the looks of it, it should be straight forward to write
GZip{In,Out}putStream classes based on zlib
I'd say call it GCompressed{In,Out}putStream and have it either auto-detect
the compression type, or have a param in the API to specify from an enum (or
both). People can add support for other types of compression as time goes
on.
The prime benefit of having dedicated classes over your approach is
having syntactically ensured that you are indeed working with GZipped
buffers. Personally I like that, but this is of course 100%
subjective.
Well, sure, but otherwise you can end up with classes for gzip, bzip2,
zip, 7zip, rar, etc. That alone is 10 extra classes, and I'm sure there
are other compression schemes people might want classes for. That
sounds messy to me.
Having a constructor for a generic class that takes a parameter for the
compression type would give you what you want, assuming it's set up so
it returns an error if the content you pass it doesn't appear to be of
the selected type.
I guess it doesn't really matter either way. You could even implement
GCompressedInputStream as a sort of class cluster that returns a
(private) subclass depending on the compression type, or have all
(public) classes be a subclass of a GCompressed{In,Out}putStream class,
and for all operations (except for construction) you'd call methods on
the superclass.
And I think there's a benefit to support format auto-detection if the
developer doesn't care what format the input is in. That's possibly a
little more difficult to do with explicit subclasses, though the
class-cluster method would still work and yet maintain separate classes
for each compression format.
-brian
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On Fri, 2009-07-31 at 21:17 +0100, Alberto Ruiz wrote:
2009/7/31 Brian J. Tarricone bj...@cornell.edu:
On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote:
From the looks of it, it should be straight forward to write
GZip{In,Out}putStream classes based on zlib
I'd say call it GCompressed{In,Out}putStream and have it either auto-detect
the compression type, or have a param in the API to specify from an enum (or
both). People can add support for other types of compression as time goes
on.
And expand the list of glib dependencies to the infinite?
I don't think I like such an idea at this level of the API.
You could claim the same about the gdk-pixbuf loaders, or the gvfs
backends. The point is that you'd want to make your original class
flexible enough that people could extend upon it (even if you don't
start off with a plugin API).
I could think of at least 5 types of compressions that would be useful
to have without having to use a command-line tool to decompress:
- gzip for anything and everything that can come from a web server (in
my case, iTunes Music Store playlist parsing, or more widely, GOffice
file formats)
- zip for OpenOffice.org documents and Comic Books (evince)
- 7zip/LZMA/Xz formats for Comic Books
- Rar for the same as above
I'd be happy seeing at least the first one, very happy to see the first
3, that all have open source implementations, and would provide more
robust support for those file types in applications.
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Re: GZip{In,Out}putStream in GIO?
On 07/31/2009 01:17 PM, Alberto Ruiz wrote:
2009/7/31 Brian J. Tarriconebj...@cornell.edu:
On 07/31/2009 05:48 AM, Mikkel Kamstrup Erlandsen wrote:
From the looks of it, it should be straight forward to write
GZip{In,Out}putStream classes based on zlib
I'd say call it GCompressed{In,Out}putStream and have it either auto-detect
the compression type, or have a param in the API to specify from an enum (or
both). People can add support for other types of compression as time goes
on.
And expand the list of glib dependencies to the infinite?
I don't think I like such an idea at this level of the API.
Eh, they can be optional, or dlopen()ed at runtime. The API could be
flexible enough (using the existing 'extension points' architecture) to
allow people to add their own implementations for their own use, or some
interested party could bundle a set of them in another package... though
in that case the entire compressed stream class could be in a separate
library. Instead of using an enum to select the compression type, a
string identifier could be used instead (perhaps the mime type).
IMO optional deps aren't really a big deal. If people thought it'd be
useful to have one or two mandatory deps (say zlib and bzip2, for
example), that might be ok too, since they're pretty ubiquitous.
-brian
___
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
