Bug#703645: packaging Postgres binary dump files
On 13340 March 1977, Daniel Pocock wrote: >>> However, if the package is formally rejected by the FTP masters then I >>> will be happy to change it to ASCII SQL if required. >> Please include the source / preferred form for modification in the >> source, and create this postgres thing from that. > I've now re-created the git repos on alioth and created a new version of > the orig.tar.gz that includes both the file downloaded from upstream and > an ASCII SQL > Only the SQL is shipped in the binary package. I think this is similar to other cases that came up in the past. The preferred form for modification isn't what is shipped and probably best to use / used by default from upstream and it's not easy to regenerate it at build time. The preferred form of modification is a running database and SQL commands issued to it with whatever interface. So what you should ship in the source is the file from upstream plus the SQL to generate it. Ideally you would now build-depend on a running postgres server, but everyone could understand the armada of black helicopters that buildd admins will send your way for this. I would even lend a chainsaw or two :) So this falls under the: "Not easy to regenerate" category. Which means that, ohwell, ship the upstream file - or one you generated yourself from the SQL, and don't rebuild it on buildds. BUT *you* *must* make entirely sure that what is shipped corresponds with the source AKA the SQL file. (And document it in README.source please) And double bonus points if there is an easy accessible way of getting from SQL to binary file, say debian/rules rebuild or so. With the implication that the user has to have a postgres ready for it to load and dump from. Or so. -- bye, Joerg Marge, it takes two to lie. One to lie and one to listen. -- To UNSUBSCRIBE, email to debian-wnpp-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87wqm95f9q@gkar.ganneff.de
Bug#703645: packaging Postgres binary dump files
On 09/20/2013 06:18 PM, Daniel Pocock wrote: > To go the other way (from an ASCII SQL into a binary dump file) during > the package build phase, it needs to be loaded into a running PostgreSQL > server and then extracted with pg_dump. I don't think that is a great > build dependency, especially if we want to support things like chroot > builds. I don't think you should distribute the files in the binary format at all as you'd have to require a pg_restore which is able to restore the files from pg_dump in the version you've used to package it - so while it might be possible that the ascii version just works well for older postgres versions, you might end up with needed pg_restore form 9.x just becuase you dumped it with it. -- Bernd ZeimetzDebian GNU/Linux Developer http://bzed.dehttp://www.debian.org GPG Fingerprint: ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F -- To UNSUBSCRIBE, email to debian-wnpp-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/523de80b.4020...@bzed.de
Bug#703645: packaging Postgres binary dump files
On 20/09/13 17:07, Paul Tagliamonte wrote: > On Fri, Sep 20, 2013 at 05:04:50PM +0200, Daniel Pocock wrote: >> On 20/09/13 15:49, Paul Tagliamonte wrote: >>> On Fri, Sep 20, 2013 at 02:47:39PM +0100, Jonathan Dowland wrote: On Fri, Sep 20, 2013 at 12:20:38PM +0200, Paul Wise wrote: > It is also impossible to patch the binary format unlike SQL. Interesting. For the first time, I've realised there can be a clash between "preferred form for modification" and "preferred form for use". >>> I mean, not really, right? >>> >>> If I want to use a .so, I want the ELF, but I want to modify it in C >>> >>> >>> This just means we ship the prefered form for use (this binary kruft) >>> but ship the preferred form for modification in the source. >> The rules file could apply changes if required, pg_restore | something | >> pg_dump again. >> >> The current version of the postbooks-schema-* packages are now in >> collab-maint git. They simply install these files to >> /usr/share/postbooks-schema but make no attempt to load them into PostgreSQL >> >> In this case, it is a client-server solution. There is no guarantee the >> client code (package: postbooks) runs on the same host as the database >> (packages postgresql and postbooks-schema-quickstart). Maybe the user >> even has some Windows clients too. So we have no easy way to >> synchronize changes to the client package and the database. >> >> If somebody wants to create any indexes in the database, details can go >> in README.Debian. The administrator can then choose how to use it. >> >> However, if the package is formally rejected by the FTP masters then I >> will be happy to change it to ASCII SQL if required. > Please include the source / preferred form for modification in the > source, and create this postgres thing from that. I've now re-created the git repos on alioth and created a new version of the orig.tar.gz that includes both the file downloaded from upstream and an ASCII SQL Only the SQL is shipped in the binary package. The file from upstream is not distributed in the binary package for now and we do not try to regenerate it either. The conversion between this format and ASCII is one way with the pg_restore command. We convert the file from binary to SQL during the creation of the orig.tar.gz (it is a repackaged upstream tarball in effect). To go the other way (from an ASCII SQL into a binary dump file) during the package build phase, it needs to be loaded into a running PostgreSQL server and then extracted with pg_dump. I don't think that is a great build dependency, especially if we want to support things like chroot builds. There are two downsides to this approach: - user doesn't get the benefit of the pg_restore features for selective restore - the usage instructions are very slightly different from what upstream describes, in particular, the SQL is zipped and has to be piped through zcat when using it -- To UNSUBSCRIBE, email to debian-wnpp-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/523c7558.6000...@pocock.com.au
Bug#703645: packaging Postgres binary dump files
On Fri, Sep 20, 2013 at 10:59 AM, Chow Loong Jin wrote: > Just speaking for myself here, but I find that the binary format is more > flexible in that pg_restore can selectively restore things, generate DROP > statements, restoring things in parallel and such. All in all, the binary > format > seems much more useful than the SQL format. In this case we are talking about installation and upgrades, presumably selective restoration etc aren't needed here. > You can also compress the binary format (pg_dump -Z0..9), but it still isn't > as > efficient as SQL compressed with xz -9. It is more efficient than SQL but less efficient than even lzop -1, I guess there is not much compression. It is also impossible to patch the binary format unlike SQL. Anyway, this is getting off-topic. As long as the ftpteam are made aware of the format's properties everything should be fine. -- bye, pabs http://wiki.debian.org/PaulWise -- To UNSUBSCRIBE, email to debian-wnpp-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/caktje6hg9g0b3lzhc9gco2e5zawbc5uheapbaubss7lgnrt...@mail.gmail.com
Bug#703645: packaging Postgres binary dump files
Re: Paul Wise 2013-09-20 > The format doesn't appear to be very efficient, the plain SQL commands > are much smaller: > > pabs@wagner:~$ pg_restore -l postbooks_empty-4.1.0.backup > foo.sql > pabs@wagner:~$ ls -Ssh > total 5.6M > 5.3M postbooks_empty-4.1.0.backup 344K foo.sql pg_restore -l will just give you the listing of what's inside the dump file, not the actual contents. > It doesn't seem possible to treat it as a compressed tarball: > > pabs@chianamo ~ $ tar zxf postbooks_empty-4.1.0.backup That'd be "pg_restore postbooks_empty-4.1.0.backup". (Note that there is also a "tar" format that "pg_dump -Ft" will create, but that format is crap because it needs to create a tempfile somewhere first. These "binary" or "custom" dumps that "pg_dump -Fc" creates are much more flexible to use because you can stream them when writing, while at the same time getting a TOC for random read access later.) Christoph -- c...@df7cb.de | http://www.df7cb.de/ signature.asc Description: Digital signature
Bug#703645: packaging Postgres binary dump files
On 20/09/2013 10:59, Chow Loong Jin wrote: > On Fri, Sep 20, 2013 at 09:07:48AM +0200, Paul Wise wrote: >> > On Thu, Sep 19, 2013 at 2:42 PM, Daniel Pocock wrote: >> > >>> > > PostBooks distributes their schema as a Postgres binary dump file for >>> > > use with pg_restore >> > >> > What is their reason for using the binary format? Could they be >> > convinced to switch to or add something more normal like compressed >> > SQL? > Just speaking for myself here, but I find that the binary format is more > flexible in that pg_restore can selectively restore things, generate DROP > statements, restoring things in parallel and such. All in all, the binary > format > seems much more useful than the SQL format. > > You can also compress the binary format (pg_dump -Z0..9), but it still isn't > as > efficient as SQL compressed with xz -9. The binary format is the preferred one for dumps because allows to selectively restore only parts of a database. Doing the same kind of manipulation using an SQL script requires a lot of proficiency in sed/awk/perl and regular expressions. Yes the format is binary but given that the tools to manipulate it are completely free and already available in Debian why distribute a less useful version of the same data? federico -- Federico Di Gregorio federico.digrego...@dndg.it Don't dream it. Be it. -- Dr. Frank'n'further signature.asc Description: OpenPGP digital signature
Bug#703645: packaging Postgres binary dump files
On Fri, Sep 20, 2013 at 09:07:48AM +0200, Paul Wise wrote: > On Thu, Sep 19, 2013 at 2:42 PM, Daniel Pocock wrote: > > > PostBooks distributes their schema as a Postgres binary dump file for > > use with pg_restore > > What is their reason for using the binary format? Could they be > convinced to switch to or add something more normal like compressed > SQL? Just speaking for myself here, but I find that the binary format is more flexible in that pg_restore can selectively restore things, generate DROP statements, restoring things in parallel and such. All in all, the binary format seems much more useful than the SQL format. You can also compress the binary format (pg_dump -Z0..9), but it still isn't as efficient as SQL compressed with xz -9. -- Kind regards, Loong Jin signature.asc Description: Digital signature
Bug#703645: packaging Postgres binary dump files
On 20/09/13 09:07, Paul Wise wrote: > On Thu, Sep 19, 2013 at 2:42 PM, Daniel Pocock wrote: > >> PostBooks distributes their schema as a Postgres binary dump file for >> use with pg_restore > What is their reason for using the binary format? Could they be > convinced to switch to or add something more normal like compressed > SQL? > Maybe they are just trying to make it easy for people to download and start using it quickly. My own database was created using their "quickstart" database, I've also tried their "demo" database, both are very easy to use for anybody with basic PostgreSQL knowledge. They also provide an installer as another way to get started. Neither Andrew nor I have tried that yet and it is not in the main upstream tarball, it is released from another repository. We may want to provide just the SQL and a postinst script wrapped in a Debian package rather than packaging their whole installer. -- To UNSUBSCRIBE, email to debian-wnpp-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/523c006c.7070...@pocock.com.au
Bug#703645: packaging Postgres binary dump files
On Thu, Sep 19, 2013 at 2:49 PM, Martijn van Oosterhout wrote: > FWIW, you can convert the file to text using pg_restore, you don't actually > need a running database server. It's really just a compressed tarball and > should be treated as such. That is, I think it can be included as-is. Unless > you're thinking of patching it, in which case you need to think of something > else. The format doesn't appear to be very efficient, the plain SQL commands are much smaller: pabs@wagner:~$ pg_restore -l postbooks_empty-4.1.0.backup > foo.sql pabs@wagner:~$ ls -Ssh total 5.6M 5.3M postbooks_empty-4.1.0.backup 344K foo.sql It doesn't seem possible to treat it as a compressed tarball: pabs@chianamo ~ $ tar zxf postbooks_empty-4.1.0.backup gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error is not recoverable: exiting now pabs@chianamo ~ $ tar jxf postbooks_empty-4.1.0.backup bzip2: (stdin) is not a bzip2 file. tar: Child returned status 2 tar: Error is not recoverable: exiting now pabs@chianamo ~ $ tar Jxf postbooks_empty-4.1.0.backup xz: (stdin): File format not recognized tar: Child returned status 1 tar: Error is not recoverable: exiting now -- bye, pabs http://wiki.debian.org/PaulWise -- To UNSUBSCRIBE, email to debian-wnpp-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CAKTje6H+3YuHOo3Tr39yuCubjdZD08XOcVUu=02tvttx9x1...@mail.gmail.com
Bug#703645: packaging Postgres binary dump files
On Thu, Sep 19, 2013 at 2:42 PM, Daniel Pocock wrote: > PostBooks distributes their schema as a Postgres binary dump file for > use with pg_restore What is their reason for using the binary format? Could they be convinced to switch to or add something more normal like compressed SQL? -- bye, pabs http://wiki.debian.org/PaulWise -- To UNSUBSCRIBE, email to debian-wnpp-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CAKTje6EhcjNwP54pMp8AGtweBVOkqsnzz=jdPJQZ6_5=b_x...@mail.gmail.com
Bug#703645: packaging Postgres binary dump files
On 19 September 2013 14:42, Daniel Pocock wrote: > > > PostBooks distributes their schema as a Postgres binary dump file for > use with pg_restore > > They are available for download here (not in the source tarball): > > > http://sourceforge.net/projects/postbooks/files/03%20PostBooks-databases/4.1.0/ > > Is this format suitable for making a source package or do we need to > load it into Postgres and then pg_dump it again as text / SQL? > > I would prefer to simply distribute the original files so that people > can compare with upstream's checksums if they wish. > FWIW, you can convert the file to text using pg_restore, you don't actually need a running database server. It's really just a compressed tarball and should be treated as such. That is, I think it can be included as-is. Unless you're thinking of patching it, in which case you need to think of something else. Have a nice day, -- Martijn van Oosterhout http://svana.org/kleptog/
Bug#703645: packaging Postgres binary dump files
PostBooks distributes their schema as a Postgres binary dump file for use with pg_restore They are available for download here (not in the source tarball): http://sourceforge.net/projects/postbooks/files/03%20PostBooks-databases/4.1.0/ The pg_dump documentation explains the binary format features: http://www.postgresql.org/docs/9.1/static/app-pgdump.html Is this format suitable for making a source package or do we need to load it into Postgres and then pg_dump it again as text / SQL? I would prefer to simply distribute the original files so that people can compare with upstream's checksums if they wish. -- To UNSUBSCRIBE, email to debian-wnpp-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/523af145.4090...@pocock.com.au
