Re: Reimplement mksplit.pl in C
Hi Marcelo! On Fri, 2010-04-02 at 22:55:38 -0600, Marcelo E. Magallon wrote: somewhere I read that a reimplementation of mksplit.pl in C was needed in order to remove a dependency of dpkg on Perl, but I can't find the reference anymore. The main reasons for this are to not be subject to the perl upgrade problems as seen in 489132, to couple those perl programs more tightly with the rest of the C parts, to be able to reuse parts of the code, for example there's programs that could benefit from a shared library for update-alternativesy, and to be able to remove perl-base from essential some day which would make life for embedded distros easier. There's other small programs in essential which would be nice to get rewritten [0], another problematic package is adduser, although right now I think most of the missing features have been implemented directly in the shadow package. [0] http://lists.debian.org/debian-devel/2008/08/msg00517.html (Err, yeah should have finished them long time ago...) Anyways, attached is a patch to do just that. It's implemented against the master Git branch. As indicated in the patch, this is a straight translation and it would like to have some refactoring love. I did compare the output with the Perl version in several test cases and it's identical. Ouch :(, first of all thanks for the patches, although I already had this written in C integrated directly in dpkg-split in my pu/c-rewrite branch [1], although not yet merged, there were still some rough egdes I wanted to polish first. And second sorry for the duplicated work. I thought I had written this down on the wiki at least, but it seems not, I might have commented it on the list and on IRC. Will amend the wiki right now, and will take a look at the patches later to see what can be reused. [1] http://git.hadrons.org/?p=debian/dpkg/dpkg.git;a=heads thanks, guillem -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100408165754.ga17...@gaara.hadrons.org
Re: Reimplement mksplit.pl in C
[ I subscribed to the list, thanks for the Cc:'s ]
On Sat, Apr 03, 2010 at 12:20:52PM -0600, Marcelo E. Magallon wrote:
The two cases I can think of are splitting an archive that
splits to a single piece, splitting to multiple pieces and
some failure cases.
Right now I do something like:
cat test.deb |
build-tree/dpkg-split/mksplit test.deb 1024 test 4608 2048 no
cp test.deb test.deb.orig
dpkg-split -j test*.deb
cmp test.deb test.deb.orig echo OK
but not much more.
I did some refactoring, and reused a couple of bits of libdpkg.
I was hoping for libdpkg to have something like:
package = get_deb_field(my_package.deb, Package);
but I didn't find it. Did I miss it? Is there something close
to that?
Anyways, the updated patches are attached.
Marcelo
From 40cbd26e938f3ffda58803c6e25d6e4c9a836b96 Mon Sep 17 00:00:00 2001
From: Marcelo E. Magallon marcelo.magal...@gmail.com
Date: Fri, 2 Apr 2010 22:27:50 -0600
Subject: [PATCH 1/5] Reimplement mksplit in C
My original intention was to do without some of the pipes, but for the
initial implementation this is just a translation of the original Perl
program into C, without much in the way of optimization. It produces
the same output and exits in the same way as the original program (mod
exit status probably).
The reimplementation can use some refactoring love.
---
dpkg-split/Makefile.am | 12 ++-
dpkg-split/mksplit.c | 279
2 files changed, 288 insertions(+), 3 deletions(-)
create mode 100644 dpkg-split/mksplit.c
diff --git a/dpkg-split/Makefile.am b/dpkg-split/Makefile.am
index 6f6043c..3ea9d00 100644
--- a/dpkg-split/Makefile.am
+++ b/dpkg-split/Makefile.am
@@ -24,10 +24,16 @@ dpkg_split_LDADD = \
../lib/compat/libcompat.a \
$(LIBINTL)
+mksplit_SOURCES = \
+ mksplit.c
-pkglib_SCRIPTS = mksplit
-EXTRA_DIST = mksplit.pl
-CLEANFILES = $(pkglib_SCRIPTS)
+mksplit_LDADD = \
+ ../lib/dpkg/libdpkg.a \
+ ../lib/compat/libcompat.a \
+ $(LIBINTL)
+
+pkglib_PROGRAMS = mksplit
+CLEANFILES = $(pkglib_PROGRAMS)
do_perl_subst = $(AM_V_GEN) sed -e s:^\#![:space:]*/usr/bin/perl:\#!$(PERL):
diff --git a/dpkg-split/mksplit.c b/dpkg-split/mksplit.c
new file mode 100644
index 000..5ca34d6
--- /dev/null
+++ b/dpkg-split/mksplit.c
@@ -0,0 +1,279 @@
+/* This program is only supposed to be called by dpkg-split.
+ * Its arguments are:
+ * sourcefile partsize prefix totalsize partsizeallow msdostruncyesno
+ * Stdin is also redirected from the source archive by dpkg-split.
+ *
+ * Copyright © 1995 Ian Jackson i...@chiark.greenend.org.uk
+ * Copyright © 2010 Marcelo E. Magallon mmaga...@debian.org
+ *
+ * This is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/.
+ */
+
+#include stdio.h
+#include stdlib.h
+#include time.h
+#include limits.h
+#include error.h
+#include string.h
+#include ctype.h
+#include errno.h
+#include unistd.h
+
+#define MSDOS_PATH_MAX 10
+
+void checked_free(void *ptr)
+{
+if (ptr) free(ptr);
+}
+
+void add(FILE *stream, char* id, char* data, size_t len)
+{
+time_t current_time;
+
+current_time = time(NULL);
+fprintf(stream, %-16s%-12d0 0 100644 %-10d%c\n,
+id, (int)current_time, (int)len, 0140);
+fwrite(data, 1, len, stream);
+fprintf(stream, %s, (len 1) ? \n : );
+}
+
+char* output(char *cmd)
+{
+FILE * file = NULL;
+int len = getpagesize();
+char *result = (char *)malloc(len);
+char *p = result;
+
+if (result == NULL)
+goto error_out;
+
+if ((file = popen(cmd, r)) == NULL)
+goto error_out;
+
+while (len 0 !feof(file))
+{
+int n;
+n = fread(p, 1, len, file);
+if (ferror(file))
+goto error_out;
+p += n;
+len -= n;
+}
+goto out;
+
+error_out:
+if (result)
+{
+free(result);
+result = NULL;
+}
+
+out:
+if (file)
+pclose(file);
+return result;
+}
+
+char* get_field(char *filename, char *field)
+{
+char *result;
+char cmd[PATH_MAX];
+sprintf(cmd, dpkg-deb --field %s %s, filename, field);
+
+result = output(cmd);
+if (result)
+{
+char *p;
+for (p = result + strlen(result) - 1; p = result isspace(*p); --p)
+*p = 0;
+}
+
+return result;
+}
+
+char* get_md5sum(char *filename)
+{
+char *result;
+char cmd[PATH_MAX];
+
Re: Reimplement mksplit.pl in C
On Sat, Apr 03, 2010 at 09:23:54PM +0200, Raphael Hertzog wrote: The more recent reference is in http://wiki.debian.org/Teams/Dpkg/RoadMap Thanks! Marcelo -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100405034717.gb21...@esk
Re: Reimplement mksplit.pl in C
Hi, Marcelo E. Magallon wrote: somewhere I read that a reimplementation of mksplit.pl in C was needed in order to remove a dependency of dpkg on Perl, but I can't find the reference anymore. I could only find approx one reference for this goal [1]. Luckily, even if that doesn’t pan out, there are plenty of other benefits to well written C code. Most importantly, it makes code sharing a little easier. I guess this would also make repairing a broken perl installation a little simpler on some system with only access to small media, though I haven’t encountered a need for dpkg-split in a long while. Thanks for working on it. Oddly enough I don't see a dependency of dpkg on perl either $ apt-cache show perl-base | grep Essential Essential: yes I did compare the output with the Perl version in several test cases and it's identical. Some test cases (or methods for producing them) would make me very happy. dpkg doesn’t have enough automated tests of basic functionality. Automake provides a rudimentary test harness [2]. Subject: [PATCH] Reimplement mksplit in C My original intention was to do without some of the pipes, but for the initial implementation this is just a translation of the original Perl program into C, without much in the way of optimization. It produces the same output and exits in the same way as the original program (mod exit status probably). I will let the dpkg-split experts comment on the important part. --- a/dpkg-split/Makefile.am +++ b/dpkg-split/Makefile.am @@ -9,7 +9,7 @@ AM_CPPFLAGS = \ -I$(top_srcdir)/lib -bin_PROGRAMS = dpkg-split +bin_PROGRAMS = dpkg-split mksplit I think this should be bin_PROGRAMS = dpkg-split pkglib_PROGRAMS = mksplit so that mksplit stays in /usr/lib/dpkg. Hmm? Jonathan -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100403061933.ga23...@progeny.tock
Re: Reimplement mksplit.pl in C
Jonathan Nieder jrnie...@gmail.com writes: Some test cases (or methods for producing them) would make me very happy. dpkg doesn’t have enough automated tests of basic functionality. Automake provides a rudimentary test harness [2]. Given that a fair bit of dpkg is written in Perl, which means that people are already familiar with the Perl test harness, you may want to consider using my C TAP Harness framework for writing tests instead. http://www.eyrie.org/~eagle/software/c-tap-harness/ It implements a pure C version of Perl's Test::Harness and prove (which is useful because prove isn't entirely happy with tests not written in Perl), but probably more relevantly, it provides TAP libraries for writing tests in C and shell. The Automake test harness is okay, but I really like the Perl test framework output and its test protocol. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/871vexm4lp@windlord.stanford.edu
Re: Reimplement mksplit.pl in C
Russ Allbery wrote: Jonathan Nieder jrnie...@gmail.com writes: Automake provides a rudimentary test harness [2]. Given that a fair bit of dpkg is written in Perl, which means that people are already familiar with the Perl test harness, you may want to consider using my C TAP Harness framework for writing tests instead. http://www.eyrie.org/~eagle/software/c-tap-harness/ Looks neat. Thanks for pointing me to it. Cheers, Jonathan My last message was missing some footnotes. Oops. [1] http://lists.debian.org/debian-devel/2001/03/msg00250.html http://lists.debian.org/debian-embedded/2009/04/msg00064.html [2] http://www.gnu.org/software/hello/manual/automake/Simple-Tests.html -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100403065849.ga23...@progeny.tock
Re: Reimplement mksplit.pl in C
On Sat, Apr 03, 2010 at 01:19:33AM -0500, Jonathan Nieder wrote: I could only find approx one reference for this goal [1]. Ah, yes, that was it, thanks! Removing Perl from the essential set. While going to bed, I actually figured that was the reason why I wasn't finding perl in the dependencies, but I didn't remember this was the reason I had read about for the reimplementation. Luckily, even if that doesn’t pan out, there are plenty of other benefits to well written C code. Most importantly, it makes code sharing a little easier. Sure. I'd like to take this implementation and factor out whatever is reusable. There are a couple of maybe-useful gimmicks in there, for example the Perl idiom `program` transliterated into C. I cheated and took a guess regarding how large should a buffer be to be large enough for this use case: my guess is one page. There are other hard-coded buffer length limits, too. Memory concious it ain't right now, this code. I guess this would also make repairing a broken perl installation a little simpler on some system with only access to small media, though I haven’t encountered a need for dpkg-split in a long while. Thanks for working on it. Same here. The last time I used dpkg-split for something was back in ... 1996? I did compare the output with the Perl version in several test cases and it's identical. Some test cases (or methods for producing them) would make me very happy. dpkg doesn’t have enough automated tests of basic functionality. Automake provides a rudimentary test harness [2]. I was thinking about that. The problem with mksplit is that the generated files contain an embedded timestamp, so when I say identical I actually mean identical modulo the embedded timestamp. I mean, you can't just cmp two sets of files. Beyond that, mksplit has several bits of code identifiable as functions that ought to be separated and those could have unit tests. I would have to look in more detail at dpkg-split to see if it's possible to devise a good automated test for this functionality. The two cases I can think of are splitting an archive that splits to a single piece, splitting to multiple pieces and some failure cases. There's a check in mksplit.pl (also implemented in mksplit.c) that I'm still trying to figure out how to trigger (the corresponding error message refers to the header being too large). I think this should be bin_PROGRAMS = dpkg-split pkglib_PROGRAMS = mksplit so that mksplit stays in /usr/lib/dpkg. Ah, right, thanks, updated patch attached. Marcelo From acbd7b59d0d48ec47191d67846029aedc3c49afe Mon Sep 17 00:00:00 2001 From: Marcelo E. Magallon marcelo.magal...@gmail.com Date: Fri, 2 Apr 2010 22:27:50 -0600 Subject: [PATCH] Reimplement mksplit in C My original intention was to do without some of the pipes, but for the initial implementation this is just a translation of the original Perl program into C, without much in the way of optimization. It produces the same output and exits in the same way as the original program (mod exit status probably). The reimplementation can use some refactoring love. --- dpkg-split/Makefile.am | 12 ++- dpkg-split/mksplit.c | 279 2 files changed, 288 insertions(+), 3 deletions(-) create mode 100644 dpkg-split/mksplit.c diff --git a/dpkg-split/Makefile.am b/dpkg-split/Makefile.am index 6f6043c..3ea9d00 100644 --- a/dpkg-split/Makefile.am +++ b/dpkg-split/Makefile.am @@ -24,10 +24,16 @@ dpkg_split_LDADD = \ ../lib/compat/libcompat.a \ $(LIBINTL) +mksplit_SOURCES = \ + mksplit.c -pkglib_SCRIPTS = mksplit -EXTRA_DIST = mksplit.pl -CLEANFILES = $(pkglib_SCRIPTS) +mksplit_LDADD = \ + ../lib/dpkg/libdpkg.a \ + ../lib/compat/libcompat.a \ + $(LIBINTL) + +pkglib_PROGRAMS = mksplit +CLEANFILES = $(pkglib_PROGRAMS) do_perl_subst = $(AM_V_GEN) sed -e s:^\#![:space:]*/usr/bin/perl:\#!$(PERL): diff --git a/dpkg-split/mksplit.c b/dpkg-split/mksplit.c new file mode 100644 index 000..5ca34d6 --- /dev/null +++ b/dpkg-split/mksplit.c @@ -0,0 +1,279 @@ +/* This program is only supposed to be called by dpkg-split. + * Its arguments are: + * sourcefile partsize prefix totalsize partsizeallow msdostruncyesno + * Stdin is also redirected from the source archive by dpkg-split. + * + * Copyright © 1995 Ian Jackson i...@chiark.greenend.org.uk + * Copyright © 2010 Marcelo E. Magallon mmaga...@debian.org + * + * This is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You
Re: Reimplement mksplit.pl in C
On Sat, 03 Apr 2010, Jonathan Nieder wrote: Hi, Marcelo E. Magallon wrote: somewhere I read that a reimplementation of mksplit.pl in C was needed in order to remove a dependency of dpkg on Perl, but I can't find the reference anymore. I could only find approx one reference for this goal [1]. The more recent reference is in http://wiki.debian.org/Teams/Dpkg/RoadMap Cheers, -- Raphaël Hertzog Like what I do? Sponsor me: http://ouaza.com/wp/2010/01/05/5-years-of-freexian/ My Debian goals: http://ouaza.com/wp/2010/01/09/debian-related-goals-for-2010/ -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100403192354.ga17...@rivendell
Reimplement mksplit.pl in C
Hi,
somewhere I read that a reimplementation of mksplit.pl in C was
needed in order to remove a dependency of dpkg on Perl, but I
can't find the reference anymore. Oddly enough I don't see a
dependency of dpkg on perl either:
Package: dpkg
Version: 1.15.6.1
Replaces: manpages-de (= 0.4-3), manpages-pl (= 20051117-1)
Pre-Depends: libbz2-1.0, libc6 (= 2.3), libselinux1 (= 1.32), zlib1g (=
1:1.1.4), coreutils (= 5.93-1), xz-utils
Suggests: apt
Conflicts: apt ( 0.7.7), aptitude ( 0.4.7-1), dpkg-dev ( 1.14.16),
dpkg-iasearch ( 0.11), sysvinit ( 2.82-1)
Breaks: emacs21 ( 21.4a+1-5.7), emacs21-nox ( 21.4a+1-5.7), emacs22 (=
22.3+1-1), emacs22-gtk (= 22.3+1-1), emacs22-nox (= 22.3+1-1), jed (
1:0.99.18+dfsg.1-13), jed-extra (= 2.5.3-2), konqueror (= 4:4.2.96-1), pinfo
( 0.6.9-3.1), tkinfo ( 2.8-3.1), xemacs21-support ( 21.4.22-2), xjed (
1:0.99.18+dfsg.1-13)
Anyways, attached is a patch to do just that. It's implemented
against the master Git branch. As indicated in the patch, this
is a straight translation and it would like to have some
refactoring love. I did compare the output with the Perl
version in several test cases and it's identical.
Please Cc: me, I'm not subscribed.
Cheers,
Marcelo
From 267685fb5750a47b536ab817a72597836abe5c5f Mon Sep 17 00:00:00 2001
From: Marcelo E. Magallon mmaga...@debian.org
Date: Fri, 2 Apr 2010 22:27:50 -0600
Subject: [PATCH] Reimplement mksplit in C
My original intention was to do without some of the pipes, but for the
initial implementation this is just a translation of the original Perl
program into C, without much in the way of optimization. It produces
the same output and exits in the same way as the original program (mod
exit status probably).
The reimplementation can use some refactoring love.
---
dpkg-split/Makefile.am | 10 ++-
dpkg-split/mksplit.c | 279
2 files changed, 287 insertions(+), 2 deletions(-)
create mode 100644 dpkg-split/mksplit.c
diff --git a/dpkg-split/Makefile.am b/dpkg-split/Makefile.am
index 6f6043c..c43025d 100644
--- a/dpkg-split/Makefile.am
+++ b/dpkg-split/Makefile.am
@@ -9,7 +9,7 @@ AM_CPPFLAGS = \
-I$(top_srcdir)/lib
-bin_PROGRAMS = dpkg-split
+bin_PROGRAMS = dpkg-split mksplit
dpkg_split_SOURCES = \
dpkg-split.h \
@@ -24,9 +24,15 @@ dpkg_split_LDADD = \
../lib/compat/libcompat.a \
$(LIBINTL)
+mksplit_SOURCES = \
+ mksplit.c
+
+mksplit_LDADD = \
+ ../lib/dpkg/libdpkg.a \
+ ../lib/compat/libcompat.a \
+ $(LIBINTL)
pkglib_SCRIPTS = mksplit
-EXTRA_DIST = mksplit.pl
CLEANFILES = $(pkglib_SCRIPTS)
diff --git a/dpkg-split/mksplit.c b/dpkg-split/mksplit.c
new file mode 100644
index 000..5ca34d6
--- /dev/null
+++ b/dpkg-split/mksplit.c
@@ -0,0 +1,279 @@
+/* This program is only supposed to be called by dpkg-split.
+ * Its arguments are:
+ * sourcefile partsize prefix totalsize partsizeallow msdostruncyesno
+ * Stdin is also redirected from the source archive by dpkg-split.
+ *
+ * Copyright © 1995 Ian Jackson i...@chiark.greenend.org.uk
+ * Copyright © 2010 Marcelo E. Magallon mmaga...@debian.org
+ *
+ * This is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/.
+ */
+
+#include stdio.h
+#include stdlib.h
+#include time.h
+#include limits.h
+#include error.h
+#include string.h
+#include ctype.h
+#include errno.h
+#include unistd.h
+
+#define MSDOS_PATH_MAX 10
+
+void checked_free(void *ptr)
+{
+if (ptr) free(ptr);
+}
+
+void add(FILE *stream, char* id, char* data, size_t len)
+{
+time_t current_time;
+
+current_time = time(NULL);
+fprintf(stream, %-16s%-12d0 0 100644 %-10d%c\n,
+id, (int)current_time, (int)len, 0140);
+fwrite(data, 1, len, stream);
+fprintf(stream, %s, (len 1) ? \n : );
+}
+
+char* output(char *cmd)
+{
+FILE * file = NULL;
+int len = getpagesize();
+char *result = (char *)malloc(len);
+char *p = result;
+
+if (result == NULL)
+goto error_out;
+
+if ((file = popen(cmd, r)) == NULL)
+goto error_out;
+
+while (len 0 !feof(file))
+{
+int n;
+n = fread(p, 1, len, file);
+if (ferror(file))
+goto error_out;
+p += n;
+len -= n;
+}
+goto out;
+
+error_out:
+if (result)
+{
+free(result);
+result = NULL;
+}
+
+out:
+if (file)
+pclose(file);
+return result;
+}
+
+char*
