Re: Adding sha256 support into jigdo tools (jigdo and jigit)
Hi Thomas! I've just released jigit v1.22 with the embedded libjte 2.0.0. Here's a (hopefully complete!) diff of the extra changes I've made on top of xorriso 1.5.2, beyond simply copying in the libjte directory from my new release. I've tested this for generating both jigdo formats and all looks good here. I'm juct committing the debian-cd changes to (optionally) use this as well, for when we're ready to throw the switch. Obviously, shout if you have any issues, and of course I'll be happy to sponsor the 1.5.3(?) upload when you're read. -- Steve McIntyre, Cambridge, UK.st...@einval.com "Further comment on how I feel about IBM will appear once I've worked out whether they're being malicious or incompetent. Capital letters are forecast." Matthew Garrett, http://www.livejournal.com/users/mjg59/30675.html diff -urbB --exclude libjte --exclude '*.texi' --exclude '*.info' xorriso-1.5.2.old/libisofs/libisofs.h xorriso-1.5.2/libisofs/libisofs.h --- xorriso-1.5.2.old/libisofs/libisofs.h 2019-10-26 19:22:45.0 +0100 +++ xorriso-1.5.2/libisofs/libisofs.h 2019-11-21 16:09:34.704652769 + @@ -2393,7 +2393,7 @@ tests. It can be prevented by ./configure option --disable-libjte . @since 0.6.38 */ -#define iso_libjte_req_major 1 +#define iso_libjte_req_major 2 #define iso_libjte_req_minor 0 #define iso_libjte_req_micro 0 diff -urbB --exclude libjte --exclude '*.texi' --exclude '*.info' xorriso-1.5.2.old/xorriso/emulators.c xorriso-1.5.2/xorriso/emulators.c --- xorriso-1.5.2.old/xorriso/emulators.c 2019-10-26 19:22:45.0 +0100 +++ xorriso-1.5.2/xorriso/emulators.c 2019-11-21 13:54:41.136396703 + @@ -657,8 +657,10 @@ "-alpha-boot", "-hppa-cmdline", "-hppa-kernel-32", "-hppa-kernel-64", "-hppa-bootloader", "-hppa-ramdisk", "-mips-boot", "-mipsel-boot", "-jigdo-jigdo", "-jigdo-template", "-jigdo-min-file-size", -"-jigdo-force-md5", "-jigdo-exclude", "-jigdo-map", "-md5-list", -"-jigdo-template-compress", +"-jigdo-force-md5", "jigdo-force-checksum", +"-md5-list", "-checksum-list", +"-jigdo-exclude", "-jigdo-map", +"-jigdo-template-compress", "-jigdo-checksum-algorithm", "-checksum_algorithm_iso", "-checksum_algorithm_template", "--stdio_sync", "--quoted_path_list", "--efi-boot", "--embedded-boot", "-isohybrid-mbr", "-e", "-partition_offset", "-partition_hd_cyl", @@ -853,10 +855,14 @@ " -jigdo-template FILEProduce a jigdo .template file as well as the .iso", " -jigdo-min-file-size SIZE Minimum size for a file to be listed in the jigdo file", " -jigdo-force-md5 PATTERNPattern(s) where files MUST match an externally-supplied MD5sum", +" -jigdo-force-checksum PTRN Pattern(s) where files MUST match an externally-supplied checksum", " -jigdo-exclude PATTERN Pattern(s) to exclude from the jigdo file", " -jigdo-map PATTERN1=PATTERN2", " Pattern(s) to map paths (e.g. Debian=/mirror/debian)", " -md5-list FILE File containing MD5 sums of the files that should be checked", +" -checksum-list FILE File containing checksums of the files that should be checked", +" -jigdo-checksum-algorithm ALGORITHM", +" Choose to use md5 or sha256 checksums in jigdo date; default is md5", " -jigdo-template-compress ALGORITHM", " Choose to use gzip or bzip2 compression for template data; default is gzip", " -checksum_algorithm_iso alg1,alg2,...", @@ -2097,11 +2103,14 @@ strcmp(argpt, "-jigdo-min-file-size") == 0 || strcmp(argpt, "-jigdo-exclude") == 0 || strcmp(argpt, "-jigdo-force-md5") == 0 || + strcmp(argpt, "-jigdo-force-checksum") == 0 || strcmp(argpt, "-jigdo-map") == 0 || + strcmp(argpt, "-jigdo-checksum-algorithm") == 0 || strcmp(argpt, "-jigdo-template-compress") == 0 || strcmp(argpt, "-checksum_algorithm_iso") == 0 || strcmp(argpt, "-checksum_algorithm_template") == 0 || - strcmp(argpt, "-md5-list") == 0) { + strcmp(argpt, "-md5-list") == 0 || + strcmp(argpt, "-checksum-list") == 0) { i++; ret= Xorriso_option_jigdo(xorriso, argpt, argv[i], 0); if(ret <= 0) diff -urbB --exclude libjte --exclude '*.texi' --exclude '*.info' xorriso-1.5.2.old/xorriso/lib_mgt.c xorriso-1.5.2/xorriso/lib_mgt.c --- xorriso-1.5.2.old/xorriso/lib_mgt.c 2019-10-26 19:22:45.0 +0100 +++ xorriso-1.5.2/xorriso/lib_mgt.c 2019-11-21 15:08:54.274180525 + @@ -765,8 +765,11 @@ goto jte_failed; xorriso->libjte_params_given|= 8; } else if(strcmp(aspect, "md5_path") == 0 || - strcmp(aspect, "-md5-list") == 0) { - ret= libjte_set_md5_path(jte, arg); + strcmp(aspect, "-md5-list") == 0 || + strcmp(aspect,
Re: Adding sha256 support into jigdo tools (jigdo and jigit)
Hi, Steve McIntyre wrote: > Let's go with "-jigdo-checksum-algorithm" to keep the separation? The user is king. Other mail: I proposed to keep versioning and SONAME at 1 > > LIBJTE1 { Steve McIntyre wrote: > Surely that works for upwards compatibility, but won't stop somebody > from building and depending on the newer libjte2 then trying to run a > binary with libjte1? The only user of libjte already has precautions against this: > > Fine check for compatibility happens at compile time by the required > > minimum version (e.g. iso_libjte_req_* versus LIBJTE_VERSION_*) and > > at run time by calling libjte__is_compatible(). This kind of test works well between e.g. xorriso's binary and libisoburn, and then between libisoburn and libisofs. The test between libisoburn/libisofs and libjte has never been challenged but i am optimistic that it works. The compile time test is at https://sources.debian.org/src/libisofs/1.5.0-1/libisofs/messages.c/#L157 (This is how i let a passive piece of source kick an unknown compiler: https://sources.debian.org/src/libisofs/1.5.0-1/libisofs/messages.c/#L174 ) Runtime test https://sources.debian.org/src/libisofs/1.5.0-1/libisofs/messages.c/#L180 Here i let libjte judge whether it finds its own version not older than the compile time version seen by libisoburn (LIBJTE_VERSION_* from libjte.h). As long as you stay with a plausible sequence of version numbers, libjte__is_compatible() should not need your attention. (For the curious: https://sources.debian.org/src/jigit/1.21-1/libjte/libjte.c/#L68 ) Have a nice day :) Thomas
Re: Adding sha256 support into jigdo tools (jigdo and jigit)
On Wed, Nov 13, 2019 at 12:04:31AM +0100, Thomas Schmitt wrote: >Hi, > >SONAME comes from libjte/configure.ac. Dunno from where libburn got the >slightly braindamaged way to compose it. From there it spread over the >other libraries. libjte is its youngest victim. Shrug. Right. >The upstream revision "2.0.0" is not necessarily the .so suffix. >The macro values from libjte.h have to be reflected in AC_INIT(). >Thus: > >-AC_INIT([libjte], [1.0.0], [http://www.libburnia-project.org]) >+AC_INIT([libjte], [2.0.0], [... better URL ...]) ACK. https://www.einval.com/~steve/software/JTE/ >The .so suffix should increase numerically from current 1.0.0. >But you are free to choose the SONAME. > >If you want SONAME=1 (libjte.so.1.1.0, libjte1*.deb) then: > >-LT_CURRENT=1 >-LT_AGE=0 >+LT_CURRENT=2 >+LT_AGE=1 > >If you want SONAME=2 (libjte.so.2.0.0, libjte2*.deb) then: > >-LT_CURRENT=1 >+LT_CURRENT=2 > LT_AGE=0 Yup. -- Steve McIntyre, Cambridge, UK.st...@einval.com "We're the technical experts. We were hired so that management could ignore our recommendations and tell us how to do our jobs." -- Mike Andrews
Re: Adding sha256 support into jigdo tools (jigdo and jigit)
On Tue, Nov 12, 2019 at 11:37:29PM +0100, Thomas Schmitt wrote: >Hi, > >while exploring the need for adaptions in the build system of libsofs, >i wonder whether it is really necessary to create LIBJTE2 in libjte.ver. >Adding new functions is upward compatible. >In my libraries i add them to the current (and only) SONAME version. > >Fine check for compatibility happens at compile time by the required >minimum version (e.g. iso_libjte_req_* versus LIBJTE_VERSION_*) and >at run time by calling libjte__is_compatible(). > >If libisoburn gets a configure.ac test for existence of >libjte_set_checksum_algorithm(), a compiler macro to enable its use, >and a configure option to disable it despite being available, >then it can even be built with the older libjte. >(Fully autostylish.) > >So how about this in libjte/libjte.ver ? > > LIBJTE1 { > global > ... > libjte_get_next_message; > libjte_clear_msg_list; >+libjte_set_checksum_algorithm; >+libjte_set_checksum_path; >+libjte_add_checksum_demand; >+libjte_decide_file_jigdo2; >+libjte_write_match_record2; > local: > *; > }; Surely that works for upwards compatibility, but won't stop somebody from building and depending on the newer libjte2 then trying to run a binary with libjte1? >(Good news is that even libjte2 would probably not cause changes in the > build system of libisofs. Currently it depends only on > and on libjte.so. I hope that both can keep their names. > debian/control depends on libjte1 indirectly via libjte-dev. > Adding "(>= 2.0.0)" would not really be necessary.) Yeah, that side's easy. -- Steve McIntyre, Cambridge, UK.st...@einval.com "When C++ is your hammer, everything looks like a thumb." -- Steven M. Haflich
Re: Adding sha256 support into jigdo tools (jigdo and jigit)
[ Arg, sorry to keep you waiting for responses. Really busy week... :-/ ] On Tue, Nov 12, 2019 at 10:10:04PM +0100, Thomas Schmitt wrote: >Hi, > >i wrote: >> > libjte_set_checksum_algorithm() became necessary > >Steve McIntyre wrote: >> ACK. I'd not thought that through fully myself here, but yes. > >Here you see why it is good to have a mutually curious user-developer >relation. :)) *grin* >> I'm open to changing things like option names here, obviously - >> it's your code exposing them at the top level. > >But you are their only user i know of, as well as i am the only user of >libjte. > >"-jigdo-checksum-algorithm" or "-checksum_algorithm_jigdo" ? > >"checksum_algorithm" or "checksum_jigdo" ? > >The latter already have similarily named neighbors in xorriso: > -checksum_algorithm_iso -checksum_algorithm_template > checksum_iso checksum_template ACK. It's possibly quite confusing. :-/ Let's go with "-jigdo-checksum-algorithm" to keep the separation? >Hm. Did you forget to mention "-jigdo-checksum-algorithm" in >xorriso/emulators.c ? >How can this pass your tests ? Doesn't > $ xorrisofs -jigdo-checksum-algorithm sha256 >/dev/null >throw > xorriso : FAILURE : -as genisofs: Unrecognized option > '-jigdo-checksum-algorithm' > >(Further the new xorrisofs options have to be mentioned in function > Xorriso_genisofs_count_args() in: >static char arg1_options[][41]= { > so that some program parts know how many arguments these options consume.) Ah, probably. I started hacking code here, but ended up using "-jigdo checksum_algorithm" in my debian-cd testing branch so didn't test it fully (as you can tell!). When I was quickly hacking in the changes here, I didn't spend a lot of time looking for all the places that might need updating. >> I based on 1.4.7 as that was the latest >> release I had handy for the merged tree. > >As soon as your tests are done and the names are decided, please send a >new updated diff and a jigit release tarball. ACK, will do. >When i have the jigit release tarball, i will port the diff to >GNU xorriso-1.5.3. When libjte2*.deb is in testing, i will backport the >GNU xorriso changes to upstream libisoburn git and make a patch for the >pending libisoburn-1.5.2 on salsa. >Probably i will have to adapt the build system of libisofs, too. > >Don't invest time in polishing xorriso code or documentation. Just make sure >that it works properly with all your use cases. >I would later appreciate a repeated test with libisoburn1_1.5.2-1_amd64.deb >when it's in unstable, and proof-reading of the docs. Nod! I've just released jigdo 0.8.0 yesterday, and uploaded to unstable. -- Steve McIntyre, Cambridge, UK.st...@einval.com "The problem with defending the purity of the English language is that English is about as pure as a cribhouse whore. We don't just borrow words; on occasion, English has pursued other languages down alleyways to beat them unconscious and rifle their pockets for new vocabulary." -- James D. Nicoll
Re: Adding sha256 support into jigdo tools (jigdo and jigit)
Hi, SONAME comes from libjte/configure.ac. Dunno from where libburn got the slightly braindamaged way to compose it. From there it spread over the other libraries. libjte is its youngest victim. Shrug. The upstream revision "2.0.0" is not necessarily the .so suffix. The macro values from libjte.h have to be reflected in AC_INIT(). Thus: -AC_INIT([libjte], [1.0.0], [http://www.libburnia-project.org]) +AC_INIT([libjte], [2.0.0], [... better URL ...]) The .so suffix should increase numerically from current 1.0.0. But you are free to choose the SONAME. If you want SONAME=1 (libjte.so.1.1.0, libjte1*.deb) then: -LT_CURRENT=1 -LT_AGE=0 +LT_CURRENT=2 +LT_AGE=1 If you want SONAME=2 (libjte.so.2.0.0, libjte2*.deb) then: -LT_CURRENT=1 +LT_CURRENT=2 LT_AGE=0 Have a nice day :) Thomas
Re: Adding sha256 support into jigdo tools (jigdo and jigit)
Hi, while exploring the need for adaptions in the build system of libsofs, i wonder whether it is really necessary to create LIBJTE2 in libjte.ver. Adding new functions is upward compatible. In my libraries i add them to the current (and only) SONAME version. Fine check for compatibility happens at compile time by the required minimum version (e.g. iso_libjte_req_* versus LIBJTE_VERSION_*) and at run time by calling libjte__is_compatible(). If libisoburn gets a configure.ac test for existence of libjte_set_checksum_algorithm(), a compiler macro to enable its use, and a configure option to disable it despite being available, then it can even be built with the older libjte. (Fully autostylish.) So how about this in libjte/libjte.ver ? LIBJTE1 { global ... libjte_get_next_message; libjte_clear_msg_list; +libjte_set_checksum_algorithm; +libjte_set_checksum_path; +libjte_add_checksum_demand; +libjte_decide_file_jigdo2; +libjte_write_match_record2; local: *; }; (Good news is that even libjte2 would probably not cause changes in the build system of libisofs. Currently it depends only on and on libjte.so. I hope that both can keep their names. debian/control depends on libjte1 indirectly via libjte-dev. Adding "(>= 2.0.0)" would not really be necessary.) Have a nice day :) Thomas
Re: Adding sha256 support into jigdo tools (jigdo and jigit)
Hi, i wrote: > > libjte_set_checksum_algorithm() became necessary Steve McIntyre wrote: > ACK. I'd not thought that through fully myself here, but yes. Here you see why it is good to have a mutually curious user-developer relation. :)) > I'm open to changing things like option names here, obviously - > it's your code exposing them at the top level. But you are their only user i know of, as well as i am the only user of libjte. "-jigdo-checksum-algorithm" or "-checksum_algorithm_jigdo" ? "checksum_algorithm" or "checksum_jigdo" ? The latter already have similarily named neighbors in xorriso: -checksum_algorithm_iso -checksum_algorithm_template checksum_iso checksum_template --- The user is king. Throw a coin and implement your decision for the next diff. --- Hm. Did you forget to mention "-jigdo-checksum-algorithm" in xorriso/emulators.c ? How can this pass your tests ? Doesn't $ xorrisofs -jigdo-checksum-algorithm sha256 >/dev/null throw xorriso : FAILURE : -as genisofs: Unrecognized option '-jigdo-checksum-algorithm' (Further the new xorrisofs options have to be mentioned in function Xorriso_genisofs_count_args() in: static char arg1_options[][41]= { so that some program parts know how many arguments these options consume.) --- > I based on 1.4.7 as that was the latest > release I had handy for the merged tree. As soon as your tests are done and the names are decided, please send a new updated diff and a jigit release tarball. When i have the jigit release tarball, i will port the diff to GNU xorriso-1.5.3. When libjte2*.deb is in testing, i will backport the GNU xorriso changes to upstream libisoburn git and make a patch for the pending libisoburn-1.5.2 on salsa. Probably i will have to adapt the build system of libisofs, too. Don't invest time in polishing xorriso code or documentation. Just make sure that it works properly with all your use cases. I would later appreciate a repeated test with libisoburn1_1.5.2-1_amd64.deb when it's in unstable, and proof-reading of the docs. Have a nice day :) Thomas
Re: Adding sha256 support into jigdo tools (jigdo and jigit)
Arg, and our mails have just crossed I see! :-) On Tue, Nov 12, 2019 at 07:24:16PM +0100, Thomas Schmitt wrote: >Hi, > >after finding > > https://git.einval.com/cgi-bin/gitweb.cgi?p=jigit.git;a=log;h=refs/heads/sha256 > > https://git.einval.com/cgi-bin/gitweb.cgi?p=jigit.git;a=blob;f=libjte/libjte.h;hb=refs/heads/sha256 >i now believe to understand that the new API call > > int libjte_set_checksum_algorithm(struct libjte_env *jte_handle, >char *algorithm, int *size); > >became necessary because in API call write_jt_header() the static function >parse_checksum_list() is called only conditionally. So you cannot rely on >it as detector of checksum type. ACK. I'd not thought that through fully myself here, but yes. >I ponder whether i should let xorriso detect or check the type in the >-md5-list file, depending on whether the checksum type was already >explicitely chosen. Maybe, that's up to you I guess. >Guessing that you added a new "aspect" case to Xorriso_jigdo_interpreter(), >i wonder what name you have chosen. >Obvious but also quite long would be "-checksum_algorithm_jigdo" or >"-checksum_algorithm_list" ("checksum_jigdo" or "checksum_list" in >native xorrisian). > >Development by cliffhanger ... i need more popcorn. *grin* I'm open to changing things like option names here, obviously - it's your code exposing them at the top level. Rather than keep you waiting, here's a quick and dirty diff for the changes I've made so far. I based on 1.4.7 as that was the latest release I had handy for the merged tree. I even started on some trivial doc updates, but they're clearly not complete... :-) -- Steve McIntyre, Cambridge, UK.st...@einval.com "... the premise [is] that privacy is about hiding a wrong. It's not. Privacy is an inherent human right, and a requirement for maintaining the human condition with dignity and respect." -- Bruce Schneier --- ./xorriso/lib_mgt.c~ 2017-06-29 22:32:23.0 +0100 +++ ./xorriso/lib_mgt.c 2019-11-10 16:16:41.938305902 + @@ -765,8 +765,11 @@ goto jte_failed; xorriso->libjte_params_given|= 8; } else if(strcmp(aspect, "md5_path") == 0 || - strcmp(aspect, "-md5-list") == 0) { - ret= libjte_set_md5_path(jte, arg); + strcmp(aspect, "-md5-list") == 0 || + strcmp(aspect, "checksum_path") == 0 || + strcmp(aspect, "-checksum-list") == 0) + { + ret= libjte_set_checksum_path(jte, arg); if(ret <= 0) goto jte_failed; xorriso->libjte_params_given|= 16; @@ -813,6 +816,21 @@ if(ret <= 0) goto jte_failed; xorriso->libjte_params_given|= 2048; + } else if(strcmp(aspect, "checksum_algorithm") == 0 || + strcmp(aspect, "-jigdo-checksum-algorithm") == 0) { + int ck_size; + ret= libjte_set_checksum_algorithm(jte, arg, _size); + if(ret <= 0) + { + msg= libjte_get_next_message(jte); + if (msg != NULL) + { +sprintf(xorriso->info_text, "%s", msg); +Xorriso_msgs_submit(xorriso, 0, xorriso->info_text, 0, "FAILURE", 0); + } + goto jte_failed; + } + xorriso->libjte_params_given|= 4096; } else { sprintf(xorriso->info_text, "-jigdo: unknown aspect '%s'", aspect); Xorriso_msgs_submit(xorriso, 0, xorriso->info_text, 0, "FAILURE", 0); --- ./xorriso/xorriso.texi~ 2017-06-29 22:32:23.0 +0100 +++ ./xorriso/xorriso.texi 2019-11-10 16:27:37.565512192 + @@ -4311,7 +4311,11 @@ @* @strong{md5_path} sets the disk_path where to find the .md5 input file. @* -Alias: -md5-list +Alias: -checksum-list +@* +@strong{checksums_path} sets the disk_path where to find the checksums input file. +@* +Alias: -checksum-list @* @strong{min_size} sets the minimum size for a data file to be listed in the .jigdo file and being a hole in the .template file. --- ./xorriso/emulators.c~ 2017-06-29 22:32:23.0 +0100 +++ ./xorriso/emulators.c 2019-11-10 16:24:08.860491692 + @@ -2091,7 +2091,8 @@ strcmp(argpt, "-jigdo-template-compress") == 0 || strcmp(argpt, "-checksum_algorithm_iso") == 0 || strcmp(argpt, "-checksum_algorithm_template") == 0 || - strcmp(argpt, "-md5-list") == 0) { + strcmp(argpt, "-md5-list") == 0 || + strcmp(argpt, "-checksum-list") == 0) { i++; ret= Xorriso_option_jigdo(xorriso, argpt, argv[i], 0); if(ret <= 0) --- ./xorriso/xorrisofs.texi~ 2017-06-29 22:32:23.0 +0100 +++ ./xorriso/xorrisofs.texi 2019-11-10 16:26:09.153079902 + @@ -2035,6 +2035,11 @@ @cindex Jigdo Template Extraction, -md5-list Set the disk_path where to find the .md5 input file. @c man .TP +@item -checksum-list disk_path +@kindex -checksum-list set path of readable checksums +@cindex Jigdo Template Extraction, -checksum-list +Set the disk_path where to find the checksums input file. +@c man .TP @item -jigdo-template-compress "gzip"|"bzip2" @kindex
Re: Adding sha256 support into jigdo tools (jigdo and jigit)
On Tue, Nov 12, 2019 at 09:15:34AM +0100, Thomas Schmitt wrote: >Hi, > >Steve McIntyre wrote: >> the external API for libjte is *very* close to what we had before. > >Now i'm curious. I expected no need for a change, assumed that you'd >automagically detect the checksum type from the lines in the -md5-list >file. I'd prefer to have jigdo creators make an explicit choice in the interface, rather than try to work it out automatically. It's easier to not make mistakes that way. IMHO. Yes, I've chosen differently for the make-image/jigit-mkimage clients - I want that to be as easy and automatic as possible for the end users who shouldn't have to care. >> I've got a simple diff right now that I'm >> just cleaning up and will send you shortly. > >Timing seems lucky. I have a new upstream release waiting for sponsored >upload in > https://salsa.debian.org/optical-media-team/libburn/ > https://salsa.debian.org/optical-media-team/libisofs/ > https://salsa.debian.org/optical-media-team/libsoburn/ > >If you upload a new libjte soon, then we can squeeze-in the necessary >source patches and ./debian changes. I told Dominique Dumont that my upload >request is postponed for now. (Something that fits well in any todo list.) Woot! I started on this last night, having found your gitlab setup etc. I was hoping to set up a fork and send you a MR, but the separation of libisofs, libisoburn and libburn into separate repos makes it a little more awkward to work that way. :-/ Patch coming shortly, I promise! -- Steve McIntyre, Cambridge, UK.st...@einval.com "I can't ever sleep on planes ... call it irrational if you like, but I'm afraid I'll miss my stop" -- Vivek Das Mohapatra
Re: Adding sha256 support into jigdo tools (jigdo and jigit)
On Mon, Nov 11, 2019 at 10:30:09PM +0100, Richard Atterer wrote: >Hi Steve, > >thank you very much for continuing to take care of jigdo! It's been a loong >time since I actively worked on it. (Must have been ~2000 that I started >creating it.) > >I've updated my site to point to the URL you gave - does the text I used >work for you? Awesome, thanks. Now I need to write some text on my side! :-) >I think given that I don't work on (or even provide support for) jigdo >anymore I have no claim to be the "primary" site. Feel free to switch >debian.org to your site once your page is up. Maybe consider adding an >"honourable mention" link pointing to my old page! :-) OK, cool. >Cheers, and thanks again for keeping jigdo alive! No worries! Thanks for an awesome idea! I'll admit to not always being a fan of your coding style, but hey - every person is different! *grin* -- Steve McIntyre, Cambridge, UK.st...@einval.com The two hard things in computing: * naming things * cache invalidation * off-by-one errors -- Stig Sandbeck Mathisen
Re: Adding sha256 support into jigdo tools (jigdo and jigit)
Hi, after finding https://git.einval.com/cgi-bin/gitweb.cgi?p=jigit.git;a=log;h=refs/heads/sha256 https://git.einval.com/cgi-bin/gitweb.cgi?p=jigit.git;a=blob;f=libjte/libjte.h;hb=refs/heads/sha256 i now believe to understand that the new API call int libjte_set_checksum_algorithm(struct libjte_env *jte_handle, char *algorithm, int *size); became necessary because in API call write_jt_header() the static function parse_checksum_list() is called only conditionally. So you cannot rely on it as detector of checksum type. I ponder whether i should let xorriso detect or check the type in the -md5-list file, depending on whether the checksum type was already explicitely chosen. Guessing that you added a new "aspect" case to Xorriso_jigdo_interpreter(), i wonder what name you have chosen. Obvious but also quite long would be "-checksum_algorithm_jigdo" or "-checksum_algorithm_list" ("checksum_jigdo" or "checksum_list" in native xorrisian). Development by cliffhanger ... i need more popcorn. Have a nice day :) Thomas
Re: Adding sha256 support into jigdo tools (jigdo and jigit)
Hi, Steve McIntyre wrote: > the external API for libjte is *very* close to what we had before. Now i'm curious. I expected no need for a change, assumed that you'd automagically detect the checksum type from the lines in the -md5-list file. > I've got a simple diff right now that I'm > just cleaning up and will send you shortly. Timing seems lucky. I have a new upstream release waiting for sponsored upload in https://salsa.debian.org/optical-media-team/libburn/ https://salsa.debian.org/optical-media-team/libisofs/ https://salsa.debian.org/optical-media-team/libsoburn/ If you upload a new libjte soon, then we can squeeze-in the necessary source patches and ./debian changes. I told Dominique Dumont that my upload request is postponed for now. (Something that fits well in any todo list.) Have a nice day :) Thomas
Re: Adding sha256 support into jigdo tools (jigdo and jigit)
Hi Steve, thank you very much for continuing to take care of jigdo! It's been a loong time since I actively worked on it. (Must have been ~2000 that I started creating it.) I've updated my site to point to the URL you gave - does the text I used work for you? I think given that I don't work on (or even provide support for) jigdo anymore I have no claim to be the "primary" site. Feel free to switch debian.org to your site once your page is up. Maybe consider adding an "honourable mention" link pointing to my old page! :-) Cheers, and thanks again for keeping jigdo alive! Richard -- __ , | ) /| Richard Atterer | \/ | http://atterer.org
Bug#887831: Info received (Adding sha256 support into jigdo tools (jigdo and jigit))
Thank you for the additional information you have supplied regarding this Bug report. This is an automatically generated reply to let you know your message has been received. Your message is being forwarded to the package maintainers and other interested parties for their attention; they will reply in due course. Your message has been sent to the package maintainer(s): Steve McIntyre <93...@debian.org> If you wish to submit further information on this problem, please send it to 887...@bugs.debian.org. Please do not send mail to ow...@bugs.debian.org unless you wish to report a problem with the Bug-tracking system. -- 887831: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=887831 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems
Adding sha256 support into jigdo tools (jigdo and jigit)
[ Sent to multiple people and Debian bugs - please respect the reply-to and follow up on the debian-cd list if you have replies/comments. ] Hi folks! For a while we've been working to move away from using MD5 in various parts of Debian, and jigdo is one of the last few things that's still using it now. We've had a few bugs raised about this (#887837 and #887831) and quite some discussion recently. I've been hacking on jigdo and jigit to add support for a new v2 jigdo format which switches from using md5 for internal checksumming to using sha256 instead, and I'm just about done now. I have a few remaining things to do next, that I'd like to ask for some help with (please!) - see further down. Prompt responses would be appreciated if possible. jigdo = I've extended jigdo to support both formats (old and new). Building a new jigdo/template pair requires the user to specify which format they want, while creating/verifying an image will auto-detect the format automatically from the input data. I think that is clearly the best design here. I'm *most* worried about updating the various clients that people may have in the field, using jigdo-lite/jigdo-mirror to make ISO images from the jigdo data that we release with Debian, so that was my first priority. I'm *not* aware of anybody actually using jigdo-file itself to create new jigdo/template pairs these days, but I've done the right thing anyway and added support for sha256 here too. I've forked from Richard's last 0.7.3 release, and put it into my own git server at https://git.einval.com/cgi-bin/gitweb.cgi?p=jigdo.git;a=shortlog;h=refs/heads/upstream along with the various fixes that we already had in Debian since that release. I've built and tested binaries locally with both jigdo formats, including on Windows. All looks good here. \o/ jigit/libjte I've also updated and extended my own jigit/libjte code to work with both formats, and I'm about to release those. The changes are not too big, and the external API for libjte is *very* close to what we had before. I've already updated a local copy of xorriso to use it, and the changes are tiny! \o/ genisoimage === I am *not* planning to update my code in genisoimage to use the new jigdo v2 format. We don't use genisoimage at all in the Debian images team any more, having moved to xorriso instead. The only reason to even think about updating genisoimage would be for powerpc images. While the debian-ports people are still supporting powerpc and periodically releasing new unofficial CD/DVD images for it, I don't think jigdo is needed here. *If you think differently*, let me know... Publishing the new format = debian-cd and some of our backend setup on our cdimage sites will need some minor updates to support the new sha256 format as well, but that's not urgent yet. We must *not* switch to publishing the new v2 jigdo format for a while (I'm thinking maybe 12 months?), to give people the chance to update their clients. I also don't want to leave this *too* long, as the Debian ftpmaster team and others would like to ditch md5 soon. We'll need to make noise about this, and update web pages etc. to mention the change. New links to new tools, etc. Richard === With your blessing, I'd like to release my new code as jigdo version 0.8.0. If you're ok with that, could you please update your jigdo web pages to mention this? I'll add a page at https://www.einval.com/~steve/software/jigdo/ that you can link to. I'll add some docs, and links back to you (of course!) and download links for Windows binaries etc. So far I've left the creator information in newer jigdo files pointing at your site as you're the inventor, but I'm also happy to change that if you'd like and reduce your web traffic - just let me know please! :-) Mattias === You're the person normally working with people using jigdo tools to mirror Debian's CD/DVD releases. We'll need to ask people to update all their tools to enable using the new v2 format. What systems are they normally using? I'm guessing a mix of Debian systems of various versions, plus maybe a few other OSes? I'm happy to do Debian backports builds of the new tool versions to help support people, but I don't know: (a) what else might need to be supported (b) what timescale these people would be happy with or updates Obviously, we don't want to be pushing new format versions until the mirror network is ready to take them. But we want that to be as soon as practically possible. Thomas == You've done an awesome job with xorriso and the libjte integration! It's been really easy to drop in my new libjte code and have xorriso generate the new format. I've got a simple diff right now that I'm just cleaning up and will send you shortly. -- Steve McIntyre, Cambridge, UK.st...@einval.com You lock the door And throw away the key There's someone in my head but it's not me