On Thu, 26 Aug 2021 19:49:37 +0200 "Thomas Schmitt" <scdbac...@gmx.net> wrote:
> Hi, > > now i know that i should have asked for the test-suite.log file. > > The problem turned out to be in Joliet, not in ISO 9660. > Because Joliet gets its file names and other texts encoded as UCS-2 > 16-bit characters it is indeed inavoidable to define the meaning of > the bytes in a Unix file name. > The UTF-8 sequence \303\251 has no meaning as ANSI_X3.4-1968 > characters (which are what we learned to love as 7-bit ASCII). So > iconv fails. > > My remedy proposal > > xorriso ... -as mkisofs -input-charset UTF-8 -output-charset UTF-8 > ... Do there need to be any UTF-8 locales installed (or any locales for that matter) for this to work? My guess is no. > turned out to be a valid alternative to > > LANG=en_US.UTF-8 > > Using above charset options unconditionally would make obsolete the > warning in Glenn Washburn's patch at least for the xorriso runs: > > +elif [ -n "${LANG##*UTF*}" ]; then > + echo "WARNING: LANG=$LANG appears to not be unicode, > international file test may fail." > > It would still be beneficial to default LANG="" to LANG=en_US.UTF-8 > because 7-bit ASCII is really an odd character interpretation for Unix > filenames. (HP-UX from 1986 maybe ...) Correct me if I'm wrong, but this logic about defaulting LANG is more general than the iso9660_test. Would you agree that this should be applied to the tests a a whole? If yes, then I should probably remove this patch altogether and put this code elsewhere. -------------------------------------------------------------------------- > > Noob question: > How can i force "make check" to do something at its second run ? I might be answering this question below. -------------------------------------------------------------------------- > Long story: > > On a Debian 10 with a freshly made GRUB git clone i see indeed a > failure of iso9660_test with mutilated volume id and a mutilated > exotic filename after > > make SUBDIRS= LANG= TESTS=iso9660_test check > > On the same machine with the same xorriso, the same file name from the > log and the volume id taken from the script, i get with > > LANG= xorriso -as mkisofs -o test.iso -V "...exotic.characters..." > test_tree > > a somewhat confused xterm (or libreadline) but > > xorriso -indev test.iso -find / -- > > reports the exotic volume id and the exotic filename. > > The "make ... check" run says on the second try > make[3]: 'iso9660_test' is up to date. > but i can reproduce the problem by > > LANG= ./grub-fs-tester joliet Yes the iso9660_test is up to date in the sense that it does not need to be generated from iso9660_test.in. However, "make check" should still be running the test. IOW, the tests are always rerun, there is no make rule that does not run tests based on them having been perviously run (that I know of). > The trigger for the file name mutilation seems to be the mkisofs > emulation option for creating a Joliet tree > -J > With default settings of mount(8) or xorriso's file name display the > problem becomes visible only if no Rock Ridge info was added to the > ISO 9660 tree. This is caused by xorriso command > --rockridge off > > Nevertheless file(1) and xorriso still see the exotic volume id. > Binary inspection shows that the ISO 9660 volume id (at offset 32768 > + 40) is originally exotic, but the Joliet volume id (at offset 34816 > + 40) was mutilated. > > So obviously the problem sits in the Joliet name conversion to UCS-2. > It is inavoidable if the file names do not match the input character > set (defaulted from locale) because the Unix file names cannot be > just copied to Joliet directory records. > > Now that i can reproduce the problem i can also confirm that my remedy > proposal would work as expected. It preservies the exotic names in the > Joliet tree and the exotic volume id in the Joliet supplementary > volume descriptor. As i understand it, your proposed change is entirely with in grub-fs-tester. Would you submit a patch with the changes you've outlined that would fix this issue for when LANG is empty? Based on the work you've done, I think it makes most sense for you to author the patch. And I'll test/review it. Glenn _______________________________________________ Grub-devel mailing list Grub-devel@gnu.org https://lists.gnu.org/mailman/listinfo/grub-devel