Bug#759406: debian-cd: Merge iso images into usb drive thanks to cat
Hi, There are descriptions in the web for boot media which contain multiple ISOs. Yes, but these tools, in theory, won't work perfect with Debian Installer till #724931 and #759744 are fixed. Lack of loop mounting is quite some handicap. Repartition free space to store extra ISO images. I would rather call it Claim free space on device as new partition. We could make a specialized little program for that. MBR is easy to manipulate. Device size can be inquired linux-specifically by a ioctl() or portably by binary search. Question is what to do with GPT. (Is Live-CD prepared for EFI isohybrid at all ?) The GPT specs prescribe that a backup table is to be stored at the very end of the storage device. Debian amd64 has this backup at the end of the ISO image. This already confuses some partition editors. To get a neat partitioning, GPT would either have to be deleted or the backup table has to be moved. Moving demands patching of fields in the backup and in the main GPT. This includes computation of new CRCs for both. Then you would need some of the implementation about detecting new ISOs from the partition directory and using them that it's already found at: #724931 . Yes. This is a tricky part after first stages of booting. I personally like the cat idea because is very new but at the same we are saving data into an space that could be thought of being empty. One could combine cat with the creation of a new partition or with expansion of the first partition up to the end of the storage device. (Again, GPT will impose extra effort.) I ask myself what's the opinion of debian-cd people on this subject. I assume they are waiting whether this discussion yields a lean and stable proposal. ls -l debian-7.6.0-i386-CD-1.iso Yes, you are right. Although fdisk -l also shows this size somewhere. One could also inquire the size of the ISO image. :)) (Bytes 32768 + 81 to + 83 gives the number of blocks as little endian, + 84 to + 87 as big endian) But actually you want to know where the next ISO will start. With cat, this depends on the previous image file sizes. (One could use dd seek= to put the ISOs at predefined block addresses regardless of the size of the previous ISOs.) What are the use cases where it will be of help ? When someone want to use CD1, CD2 and CD3 from the same usb. E.g. giving the usb to someone else who is offline. I now wonder how much effort it would be to merge Debian installation ISOs into a single ISO filesystem. The ISO part could be easily done. But there are files inside the ISOs which would need to be merged or otherwise adapted. The benefit would be to get a normal Debian ISO with no need for any new installer features. Have a nice day :) Thomas -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#759406: debian-cd: Merge iso images into usb drive thanks to cat
Hi, i am watching this bug as upstream developer of xorriso, which packs up Debian ISOs under control of debian-cd. So i can mainly comment about early boot stages and mounting of ISO 9660, not much about Debian installer issues which arise after or before these steps. From your initial report i got the impression that you wanted to let the user choose any of the ISOs for booting. There are descriptions in the web for boot media which contain multiple ISOs. Google boot multiple iso found: http://www.pendrivelinux.com/boot-multiple-iso-from-usb-via-grub2-using-linux/ But now it seems that you want to install Debian from the first ISO onto hard disk and then only want to use the other ISOs as alternative sources of packages. So i wonder why you do not dd the first ISO onto the USB stick (which will install an MBR and declare partition 1). Then you could use a partition editor to create partition 2 which covers the remaining blocks of the USB stick. There you could create a mountable filesystem (e.g. FAT) and store the other ISOs for loop-mounting. Well, this works if you use mount command from util-linux package. busybox's mount does not implement offset feature, so it either ignores it or complains about it. Originally offset is a feature of losetup(8) which creates a loop device for mount if option loop is given. The Archlinux example shows option loop. Did you try busybox mount with loop ? == Getting to know Debian CD 1 size == # fdisk -l debian-7.6.0-i386-CD-1.iso This tells the size of the MBR partition which was wrapped around the Debian image by xorriso. It may or may not give the size of the ISO image data file. Since you concatenate the image files and not the partition contents, you should rather inquire the file size. Like: ls -l debian-7.6.0-i386-CD-1.iso chroot /target /bin/mount /dev/sdb /mnt/apt2 -o offset=679477248 -t iso9660 you will see that you can read Debian CD 2 disk while it's have been concatenated to the USB !!! In the case of a FAT filesystem in /dev/sdb2, which holds the other ISOs you would first mount the FAT mkdir /target/mnt/usb_fat chroot /target /bin/mount /dev/sdb2 /mnt/usb_fat and then mount the desired ISO from inside the FAT: mkdir /target/mnt/apt2 chroot /target /bin/mount -o loop /mnt/usb_fat/apt2.iso /mnt/apt2 Note that no size information is needed to find the ISO but rather its file name inside the FAT filesystem. I assume that the name of debian-7.6.0-i386-CD-2.iso copy in the FAT is /apt2.iso. One could also put the desired ISOs into a ext2/3/4 filesystem instead of a FAT. This would allow the use of long filenames. If they all are put into an ISO then they would be write-protected for normal users. == Busybox based Implementation == This implementation would involve hacking busybox's mount Would probably not be necessary if the ISOs are presented as data files of a mountable filesystem. Somehow a tool would be needed so that the user can mount its Debian CD 2 from its USB as easier as possible (Maybe a noauto entry in fstab). fstab could contain lines for /target/mnt/fat and /target/mnt/apt*. For Windows people it would be nice to have a GUI where you can select (always with CD 1 as the first one) all the cds you want and then concatenate them into a USB device. The normal MS File Manager could populate the mounted FAT filesystem. As you might suspect you need to know the iso size in advance so that this works in a reasonable way. Not if the ISOs are data files inside a filesystem. Do you think it's something that it's easy to implement ? Not really. I get the impression that you plan to create a kindof minimal filesystem. My main objection is that suitable filesystems already exist and are supported by a wide range of operating systems and tools. Outside my personal scope is the motivation for your proposal. A Debian person might ask: What are the use cases where it will be of help ? What benefits would it have for Debian's ISO production and download sites ? Have a nice day :) Thomas -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#759406: debian-cd: Merge iso images into usb drive thanks to cat
Well, this works if you use mount command from util-linux package. busybox's mount does not implement offset feature, so it either ignores it or complains about it. == Getting Debian images == I have downloaded * debian-7.6.0-i386-CD-1.iso * debian-7.6.0-i386-CD-2.iso for doing my tests. == Getting to know Debian CD 1 size == # fdisk -l debian-7.6.0-i386-CD-1.iso GNU Fdisk 1.2.5 Copyright (C) 1998 - 2006 Free Software Foundation, Inc. This program is free software, covered by the GNU General Public License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. Disk /mnt/ssd/adrian_sata/conocimiento/distribuciones_isos/debian-installer/debian-7.6.0-i386-CD-1.iso: 679 MB, 679477248 bytes 64 heads, 32 sectors/track, 648 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Device Boot Start End Blocks Id System /mnt/ssd/adrian_sata/conocimiento/distribuciones_isos/debian-installer/debian-7.6.0-i386-CD-1.iso1 * 1 648 663536 17 Hidd HPFS/NTFS So size, in bytes, is: 679477248 . == Additional notes == Although I am booting from a cdrom, the same procedure can be replicated if you boot from a usb hard disk. == Virtual machine description == Cdrom 1: Debian Live CD Cdrom 2: Debian CD 1 Cdrom 3: Debian CD 2 Hard disk 1: 8 GB . Blank disk for a new Debian installation. Hard disk 2: 4 GB . Blank disk. USB that has concatenated images. == USB preparation == I boot the virtual machine from: Cdrom 1: Debian Live CD . Let's see how GNU/Linux detects the disks: /dev/sr0: Cdrom 1: Debian Live CD /dev/sr1: Cdrom 2: Debian CD 1 /dev/sr2: Cdrom 3: Debian CD 2 /dev/sda: Hard disk 1: 8 GB . Blank disk for a new Debian installation. /dev/sdb: Hard disk 2: 4 GB . Blank disk. USB that has concatenated images. Now we are going to concatenate the disks. Be aware that final user will not concatenate cdrom devices but iso files directly. I use cdrom device because, as I use Virtualbox, it makes it easier for me. cat /dev/sr1 /dev/sr2 /dev/sdb sync == Testing the idea == Now I change a bit cdroms so that I boot from Debian CD 1. Cdrom 1: Debian CD 1 Cdrom 2: Debian CD 2 I boot the virtual machine from: Cdrom 1:Debian CD 1. So, first of all install Debian (making sure that Standard tools from system are installed although I'm not sure if they are absolutely needed or not) into /dev/sda hard disk up to the point where you are asked if you want to install bootloader or not. Don't ask the question. Instead do: ALT+F2 . This is going to open a shell where sda is mounted in /target and where, let's remember, sdb is: Hard disk 2: 4 GB . Blank disk. USB that has concatenated images. mkdir /target/mnt/apt2 chroot /target /bin/mount /dev/sdb /mnt/apt2 -o offset=679477248 -t iso9660 Now if you run: more /target/mnt/apt2/README.txt you will see that you can read Debian CD 2 disk while it's have been concatenated to the USB !!! == Target based Implementation == This is the implementation you have seen before. You need to have a basic installation of Debian into the hard disk. That's not problem because it is provided by Debian CD 1. Once this basic installation (included the Grub installation to disk) has finished the user is prompted: Do you want to search packages into appended images in your usb? If the user answer yes, then the offset mount is tested, packages path is added to /etc/apt/sources.list (or whatever), apt-get update is run, and you somehow let the user add more packages in the installation if he wants to. These packages are found in second cdrom. == Busybox based Implementation == This implementation would involve hacking busybox's mount command so that it supports offset parametre. Then asking for the user which packages to install would be done once, because the next Debian CDs would have been found at first stages of the installation. == Using Disk 2 from USB everyday == Somehow a tool would be needed so that the user can mount its Debian CD 2 from its USB as easier as possible (Maybe a noauto entry in fstab). This way when running: apt-get update the files will be finally found in /mnt/apt2 (or whatever) . == Special tool for making USBs in Windows == For Windows people it would be nice to have a GUI where you can select (always with CD 1 as the first one) all the cds you want and then concatenate them into a USB device. == Padding when generating iso images == As you might suspect you need to know the iso size in advance so that this works in a reasonable way. I think the idea is to embed the disk size in a file somewhere inside of each one of the disks. How do you manage to know the final size before generating the disk? By padding it. So, if you want your disk to be 900
Bug#759406: debian-cd: Merge iso images into usb drive thanks to cat
Package: debian-cd Severity: wishlist Dear Maintainer, == Introduction == In Debconf 14 Debian installer and CD BoF talk which it's archived here: http ://meetings-archive.debian.net/pub/debian- meetings/2014/debconf14/webm/Debian_installer_and_CD_BoF.webm someone in the public suggested to concatenate 1 GB size images (Skip to time: 28:58 in the video ) so that you can use them in 4 GB usbs. This will also mean that you won't have to maintain 1 GB, 2 GB, 4 GB and 8 GB size installer images but only: 1 GB, 1 GB, 1 GB and 1 GB with most no repeated packages. So you would only need to do something like: cat debian-7.6.0-i386-USB-1.iso debian-7.6.0-i386-USB-2.iso debian-7.6.0-i386-USB-3.iso debian-7.6.0-i386-USB-4.iso /dev/sdc where sdc is your hard disk device. Steve McIntyre replied that was not possible because filesystems don't work that way. Another reply from the same person that initially suggested the idea was: But we can fix it. Finally Steve turned down the idea. == Overview == After thinking about it, although I'm not sure 100%, I think it could work. The idea is, as you know, how big your initial image size you can mount the next image. How would be do that ? Just by offsetting the mount of the partition. Just as qemu does: https://wiki.archlinux.org/index.php/QEMU#Mounting_a_partition_inside_a_raw_disk_image So the suggested: mount -o loop,offset=32256 disk_image mountpoint would have to be rewritten as: mount -t auto -o offset=966367673856 /dev/sdc /mnt/apt-2 where the offset is based on an hypotetical 900 MB image as is: (900*1024*1024*1024)+32256 . == Possible problem == The only doubt I have is that when you initially dd the first 1 GB image you are defining a 1 GB partition table. So I'm not sure if this 1 GB partition table will disable the kernel to read data on usb hard drive beyond the 1 GB limit or not. I will try to make some tests in the next days to prove if the minimal functionality needed actually works or if it would need hacks on how Gnu/Linux kernel reads devices bigger than its partition table suggested size. == Implementation idea == (Loop till you do not find a new valid dd'ed hard disk partition) :Try to mount the next dd'ed hard disk first partition in the disk. Add the packages directory found on that partition to /etc/apt/sources.list (or whatever you use in Debian Installer). Run apt-get update to add these packages to the available ones. == Similar bugs == This is a very similar bug to: #617853 . The only difference is that in my implementation the debian-installer sorts out the next images while in the other bug I suppose the tool is meant to read inside the image, extract its contents and copy the packages in the usb. Feel free to relate both bugs if it's worth. Actually Steve McIntyre describes this bug in his talk. -- System Information: Debian Release: jessie/sid APT prefers unstable APT policy: (500, 'unstable') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 3.13-1-amd64 (SMP w/4 CPU cores) Locale: LANG=es_ES.UTF-8, LC_CTYPE=es_ES.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
