Hi, Sorry for not being online, my sister was there and it's my holidays, so I decided it was nicer to be with her and enjoy life. Though I have the IRC logs, and want to follow-up on the discussion.
1/ Releasing updated images What Sledge did was comparing the security repository with the versions of packages in the archive, so we could generate a new image on Casulana (and later upload to Paterson) if there was a package with a newer version. The script, if I'm not mistaking, is here: https://salsa.debian.org/images-team/live-setup/blob/master/bin/check-openstack-updates What IMO we need, is something that opens up the .json of the last uploaded image to Paterson, and compares it to the security repo. Though in here: http://cdimage.debian.org/cdimage/cloud/buster/ I'm seeing images with dates, which seems to more or less match the release of the "other" OpenStack images, so I wonder if that work is already done or what... Even if we do have such implementation already (which I cannot confirm), it is my opinion that we still don't have feature parity. With Sledge, we made sure that the current image always has the same URL, ie: http://cdimage.debian.org/cdimage/openstack/current/ and that the image name includes the stable minor release number. I know I am the only one in the team that thinks it is useful though, but I was no convince by the only argument which basically was "I don't care, why should I ...". With the name: "debian-10.4.3-20200610-openstack-amd64.qcow2" we know we're talking about the 3rd micro release of the image after the 4th minor release of Buster, and it is my opinion that this is useful. For example, I can tell very quickly that this is the last version with cloud-init 18.8. I wouldn't be able to tell with only the date as hint... I still don't understand why we would need to keep daily images on Salsa. Aren't they useless if we have the feature above? Can't we then just drop everything that is, let's say, one week old, as we'd otherwise just be keeping duplicates of the same thing? 2/ Making images smaller As I showed, it's possible to resize partitions very simply. However, the FAI produced image is having a very weird layout. The partition 1 is at the end, and then we have partition 14 and 15 before that. Why can't we have something more simple, like: 1/ The EFI partition as partition 1 2/ The Ext4 partition as partition 2 Is there any reason it's the way it is right now? Why is the EFI partition FIRST, but then labelled as partition 15? By the way, what is the partition 14 for? It has a bios_grub flag, but it cannot be mounted. What's the point? In what I tried, which is insipred from the automatic resize of openstack-debian-images, resize SEEM ok, but it destroys the partition. Here's my script: # Uncompress the disk.raw tar -xvf debian-10-generic-amd64-daily-20200812-356.tar.xz # Create the loop devices TMPFILE_KPARTX=$(mktemp -t shrink-XXXXXX) kpartx -av disk.raw >${TMPFILE_KPARTX} echo "kpartx output:" cat ${TMPFILE_KPARTX} LOOP_DEVICE=$(cat ${TMPFILE_KPARTX} | head -n 1 | awk '{print $3}') echo "Loop device is ${LOOP_DEVICE}" rm ${TMPFILE_KPARTX} # Mount the first partition, calculate it's space + 60M, then unmount MOUNT_FOLDER=$(mktemp -d -t shrink-mount-folder-XXXXXX) mount /dev/mapper/${LOOP_DEVICE} ${MOUNT_FOLDER} USED_SPACE=$(df --block-size=M ${MOUNT_FOLDER} | tail -n1 | awk '{print $3}' | sed s/M//) WANTED_SPACE=$(( ${USED_SPACE} + 60)) umount ${MOUNT_FOLDER} rmdir ${MOUNT_FOLDER} # Resize the filesystem... fsck.ext4 -f /dev/mapper/${LOOP_DEVICE} resize2fs -f /dev/mapper/${LOOP_DEVICE} ${WANTED_SPACE}M # See how many blocks the filesystem now takes, and calculate the partition and image size FINAL_FS_BLOCKS=$(tune2fs -l /dev/mapper/loop0p1 | awk '/Block count/{print $3}') FINAL_PART_SIZE=$(expr '(' $FINAL_FS_BLOCKS + 258 ')' '*' 4 '/' 1024) FINAL_IMAGE_SIZE=$(( ${FINAL_PART_SIZE} + 128 )) # Remove the loop devices kpartx -d disk.raw # Resize the partition yes | parted ---pretend-input-tty disk.raw resizepart 1 ${FINAL_PART_SIZE}M # Finally, truncate the final disk truncate -s ${FINAL_IMAGE_SIZE}M disk.raw After the parted + truncate, the partition 1 is broken, I don't understand why, but that's probably due to this weird layout. 2/ Octavia images larger than 256 MB I saw that the discussion was around publishing raw images. Since we can't store the Octavia images on Salsa, could we imagine publishing them on Paterson with the pipeline, without storing them on Salsa at all? Does anyone have a suggestion? 3/ Raw images If someone wants to implement HTTP compression (IMO a good idea), I can easily confirm if that works with: - Openstack Mitaka (very old, from 2016) - OpenStack Rocky (ie: what's in Buster right now, released in 2018) - OpenStack Stein (I could ask Michal Arbet, or deploy it to test) - OpenStack Train (the release from last October) - OpenStack Ussuri (the latest release, from last May) I don't have access to any other deployments. However, I still agree that this is probably a too early optimization that we could delay for later. 4/ Image finder My merge request is still pending here: https://salsa.debian.org/cloud-team/image-finder/-/merge_requests/89 I still don't understand why my debian-cloud-image-finder-db-recreate doesn't work. Doing an strace, I can see that the configuration file is read, and that the connection to the DB is successfuly opened (this worked with both Postgress and MySQL), though the DB isn't re-created. This goes in the internals of the image-finder that are beyond me. It feels like the calls to: db.drop_all() db.create_all() db.session.commit() have no effect, and I don't understand why. Arthur hasn't follow-up on this. Does any of you has the time to look? Cheers, Thomas Goirand (zigo)
