Re: Documenting the generation of the live images in Debian

2020-03-23 Thread Steve McIntyre
Hi Roland!

On Sat, Mar 21, 2020 at 05:27:55PM +0100, Roland Clobus wrote:
>Hello Steve McIntyre, Eduard Bloch and lists,
>
>On 11/07/2019 12:21, Steve McIntyre wrote on debian-live:
>> On Thu, Jul 11, 2019 at 03:13:48PM +0700, Azure Zanculmarktum wrote:
>>> Where can I find the source to build debian live? ... the live-build
>config.
>>
>> The setup we use is in git:
>>   https://salsa.debian.org/images-team/live-setup/
>...
>>
>https://salsa.debian.org/images-team/live-setup/blob/master/available/run-30live-wrapper
>
>As part of my documentation effort for live-manual, I would like to
>include the procedure for the official Debian CD image generation (which
>will effectively replace chapter 16) in
>https://live-team.pages.debian.net/live-manual/html/live-manual/procedures.en.html
>
>This mail is rather technical, but I want to document the steps that
>I've taken.
>
>From what I've seen in the live-setup repository [5], there are two
>strategies for generating Debian CD images.
>
>A) Use lb config (run-30live-build)
>B) Use lwr (run-30live-wrapper)
>
>I downloaded 'debian-live-10.3.0-amd64-standard.iso' from [1], which
>contains the string 'Official Debian GNU/Linux Live 10.3.0 standard
>2020-02-08T12:27'. This string is generated by the run-30live-wrapper
>script, so I assume that the official images are generated by method B, lwr.

Correct. We used to use live-build up to and including 8.x (Jessie). Then
we switched to live-wrapper.

>I've read through run-30live-wrapper, and I think by now I have
>correctly extracted the commands for the generation of the Debian Stable
>CDs, but I'm unsure how to proceed.
>
>I have the following questions/requests:
>* lwr depends on vmdebootstrap, which is currently only in Buster
>(stable). According to its author vmdebootstrap is to be replaced by
>vmdb2 or something else [2]. Is someone working on this?

The current live-wrapper code, and vmdebootstrap, are both basically
dead IMHO. I've suggested moving to something else supported like FAI
instead, like the Debian cloud images. highvoltage has other
ideas. I'm not working on anything live-related at all any more.

>* I've seen a huge speed-up when I'm using a ramdisk, but that needs a
>merge request to be accepted [3].
>  When invoking 'chroot' in live-setup/available/live-customise.sh the
>variable TMPDIR must not be set, otherwise it is not possible to
>generate temporary files within the chroot.

Ah, thanks for mentioning. For some reason I'd not seen any
notifications for that. Now merged.

>* To reduce the amount of downloads, I am using a slightly patched
>version of apt-cacher-ng, in order to be able to use the installer. See
>my bug report [4]

ACK. On the official build machine we have a local mirror on the host,
and the VMs doing the build all point to that.

>  The live-wrapper script downloads the installer, but apt-cacher-ng
>rejects a path ending in a slash. The following command simulates this
>download:
>
>  wget -S
>"http://localhost:3142/deb.debian.org/debian/dists/buster/main/installer-amd64/current/images/cdrom/;
>
>* The run-30live-wrapper mentions wheezy and jessie, so it is rather old
>
>I would like to add to the live-manual the steps to build an image from
>the current stable and testing versions of Debian. Do you have
>hints/ideas how I can proceed?

It sounds like you understand the process well already!

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
< sladen> I actually stayed in a hotel and arrived to find a post-it
  note stuck to the mini-bar saying "Paul: This fridge and
  fittings are the correct way around and do not need altering"



Re: Patching apt-cacher-ng (Was: Documenting the generation of the live images in Debian)

2020-03-21 Thread Roland Clobus
On 21/03/2020 18:53, Eduard Bloch wrote:

>> In reply to the patch at https://bugs.debian.org/954437

> I cannot accept your patch as-is. Because it's brute-force, a directory
> is not neccessarily a non-volatile file. Can you give me any hints how
> to distinguish between volatile and non-volatile directories? Maybe the
> same as it's done for for other d-i files, by regex, so
> 
>   "|/dists/.*/installer-[^/]+/[0-9][^/]+/images/.*" // d-i stuff 
> with revision
> 
> means frozen contents and
> 
>   "|/dists/.*/installer-[^/]+/[^0-9][^/]+/images/.*" // d-i stuff 
> but not containing a date (year number) in the revision directory (like 
> "current", "beta", ...)
> 
> is volatile? (those regexps are already defined, I just need to change
> your patch to make them applicable to directories)

Agreed. The patch is a brute-force hack, but at least it allowed me to
build my first CD unofficial images.

After I've written my summary to the list, I dug deeper into
live-wrapper. It uses either the released installer (using the apt
proxy), which doesn't contain versioning information in the path, but
will most probably be rather non-volatile, from
https://deb.debian.org/debian/dists/buster/main/installer-amd64/current/images/cdrom/
or the daily build directly (without apt proxy) from
https://d-i.debian.org/daily-images/%s/daily/cdrom/

However, I currently think that my issue is located within the
live-wrapper package. lwr performs a download which isn't used. In
lwr/utils.py the line check_url(base_url) should be removed. Then I
wouldn't need a patched version of apt-cacher-ng at all...

> Also, what's your deadline? I was planing to release a new version of
> ACNG in a couple of days anyway. Do you also need it in backports?

If you still would like to include the patch, I think that the content
listing of a directory should always be considered volatile.

But, as said, I think that the bug should be reassigned to the
live-wrapper package.

With kind regards,
Roland Clobus



signature.asc
Description: OpenPGP digital signature


Re: Documenting the generation of the live images in Debian

2020-03-21 Thread Eduard Bloch
Hallo,
* Roland Clobus [Sat, Mar 21 2020, 05:27:55PM]:

> * To reduce the amount of downloads, I am using a slightly patched
> version of apt-cacher-ng, in order to be able to use the installer. See
> my bug report [4]
>   The live-wrapper script downloads the installer, but apt-cacher-ng
> rejects a path ending in a slash. The following command simulates this
> download:
> 
>   wget -S
> "http://localhost:3142/deb.debian.org/debian/dists/buster/main/installer-amd64/current/images/cdrom/;
> 

I cannot accept your patch as-is. Because it's brute-force, a directory
is not neccessarily a non-volatile file. Can you give me any hints how
to distinguish between volatile and non-volatile directories? Maybe the
same as it's done for for other d-i files, by regex, so

"|/dists/.*/installer-[^/]+/[0-9][^/]+/images/.*" // d-i stuff 
with revision

means frozen contents and

"|/dists/.*/installer-[^/]+/[^0-9][^/]+/images/.*" // d-i stuff 
but not containing a date (year number) in the revision directory (like 
"current", "beta", ...)

is volatile? (those regexps are already defined, I just need to change
your patch to make them applicable to directories)

Also, what's your deadline? I was planing to release a new version of
ACNG in a couple of days anyway. Do you also need it in backports?

Best regards,
Eduard.



Re: Documenting the generation of the live images in Debian

2020-03-21 Thread PICCORO McKAY Lenz
this mail are very important.. but i cannot find a archive link to
distribute it! how can i get a link of this archive mail?
ALSO:
El sáb., 21 de mar. de 2020 a la(s) 12:44, Roland Clobus (rclo...@rclobus.nl)
escribió:

>
> https://salsa.debian.org/images-team/live-setup/blob/master/available/run-30live-wrapper
>
> As part of my documentation effort for live-manual, I would like to
> include the procedure for the official Debian CD image generation (which
> will effectively replace chapter 16) in
>
> https://live-team.pages.debian.net/live-manual/html/live-manual/procedures.en.html
>
> This mail is rather technical, but I want to document the steps that
> I've taken.
>
> From what I've seen in the live-setup repository [5], there are two
> strategies for generating Debian CD images.
>
> A) Use lb config (run-30live-build)
> B) Use lwr (run-30live-wrapper)
>

this only generates lasted today images? how to generates for debian 9 and
debian 8 live? and cd/dvd 1 set?


>
> I downloaded 'debian-live-10.3.0-amd64-standard.iso' from [1], which
> contains the string 'Official Debian GNU/Linux Live 10.3.0 standard
> 2020-02-08T12:27'. This string is generated by the run-30live-wrapper
> script, so I assume that the official images are generated by method B,
> lwr.
>
> I've read through run-30live-wrapper, and I think by now I have
> correctly extracted the commands for the generation of the Debian Stable
> CDs, but I'm unsure how to proceed.
>
> I have the following questions/requests:
> * lwr depends on vmdebootstrap, which is currently only in Buster
> (stable). According to its author vmdebootstrap is to be replaced by
> vmdb2 or something else [2]. Is someone working on this?
> * I've seen a huge speed-up when I'm using a ramdisk, but that needs a
> merge request to be accepted [3].
>   When invoking 'chroot' in live-setup/available/live-customise.sh the
> variable TMPDIR must not be set, otherwise it is not possible to
> generate temporary files within the chroot.
> * To reduce the amount of downloads, I am using a slightly patched
> version of apt-cacher-ng, in order to be able to use the installer. See
> my bug report [4]
>   The live-wrapper script downloads the installer, but apt-cacher-ng
> rejects a path ending in a slash. The following command simulates this
> download:
> 
>   wget -S
> "
> http://localhost:3142/deb.debian.org/debian/dists/buster/main/installer-amd64/current/images/cdrom/
> "
> 
> * The run-30live-wrapper mentions wheezy and jessie, so it is rather old
>
> I would like to add to the live-manual the steps to build an image from
> the current stable and testing versions of Debian. Do you have
> hints/ideas how I can proceed?
>
> With kind regards,
> Roland Clobus
>
> --- Commands
> This is the list of commands that I use (all as root):
> mkdir -p /tmp/mytmp
> mount -t tmpfs tmpfs /tmp/mytmp
> export TMPDIR=/tmp/mytmp
> lwr -m http://localhost:3142/deb.debian.org/debian \
> --apt-mirror http://deb.debian.org/debian \
> --customise /home/roland/git/live-setup/available/live-customise.sh \
> --architecture amd64 \
> -d buster \
> --isolinux \
> --grub \
> --log stderr \
> --installer \
> -t live-task-standard \
> -f "" \
> --base_debs "eject pciutils usbutils keyutils keyboard-configuration
> console-setup lvm2 mdadm dmsetup cryptsetup dmraid e2fsprogs btrfs-progs
> xfsprogs jfsutils grub-efi-amd64 grub-efi-amd64-bin grub-pc
> grub-efi-amd64-signed shim-signed" \
> --description "Unofficial Debian GNU/Linux Live 10.3 standard" \
> --volume_id "d-live 10.3 st amd64" \
> --image_output debian-live-10.3.iso
>
> --- Links
> [1] https://www.debian.org/CD/live/
> [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=910201
> [3] https://salsa.debian.org/images-team/live-setup/-/merge_requests/1
> [4] https://bugs.debian.org/954437
> [5] https://salsa.debian.org/images-team/live-setup
>
>


Documenting the generation of the live images in Debian

2020-03-21 Thread Roland Clobus
Hello Steve McIntyre, Eduard Bloch and lists,

On 11/07/2019 12:21, Steve McIntyre wrote on debian-live:
> On Thu, Jul 11, 2019 at 03:13:48PM +0700, Azure Zanculmarktum wrote:
>> Where can I find the source to build debian live? ... the live-build
config.
>
> The setup we use is in git:
>   https://salsa.debian.org/images-team/live-setup/
...
>
https://salsa.debian.org/images-team/live-setup/blob/master/available/run-30live-wrapper

As part of my documentation effort for live-manual, I would like to
include the procedure for the official Debian CD image generation (which
will effectively replace chapter 16) in
https://live-team.pages.debian.net/live-manual/html/live-manual/procedures.en.html

This mail is rather technical, but I want to document the steps that
I've taken.

From what I've seen in the live-setup repository [5], there are two
strategies for generating Debian CD images.

A) Use lb config (run-30live-build)
B) Use lwr (run-30live-wrapper)

I downloaded 'debian-live-10.3.0-amd64-standard.iso' from [1], which
contains the string 'Official Debian GNU/Linux Live 10.3.0 standard
2020-02-08T12:27'. This string is generated by the run-30live-wrapper
script, so I assume that the official images are generated by method B, lwr.

I've read through run-30live-wrapper, and I think by now I have
correctly extracted the commands for the generation of the Debian Stable
CDs, but I'm unsure how to proceed.

I have the following questions/requests:
* lwr depends on vmdebootstrap, which is currently only in Buster
(stable). According to its author vmdebootstrap is to be replaced by
vmdb2 or something else [2]. Is someone working on this?
* I've seen a huge speed-up when I'm using a ramdisk, but that needs a
merge request to be accepted [3].
  When invoking 'chroot' in live-setup/available/live-customise.sh the
variable TMPDIR must not be set, otherwise it is not possible to
generate temporary files within the chroot.
* To reduce the amount of downloads, I am using a slightly patched
version of apt-cacher-ng, in order to be able to use the installer. See
my bug report [4]
  The live-wrapper script downloads the installer, but apt-cacher-ng
rejects a path ending in a slash. The following command simulates this
download:

  wget -S
"http://localhost:3142/deb.debian.org/debian/dists/buster/main/installer-amd64/current/images/cdrom/;

* The run-30live-wrapper mentions wheezy and jessie, so it is rather old

I would like to add to the live-manual the steps to build an image from
the current stable and testing versions of Debian. Do you have
hints/ideas how I can proceed?

With kind regards,
Roland Clobus

--- Commands
This is the list of commands that I use (all as root):
mkdir -p /tmp/mytmp
mount -t tmpfs tmpfs /tmp/mytmp
export TMPDIR=/tmp/mytmp
lwr -m http://localhost:3142/deb.debian.org/debian \
--apt-mirror http://deb.debian.org/debian \
--customise /home/roland/git/live-setup/available/live-customise.sh \
--architecture amd64 \
-d buster \
--isolinux \
--grub \
--log stderr \
--installer \
-t live-task-standard \
-f "" \
--base_debs "eject pciutils usbutils keyutils keyboard-configuration
console-setup lvm2 mdadm dmsetup cryptsetup dmraid e2fsprogs btrfs-progs
xfsprogs jfsutils grub-efi-amd64 grub-efi-amd64-bin grub-pc
grub-efi-amd64-signed shim-signed" \
--description "Unofficial Debian GNU/Linux Live 10.3 standard" \
--volume_id "d-live 10.3 st amd64" \
--image_output debian-live-10.3.iso

--- Links
[1] https://www.debian.org/CD/live/
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=910201
[3] https://salsa.debian.org/images-team/live-setup/-/merge_requests/1
[4] https://bugs.debian.org/954437
[5] https://salsa.debian.org/images-team/live-setup



signature.asc
Description: OpenPGP digital signature