Re: Patching apt-cacher-ng (Was: Documenting the generation of the live images in Debian)

2020-03-21 Thread Roland Clobus
On 21/03/2020 18:53, Eduard Bloch wrote:

>> In reply to the patch at https://bugs.debian.org/954437

> I cannot accept your patch as-is. Because it's brute-force, a directory
> is not neccessarily a non-volatile file. Can you give me any hints how
> to distinguish between volatile and non-volatile directories? Maybe the
> same as it's done for for other d-i files, by regex, so
> 
>   "|/dists/.*/installer-[^/]+/[0-9][^/]+/images/.*" // d-i stuff 
> with revision
> 
> means frozen contents and
> 
>   "|/dists/.*/installer-[^/]+/[^0-9][^/]+/images/.*" // d-i stuff 
> but not containing a date (year number) in the revision directory (like 
> "current", "beta", ...)
> 
> is volatile? (those regexps are already defined, I just need to change
> your patch to make them applicable to directories)

Agreed. The patch is a brute-force hack, but at least it allowed me to
build my first CD unofficial images.

After I've written my summary to the list, I dug deeper into
live-wrapper. It uses either the released installer (using the apt
proxy), which doesn't contain versioning information in the path, but
will most probably be rather non-volatile, from
https://deb.debian.org/debian/dists/buster/main/installer-amd64/current/images/cdrom/
or the daily build directly (without apt proxy) from
https://d-i.debian.org/daily-images/%s/daily/cdrom/

However, I currently think that my issue is located within the
live-wrapper package. lwr performs a download which isn't used. In
lwr/utils.py the line check_url(base_url) should be removed. Then I
wouldn't need a patched version of apt-cacher-ng at all...

> Also, what's your deadline? I was planing to release a new version of
> ACNG in a couple of days anyway. Do you also need it in backports?

If you still would like to include the patch, I think that the content
listing of a directory should always be considered volatile.

But, as said, I think that the bug should be reassigned to the
live-wrapper package.

With kind regards,
Roland Clobus



signature.asc
Description: OpenPGP digital signature


Re: Documenting the generation of the live images in Debian

2020-03-21 Thread Eduard Bloch
Hallo,
* Roland Clobus [Sat, Mar 21 2020, 05:27:55PM]:

> * To reduce the amount of downloads, I am using a slightly patched
> version of apt-cacher-ng, in order to be able to use the installer. See
> my bug report [4]
>   The live-wrapper script downloads the installer, but apt-cacher-ng
> rejects a path ending in a slash. The following command simulates this
> download:
> 
>   wget -S
> "http://localhost:3142/deb.debian.org/debian/dists/buster/main/installer-amd64/current/images/cdrom/";
> 

I cannot accept your patch as-is. Because it's brute-force, a directory
is not neccessarily a non-volatile file. Can you give me any hints how
to distinguish between volatile and non-volatile directories? Maybe the
same as it's done for for other d-i files, by regex, so

"|/dists/.*/installer-[^/]+/[0-9][^/]+/images/.*" // d-i stuff 
with revision

means frozen contents and

"|/dists/.*/installer-[^/]+/[^0-9][^/]+/images/.*" // d-i stuff 
but not containing a date (year number) in the revision directory (like 
"current", "beta", ...)

is volatile? (those regexps are already defined, I just need to change
your patch to make them applicable to directories)

Also, what's your deadline? I was planing to release a new version of
ACNG in a couple of days anyway. Do you also need it in backports?

Best regards,
Eduard.



Re: Documenting the generation of the live images in Debian

2020-03-21 Thread PICCORO McKAY Lenz
this mail are very important.. but i cannot find a archive link to
distribute it! how can i get a link of this archive mail?
ALSO:
El sáb., 21 de mar. de 2020 a la(s) 12:44, Roland Clobus (rclo...@rclobus.nl)
escribió:

>
> https://salsa.debian.org/images-team/live-setup/blob/master/available/run-30live-wrapper
>
> As part of my documentation effort for live-manual, I would like to
> include the procedure for the official Debian CD image generation (which
> will effectively replace chapter 16) in
>
> https://live-team.pages.debian.net/live-manual/html/live-manual/procedures.en.html
>
> This mail is rather technical, but I want to document the steps that
> I've taken.
>
> From what I've seen in the live-setup repository [5], there are two
> strategies for generating Debian CD images.
>
> A) Use lb config (run-30live-build)
> B) Use lwr (run-30live-wrapper)
>

this only generates lasted today images? how to generates for debian 9 and
debian 8 live? and cd/dvd 1 set?


>
> I downloaded 'debian-live-10.3.0-amd64-standard.iso' from [1], which
> contains the string 'Official Debian GNU/Linux Live 10.3.0 standard
> 2020-02-08T12:27'. This string is generated by the run-30live-wrapper
> script, so I assume that the official images are generated by method B,
> lwr.
>
> I've read through run-30live-wrapper, and I think by now I have
> correctly extracted the commands for the generation of the Debian Stable
> CDs, but I'm unsure how to proceed.
>
> I have the following questions/requests:
> * lwr depends on vmdebootstrap, which is currently only in Buster
> (stable). According to its author vmdebootstrap is to be replaced by
> vmdb2 or something else [2]. Is someone working on this?
> * I've seen a huge speed-up when I'm using a ramdisk, but that needs a
> merge request to be accepted [3].
>   When invoking 'chroot' in live-setup/available/live-customise.sh the
> variable TMPDIR must not be set, otherwise it is not possible to
> generate temporary files within the chroot.
> * To reduce the amount of downloads, I am using a slightly patched
> version of apt-cacher-ng, in order to be able to use the installer. See
> my bug report [4]
>   The live-wrapper script downloads the installer, but apt-cacher-ng
> rejects a path ending in a slash. The following command simulates this
> download:
> 
>   wget -S
> "
> http://localhost:3142/deb.debian.org/debian/dists/buster/main/installer-amd64/current/images/cdrom/
> "
> 
> * The run-30live-wrapper mentions wheezy and jessie, so it is rather old
>
> I would like to add to the live-manual the steps to build an image from
> the current stable and testing versions of Debian. Do you have
> hints/ideas how I can proceed?
>
> With kind regards,
> Roland Clobus
>
> --- Commands
> This is the list of commands that I use (all as root):
> mkdir -p /tmp/mytmp
> mount -t tmpfs tmpfs /tmp/mytmp
> export TMPDIR=/tmp/mytmp
> lwr -m http://localhost:3142/deb.debian.org/debian \
> --apt-mirror http://deb.debian.org/debian \
> --customise /home/roland/git/live-setup/available/live-customise.sh \
> --architecture amd64 \
> -d buster \
> --isolinux \
> --grub \
> --log stderr \
> --installer \
> -t live-task-standard \
> -f "" \
> --base_debs "eject pciutils usbutils keyutils keyboard-configuration
> console-setup lvm2 mdadm dmsetup cryptsetup dmraid e2fsprogs btrfs-progs
> xfsprogs jfsutils grub-efi-amd64 grub-efi-amd64-bin grub-pc
> grub-efi-amd64-signed shim-signed" \
> --description "Unofficial Debian GNU/Linux Live 10.3 standard" \
> --volume_id "d-live 10.3 st amd64" \
> --image_output debian-live-10.3.iso
>
> --- Links
> [1] https://www.debian.org/CD/live/
> [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=910201
> [3] https://salsa.debian.org/images-team/live-setup/-/merge_requests/1
> [4] https://bugs.debian.org/954437
> [5] https://salsa.debian.org/images-team/live-setup
>
>


Re: Historical images

2020-03-21 Thread Roland Clobus
On 21/03/2020 18:01, PICCORO McKAY Lenz wrote:
> this mail are very important.. but i cannot find a archive link to
> distribute it! how can i get a link of this archive mail?

https://lists.debian.org/debian-live/2020/03/msg00225.html
https://lists.debian.org/debian-cd/2020/03/msg5.html

> ALSO:
>> From what I've seen in the live-setup repository [5], there are two
>> strategies for generating Debian CD images.
>>
>> A) Use lb config (run-30live-build)
>> B) Use lwr (run-30live-wrapper)
>>
> this only generates lasted today images? how to generates for debian 9
> and debian 8 live? and cd/dvd 1 set?

You are perhaps looking for the archive?
https://cdimage.debian.org/cdimage/archive/

With kind regards,
Roland Clobus




signature.asc
Description: OpenPGP digital signature


Documenting the generation of the live images in Debian

2020-03-21 Thread Roland Clobus
Hello Steve McIntyre, Eduard Bloch and lists,

On 11/07/2019 12:21, Steve McIntyre wrote on debian-live:
> On Thu, Jul 11, 2019 at 03:13:48PM +0700, Azure Zanculmarktum wrote:
>> Where can I find the source to build debian live? ... the live-build
config.
>
> The setup we use is in git:
>   https://salsa.debian.org/images-team/live-setup/
...
>
https://salsa.debian.org/images-team/live-setup/blob/master/available/run-30live-wrapper

As part of my documentation effort for live-manual, I would like to
include the procedure for the official Debian CD image generation (which
will effectively replace chapter 16) in
https://live-team.pages.debian.net/live-manual/html/live-manual/procedures.en.html

This mail is rather technical, but I want to document the steps that
I've taken.

From what I've seen in the live-setup repository [5], there are two
strategies for generating Debian CD images.

A) Use lb config (run-30live-build)
B) Use lwr (run-30live-wrapper)

I downloaded 'debian-live-10.3.0-amd64-standard.iso' from [1], which
contains the string 'Official Debian GNU/Linux Live 10.3.0 standard
2020-02-08T12:27'. This string is generated by the run-30live-wrapper
script, so I assume that the official images are generated by method B, lwr.

I've read through run-30live-wrapper, and I think by now I have
correctly extracted the commands for the generation of the Debian Stable
CDs, but I'm unsure how to proceed.

I have the following questions/requests:
* lwr depends on vmdebootstrap, which is currently only in Buster
(stable). According to its author vmdebootstrap is to be replaced by
vmdb2 or something else [2]. Is someone working on this?
* I've seen a huge speed-up when I'm using a ramdisk, but that needs a
merge request to be accepted [3].
  When invoking 'chroot' in live-setup/available/live-customise.sh the
variable TMPDIR must not be set, otherwise it is not possible to
generate temporary files within the chroot.
* To reduce the amount of downloads, I am using a slightly patched
version of apt-cacher-ng, in order to be able to use the installer. See
my bug report [4]
  The live-wrapper script downloads the installer, but apt-cacher-ng
rejects a path ending in a slash. The following command simulates this
download:

  wget -S
"http://localhost:3142/deb.debian.org/debian/dists/buster/main/installer-amd64/current/images/cdrom/";

* The run-30live-wrapper mentions wheezy and jessie, so it is rather old

I would like to add to the live-manual the steps to build an image from
the current stable and testing versions of Debian. Do you have
hints/ideas how I can proceed?

With kind regards,
Roland Clobus

--- Commands
This is the list of commands that I use (all as root):
mkdir -p /tmp/mytmp
mount -t tmpfs tmpfs /tmp/mytmp
export TMPDIR=/tmp/mytmp
lwr -m http://localhost:3142/deb.debian.org/debian \
--apt-mirror http://deb.debian.org/debian \
--customise /home/roland/git/live-setup/available/live-customise.sh \
--architecture amd64 \
-d buster \
--isolinux \
--grub \
--log stderr \
--installer \
-t live-task-standard \
-f "" \
--base_debs "eject pciutils usbutils keyutils keyboard-configuration
console-setup lvm2 mdadm dmsetup cryptsetup dmraid e2fsprogs btrfs-progs
xfsprogs jfsutils grub-efi-amd64 grub-efi-amd64-bin grub-pc
grub-efi-amd64-signed shim-signed" \
--description "Unofficial Debian GNU/Linux Live 10.3 standard" \
--volume_id "d-live 10.3 st amd64" \
--image_output debian-live-10.3.iso

--- Links
[1] https://www.debian.org/CD/live/
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=910201
[3] https://salsa.debian.org/images-team/live-setup/-/merge_requests/1
[4] https://bugs.debian.org/954437
[5] https://salsa.debian.org/images-team/live-setup



signature.asc
Description: OpenPGP digital signature