Re: Bug#861263: debian-installer: zfs support

[Philip Hands]

On Sat, 06 May 2017, Sam Kuper  wrote:
...
> Given that a machine intended to run ZFS is likely to be provisioned
> with >2GB of RAM ...

debian-installer is effectively an embedded OS for the purpose of
installing Debian -- trying to squeeze the ability to build ZFS into
that is a mistake IMO.

Given that you are assuming systems with a decent chunk of RAM, I'd
suggest that you instead boot from live media, thus getting a full
Debian system immediately, running in RAM, without the need to be
restrained by the limitations of debian-installer.

Then you ought to be able to simply install the zfs tools and modules
into the in-RAM system (assuming one can load them without reboot),
format disks as you require, and then install using e.g. debootstrap,
without needing to worry about what debian-installer can or cannot do.

I would guess that debian-live should be up to the task, but if doesn't
work for some reason, you could also look at grml.

Cheers, Phil.
-- 
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: Bug#861263: debian-installer: zfs support

[Nicholas D Steeves]

On Wed, May 10, 2017 at 05:24:48PM +0200, Wouter Verhelst wrote:
> On Sat, May 06, 2017 at 03:35:52PM +0100, Sam Kuper wrote:
> > On 06/05/2017, Ian Campbell  wrote:
> > > It would in theory be possible to arrange build and install modules
> > > during installation using the in-progress target installation (where
> > > the normal toolchain packages could be installed) such that they are
> > > available during the latter parts of the install procedure -- that is
> > > clearly of limited use for any modules which are required for the
> > > filesystems which you want to install to (certainly the root fs and
> > > perhaps any separate /usr or /var partition(s)).
> > 
> > Given that a machine intended to run ZFS is likely to be provisioned
> > with >2GB of RAM (much more than the Debian Installer would normally
> > require), might a viable workaround be for the Debian Installer to
> > take the following steps?
> > 
> > - Ensure that some minimum amount of RAM is available, and refuse to
> > proceed with a root-on-ZFS installation if not;
> > 
> > - install the toolchain, ZFS source, and any other packages from
> > "main" or "contrib" necessary to support these, to a RAM disk;
> > 
> > - compile and load ZFS functionality;
> > 

Add partitioning here, and add an extra partition used for holding the
debootstrap system. [1]

> > - format the target HDD/SSD using ZFS;
> > 
> > - copy the toolchain binaries and ZFS source packages, etc, to the
> > relevant places on the target HDD/SSD (/usr/bin ,
> > /var/cache/apt/archives/ , etc);
> >
> > - resume the installation as usual.

Before installing the bootloader, check if it's a ZFS installation,
and if so, then replicate the debootstrapped system from the extra
partition to the ZFS dataset marked as rootfs, using something that
supports --numeric-owner. [2]

Then delete the extra partition, expand the ZFS one to the end of the
device. eg:

[1]
sda1: 256MB /boot
sda2: 930GB zfs-vdev

And then the installer substracts 4GB from sda2, creates sda3 for
debootstrap, and takes care of the removal transparently.  Using this
method, sda2 only gets expanded to the size that is requested, instead
of the whole disk--allows underprovisioning.

> I don't see why not. We already need the ability to run debootstrap;
> this would just change that to a need to do so twice.
> 
> It wouldn't be terribly efficient for a netinstall, since it requires
> that you download the base system twice; but it could imagine a special
> "non-free netinstall" image which would contain zfs-dkms and its
> dependencies on top of the base system, so that you don't need to
> download things more than once.

I'm not sure if [2] is the right place in the installation sequence
for this, but it seems like something like this method could make
efficient use of bandwidth for a netinstall :-)

Cheers,
Nicholas



signature.asc
Description: PGP signature

Bug#861263: debian-installer: zfs support

[Timo Haas]

On Wed, 10 May 2017 17:24:48 +0200 Wouter Verhelst 
wrote:
> On Sat, May 06, 2017 at 03:35:52PM +0100, Sam Kuper wrote:
> > On 06/05/2017, Ian Campbell  wrote:
> > > It would in theory be possible to arrange build and install modules
> > > during installation using the in-progress target installation (where
> > > the normal toolchain packages could be installed) such that they are
> > > available during the latter parts of the install procedure -- that is
> > > clearly of limited use for any modules which are required for the
> > > filesystems which you want to install to (certainly the root fs and
> > > perhaps any separate /usr or /var partition(s)).
> >
> > Given that a machine intended to run ZFS is likely to be provisioned
> > with >2GB of RAM (much more than the Debian Installer would normally
> > require), might a viable workaround be for the Debian Installer to
> > take the following steps?
> >
> > - Ensure that some minimum amount of RAM is available, and refuse to
> > proceed with a root-on-ZFS installation if not;
> >
> > - install the toolchain, ZFS source, and any other packages from
> > "main" or "contrib" necessary to support these, to a RAM disk;
> >
> > - compile and load ZFS functionality;
> >
> > - format the target HDD/SSD using ZFS;
> >
> > - copy the toolchain binaries and ZFS source packages, etc, to the
> > relevant places on the target HDD/SSD (/usr/bin ,
> > /var/cache/apt/archives/ , etc);
> >
> > - resume the installation as usual.
>
> I don't see why not. We already need the ability to run debootstrap;
> this would just change that to a need to do so twice.
>
> It wouldn't be terribly efficient for a netinstall, since it requires
> that you download the base system twice; but it could imagine a special
> "non-free netinstall" image which would contain zfs-dkms and its
> dependencies on top of the base system, so that you don't need to
> download things more than once.
>
> I don't know whether anyone on this list would be actually interested in
> adding the necessary support to d-i, but hey, if you are, I'm sure
> patches are welcome. As a side note, I'll add that there is already a
> partman-zfs, due to Debian-kFreeBSD; this should mean that you'd only
> need to extend that to also do the compilation at first.
>
> --
> < ron> I mean, the main *practical* problem with C++, is there's like
a dozen
> people in the world who think they really understand all of its rules,
> and pretty much all of them are just lying to themselves too.
> -- #debian-devel, OFTC, 2016-02-12
>
>

I made a simple  script which downloads, compiles and installes the zfs
kernel module and zhe zfs,spool commands.
Tested with the stable netinstall iso, after network is configured and
deboostrap is downloaded.

It makes a simple chroot environment, apt-get install zfsutils-linux and
then copies the binaries and the necessary libraries over.


zfs.sh
Description: application/shellscript

Re: Bug#861263: debian-installer: zfs support

[Wouter Verhelst]

On Sat, May 06, 2017 at 03:35:52PM +0100, Sam Kuper wrote:
> On 06/05/2017, Ian Campbell  wrote:
> > It would in theory be possible to arrange build and install modules
> > during installation using the in-progress target installation (where
> > the normal toolchain packages could be installed) such that they are
> > available during the latter parts of the install procedure -- that is
> > clearly of limited use for any modules which are required for the
> > filesystems which you want to install to (certainly the root fs and
> > perhaps any separate /usr or /var partition(s)).
> 
> Given that a machine intended to run ZFS is likely to be provisioned
> with >2GB of RAM (much more than the Debian Installer would normally
> require), might a viable workaround be for the Debian Installer to
> take the following steps?
> 
> - Ensure that some minimum amount of RAM is available, and refuse to
> proceed with a root-on-ZFS installation if not;
> 
> - install the toolchain, ZFS source, and any other packages from
> "main" or "contrib" necessary to support these, to a RAM disk;
> 
> - compile and load ZFS functionality;
> 
> - format the target HDD/SSD using ZFS;
> 
> - copy the toolchain binaries and ZFS source packages, etc, to the
> relevant places on the target HDD/SSD (/usr/bin ,
> /var/cache/apt/archives/ , etc);
>
> - resume the installation as usual.

I don't see why not. We already need the ability to run debootstrap;
this would just change that to a need to do so twice.

It wouldn't be terribly efficient for a netinstall, since it requires
that you download the base system twice; but it could imagine a special
"non-free netinstall" image which would contain zfs-dkms and its
dependencies on top of the base system, so that you don't need to
download things more than once.

I don't know whether anyone on this list would be actually interested in
adding the necessary support to d-i, but hey, if you are, I'm sure
patches are welcome. As a side note, I'll add that there is already a
partman-zfs, due to Debian-kFreeBSD; this should mean that you'd only
need to extend that to also do the compilation at first.

-- 
< ron> I mean, the main *practical* problem with C++, is there's like a dozen
   people in the world who think they really understand all of its rules,
   and pretty much all of them are just lying to themselves too.
 -- #debian-devel, OFTC, 2016-02-12

Re: Bug#861263: debian-installer: zfs support

[Steve McIntyre]

On Sun, May 07, 2017 at 06:44:11PM +0100, Sam Kuper wrote:
>On 05/05/2017, Sam Kuper  wrote:
>> Bill McGrath's request matches my suggestion above[3],
>> and Neil McGovern's reply suggests it is already on Debian's roadmap.
>> Why, then, is this bug (Bug#861263) marked as wontfix?
>>
>> [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=861263#17
>
>In the light of this, and of the paths forward described above, please
>can I reiterate my earlier request for the "wontfix" label to be
>removed? Even if it takes years for ZFS support to be added to the
>Debian Installer, it would be worthwhile.
>
>Failing that, it would be good to have an explanation of why ZFS will
>*never* be supported in the Debian Installer.

While there might be a way to do this work technically, it's probably
fair to say that none of the existing maintainers have any plans to
work on it. It would need massive effort, and there's no sign of those
maintainers showing any interest in doing it. That sounds like a
reasonable description of "wontfix" to me.

If somebody was actually working on patches, that might change the
situation.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
Is there anybody out there?

Re: Bug#861263: debian-installer: zfs support

[Sam Kuper]

On 05/05/2017, Sam Kuper  wrote:
> Bill McGrath's request matches my suggestion above[3],
> and Neil McGovern's reply suggests it is already on Debian's roadmap.
> Why, then, is this bug (Bug#861263) marked as wontfix?
>
> [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=861263#17

In the light of this, and of the paths forward described above, please
can I reiterate my earlier request for the "wontfix" label to be
removed? Even if it takes years for ZFS support to be added to the
Debian Installer, it would be worthwhile.

Failing that, it would be good to have an explanation of why ZFS will
*never* be supported in the Debian Installer.

Thanks :)

Re: Bug#861263: debian-installer: zfs support

[Sam Kuper]

On 06/05/2017, Ian Campbell  wrote:
> On Fri, 2017-05-05 at 22:52 +0100, Sam Kuper wrote:
>> If so, is there
>> any reason in principle why that installer could not in future be
>> distributed with the capability to (download and) compile and run ZFS,
>> and to provide the user with the option to install Debian onto a ZFS
>> root partition?
>
> There is no build toolchain (gcc et al) available for the installer
> environment itself, nor are kernel headers provided there.

I was not aware of that. Thank you for making this crucial point :)

> It would in theory be possible to arrange build and install modules
> during installation using the in-progress target installation (where
> the normal toolchain packages could be installed) such that they are
> available during the latter parts of the install procedure -- that is
> clearly of limited use for any modules which are required for the
> filesystems which you want to install to (certainly the root fs and
> perhaps any separate /usr or /var partition(s)).

Given that a machine intended to run ZFS is likely to be provisioned
with >2GB of RAM (much more than the Debian Installer would normally
require), might a viable workaround be for the Debian Installer to
take the following steps?

- Ensure that some minimum amount of RAM is available, and refuse to
proceed with a root-on-ZFS installation if not;

- install the toolchain, ZFS source, and any other packages from
"main" or "contrib" necessary to support these, to a RAM disk;

- compile and load ZFS functionality;

- format the target HDD/SSD using ZFS;

- copy the toolchain binaries and ZFS source packages, etc, to the
relevant places on the target HDD/SSD (/usr/bin ,
/var/cache/apt/archives/ , etc);

- resume the installation as usual.


> For most other
> partitions it would probably be just as easy to format them post-
> install anyway.

True. So to proceed towards solving this feature request, I guess the
initial focus should be on how to install the root ZFS
partition/dataset. Figuring out how to handle other
partitions/datasets can wait until that has been solved.

Bug#861263: debian-installer: zfs support

[Ian Campbell]

On Fri, 2017-05-05 at 22:52 +0100, Sam Kuper wrote:
> If so, is there
> any reason in principle why that installer could not in future be
> distributed with the capability to (download and) compile and run ZFS,
> and to provide the user with the option to install Debian onto a ZFS
> root partition?

There is no build toolchain (gcc et al) available for the installer
environment itself, nor are kernel headers provided there.

It would in theory be possible to arrange build and install modules
during installation using the in-progress target installation (where
the normal toolchain packages could be installed) such that they are
available during the latter parts of the install procedure -- that is
clearly of limited use for any modules which are required for the
filesystems which you want to install to (certainly the root fs and
perhaps any separate /usr or /var partition(s)). For most other
partitions it would probably be just as easy to format them post-
install anyway.

Ian.

Re: Bug#861263: debian-installer: zfs support

[Sam Kuper]

On 06/05/2017, Nicholas D Steeves  wrote:
> I would recommend the second of the following options:
>
> 1. Install using the non-free media with "Advanced options" -> "Expert
> install"
> 2. Install using the non-free media, then cleanup [...]
>
> It's faster than an "Advanced
> options" -> "Expert install", where I believe it is also possible to
> install a system which pulls uniquely from main and contrib.

Thank you for reminding me of the existence of the "Expert install"
option in the Debian Installer! :)

My understanding of this feature request (#861263) is that it would be
satisfied when a Debian Installer exists in which: the user can
install Debian by simply click through the ncurses interface pretty
much as usual, but in addition to the current guided[1] and manual[2]
partitioning options, the user would have the option to select guided
or manual partitioning with ZFS. Ideally, that would also include the
option for encrypted ZFS using either a LUKS container or native ZFS
encryption.[3]

So, between your two options, I think the "Expert install" would
probably be a better fit for resolving this feature request. However,
although "Expert install" would be appropriate, it might not be
necessary to use an unofficial installer. See below.


On 05/05/2017, Ben Hutchings  wrote:
> On Fri, 2017-05-05 at 14:26 +0100, Sam Kuper wrote:
>> If the Debian Installer were instead to ship with, or to download at
>> runtime, the ZFS on Linux source code, would that be acceptable from a
>> licensing standpoint?
>
> I imagine this would be acceptable (though not in the default
> installer, which only uses and installs packages from main).
>
> [...] there is already an (officially unofficial) installer that
> includes non-free firmware.

I have just run an *official* Jessie NetInst CD, using "Expert
install" mode. Fairly late in the process, there is a step titled
"Configure the package manager".[4] This step asks the user if they
want software from "non-free" and/or "contrib" to be available to the
system.

So, it seems that there is no need in principle to use an unofficial
installer just to be presented with the option to enable "contrib".


One piece of work that would need to be done to the Debian Installer
to enable it to download, compile and run ZFS before partitioning the
HDD/SSD, is for the "Configure the package manager" step to be moved
to an earlier point in the installation process. Let me explain. In my
Jessie NetInst CD, the "Configure the package manager" step occurs
*after* the Debian Installer has partitioned the drive and installed
Debian to it: too late to make a difference, from the perspective of
enabling ZFS root!

I would suggest that the "Configure the package manager" step should
be placed immediately *before* the "Detect disks" step. This means the
"Configure the package manager" step would have to be modified. Rather
than straight away writing to /etc/apt/sources.list and handing over
to the following step (as it currently seems to do), it would instead:

- record the user's selections to memory;

- enable guided and manual ZFS options to become available in the
"Partition disks" step (but only if the user chose to enable
"contrib"); and

- write the user's selections to /etc/apt/sources.list *after* the
target drive has been formatted and populated.

Additionally, of course, the Debian Installer would need to have code
incorporated to perform the download-compile-run steps for ZFS.

I can see that these are not trivial changes, but I also can see no
reason in principle why they should not be made to the Debian
Installer at some point during the Stretch lifecycle. Even if they end
up taking many months to bring up to release quality, they would be
very valuable additions to the Debian Installer.

I would be very grateful if the "wontfix" label could be removed from
this feature request.


Thanks again to both of you; and Ben, I really did mean no offence to
you by mentioning Moglen. I'm sorry if that came across as
supercilious. I really was just trying to explain the basis of my
understanding.


[1] https://www.debian.org/releases/stable/amd64/ch06s03.html.en#partman-auto
[2] https://www.debian.org/releases/stable/amd64/ch06s03.html.en#partman-manual
[3] AFAIK, native ZFS encryption with Linux is not stable enough to
make sense for a stable distribution. Until it is, ZFS-on-LUKS seems
to be the best substitute.
[4] https://www.debian.org/releases/stable/amd64/ch06s03.html.en#apt-setup

Re: Bug#861263: debian-installer: zfs support

[Nicholas D Steeves]

On 5 May 2017 at 15:27, Sam Kuper  wrote:
> On 05/05/2017, Ben Hutchings  wrote:
>>On Fri, 2017-05-05 at 19:50 +0100, Sam Kuper wrote:
>
>>> 2. Add ZFS to a Debian Installer that is not the *default* Debian
>>> Installer. Does Debian distribute such an installer, to which the
>>> facility to compile and run ZFS could be added?
>>
>> Yes, there is already an (officially unofficial) installer that
>> includes non-free firmware.
>
> Thanks for the information. Can the non-free aspect of that installer
> be disabled by the user during installation? If not, then it would be
> no use to anyone I know who would be interested in running ZFS under
> Debian. That is because a key reason to use Debian in preference to
> other distros is that Debian's blob-free kernel and DFSG-compliant
> main and contrib repositories make it easy to avoid installing
> non-free software. If a person doesn't mind the risk of installing
> non-free firmware then they may as well just skip Debian and use
> Ubuntu or FreeBSD instead, which ship with ZFS in the installer by
> default.
>

I would recommend the second of the following options:

1. Install using the non-free media with "Advanced options" -> "Expert install"
2. Install using the non-free media, then cleanup

#!/bin/sh
apt-get install aptitude
sed -i 's/ non-free//' /etc/apt/sources.list
apt-get update
aptitude search ?obsolete -F '%p' --disable-columns \
| apt-get purge

...and the non-free packages should be gone.  And if you don't want
aptitude you can purge that too.  It's faster than an "Advanced
options" -> "Expert install", where I believe it is also possible to
install a system which pulls uniquely from main and contrib.

There are a more reasons to use Debian than just default package
selection... eg: updates policy, minimal sysadmin headaches, smooth
upgrades even from major version to major version, very high quality
packaging standards, etc.  These are pragmatic reasons to prefer
Debian.  In my opinion embracing CDDL constitutes ideological
compromise, because it forbids "mixing" with with GPL--the most
socioally conscious and not neoliberal license.  And if Debian isn't
'pure' enough, there are always these:
https://www.gnu.org/distros/free-distros.html

Cheers,
Nicholas

Re: Bug#861263: debian-installer: zfs support

[Sam Kuper]

On 05/05/2017, Ben Hutchings  wrote:
> I shall not share my opinion of Eben Moglen, because I don't want to
> get sued.  But I would say that "Eben Moglen says X" is not going to
> convince me of X.
>
> And, the FTP team has made its decision.
>
> I'm not going to argue this further.

Apologies if I offended you. I wasn't trying to argue the merits of
the FTP team's decision (which I respect and agree with!), or of your
opinion about distributing ZFSonLinux binaries. I was just trying to
explain the background to my question.

I would be very grateful if you (or anybody else with the relevant
knowledge) would answer the question I posted:


Does Debian distribute a Debian Installer that will (either by
default, or at the user's request) install source or binary packages
from no repositories other than "main" and "contrib"? If so, is there
any reason in principle why that installer could not in future be
distributed with the capability to (download and) compile and run ZFS,
and to provide the user with the option to install Debian onto a ZFS
root partition?

Re: Bug#861263: debian-installer: zfs support

[Ben Hutchings]

On Fri, 2017-05-05 at 22:52 +0100, Sam Kuper wrote:
> On 05/05/2017, Ben Hutchings  wrote:
> > On Fri, 2017-05-05 at 21:40 +0100, Sam Kuper wrote:
> > > I am not sure why you say that ZFSonLinux binaries are non-free.
> > > Please could you explain?
> > 
> > I was referring specifically to the binary kernel modules, which
> > have a
> > mixture of CDDL and GPLv2 code.  These licences are incompatible so
> > the
> > binaries cannot be distribured, thus are non-free.
> 
> I see. Thanks for explaining your view.
> 
> Eben Moglen's take is more nuanced:
[...]

I shall not share my opinion of Eben Moglen, because I don't want to
get sued.  But I would say that "Eben Moglen says X" is not going to
convince me of X.

And, the FTP team has made its decision.

I'm not going to argue this further.

Ben.

-- 
Ben Hutchings
The program is absolutely right; therefore, the computer must be wrong.



signature.asc
Description: This is a digitally signed message part

Re: Bug#861263: debian-installer: zfs support

[Sam Kuper]

P.S. Ben, thank you again for taking the time on this. It is providing
a great deal of clarity to me, and I hope that other people who also
desire a ZFS-capable Debian Installer will also find it helpful.

Re: Bug#861263: debian-installer: zfs support

[Sam Kuper]

On 05/05/2017, Ben Hutchings  wrote:
> On Fri, 2017-05-05 at 21:40 +0100, Sam Kuper wrote:
>> I am not sure why you say that ZFSonLinux binaries are non-free.
>> Please could you explain?
>
> I was referring specifically to the binary kernel modules, which have a
> mixture of CDDL and GPLv2 code.  These licences are incompatible so the
> binaries cannot be distribured, thus are non-free.

I see. Thanks for explaining your view.

Eben Moglen's take is more nuanced: "If [Linux kernel copyright
holders] prefer the literal meaning to the equity of the license, the
copyright holders can, at their discretion, object to the distribution
of such combinations. ... If they do not[, then] the equity of the
license will eventually come to be seen as the measuring rod [i.e.
distributions will be legally permissible]." Moglen also notes that
there are, "good reasons for [Linux kernel developers] not to object
[to distribution]".[1]

In other words, right now and for as long as the Linux developers do
not object en masse to the distribution of such binaries, there is
nothing in law to prevent a person from distributing such binaries.
But there *is* a risk that at some point in the future the Linux
developers will raise such an objection. If they do so successfully,
then the binaries would indeed become non-free.

I understand that Debian is erring on the side of caution on this
matter, whereas Canonical is being less cautious and is evidently
hoping the binaries remain legal to distribute.

> That is why
> ZFSonLinux module source is in the 'contrib' section, not 'main'.

Yes, I understand that now from Neil's comment.


Let me rephrase my earlier question again:

Does Debian distribute a Debian Installer that will (either by
default, or at the user's request) install source or binary packages
from no repositories other than "main" and "contrib"? If so, is there
any reason in principle why that installer could not in future be
distributed with the capability to (download and) compile and run ZFS,
and to provide the user with the option to install Debian onto a ZFS
root partition?



[1] https://softwarefreedom.org/resources/2016/linux-kernel-cddl.html

Re: Bug#861263: debian-installer: zfs support

[Ben Hutchings]

On Fri, 2017-05-05 at 21:40 +0100, Sam Kuper wrote:
[...]
> So, I am not sure why you say that ZFSonLinux binaries are non-free.
> Please could you explain?

I was referring specifically to the binary kernel modules, which have a
mixture of CDDL and GPLv2 code.  These licences are incompatible so the
binaries cannot be distribured, thus are non-free.  That is why
ZFSonLinux module source is in the 'contrib' section, not 'main'.

I assume, though I haven't checked, that the userland packages are
CDDL-only (and dynamically linked to LGPL libraries) which is fine.

Ben.

-- 
Ben Hutchings
The program is absolutely right; therefore, the computer must be wrong.



signature.asc
Description: This is a digitally signed message part

Re: Bug#861263: debian-installer: zfs support

[Sam Kuper]

On 05/05/2017, Sam Kuper  wrote:
> On 05/05/2017, Ben Hutchings  wrote:
>> The legal status of ZFSonLinux was discussed by the FTP team and DPL
>> over a long period, with input from legal counsel, resulting in a
>> decision to put it in the 'contrib' section.  That decision is unlikely
>> to be revisited soon.
>
> Thanks. I have searched for such a discussion but have not yet found
> it. Do you have a link to the discussion? Did its conclusions
> definitely cover source distribution, or only binaries?

I still have not found the discussion, but I have found a helpful
summary by Neil McGovern. In a comment thread there, Neil summarises
the reason to put the ZFS DKMS into "main", even though it is
distributed as source:[1]

Martin (February 28, 2017 at 6:14 pm):
> I understand the decision to distribute ZFS as source
> only, but could you elaborate on why the package is
> going into contrib rather than main?

Neil McGovern (February 28, 2017 at 6:26 pm):
> Sure – it’s about the promise that Debian makes to
> the end user. Basically, by it being in main you’re
> legally able to redistribute the end product (along with
> source). With a CDDL module and a GPL2+ kernel,
> that becomes – at best – unclear.


I would still like to see the original discussion, but for the time
being, this comment of Neil's adequately answer my question about
Debian's rationale re: source vs binary and "contrib" vs "main".

However, there is another comment thread on Neil's summary that seems
very pertinent to this bug (Bug#861263):[2]

Bill McGrath (March 12, 2016 at 1:27 am):
> [...] If source is the only option, might I
> suggest building a script into the installer to do the
> downloading and compiling so that installation will still
> be a breeze.

Neil McGovern (March 14, 2016 at 9:11 am):
> This is what’ll happen already, we’re using DKMS

In other words, Bill McGrath's request matches my suggestion above[3],
and Neil McGovern's reply suggests it is already on Debian's roadmap.
Why, then, is this bug (Bug#861263) marked as wontfix? Did something
change Neil's mind after that comment was posted? Or was Neil wrong at
the time to suggest that the Debian Installer will include a script to
download and compile ZFS?



[1] https://blog.halon.org.uk/2016/01/on-zfs-in-debian/#comment-13678
[2] https://blog.halon.org.uk/2016/01/on-zfs-in-debian/#comment-9055
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=861263#17

Re: Bug#861263: debian-installer: zfs support

[Sam Kuper]

On 05/05/2017, Ben Hutchings  wrote:
> On Fri, 2017-05-05 at 20:27 +0100, Sam Kuper wrote:
>> On 05/05/2017, Ben Hutchings  wrote:
>> > On Fri, 2017-05-05 at 19:50 +0100, Sam Kuper wrote:
>> > > 1. Move ZFS *source* into "main". Would this be possible without
>> > > compromising Debian's "obviously prudent" arrangement?[1] Should I CC
>> > > debian-legal?
>> >
>> > This will not happen.
>>
>> Forgive my ignorance, but why will it not happen?
>
> The legal status of ZFSonLinux was discussed by the FTP team and DPL
> over a long period, with input from legal counsel, resulting in a
> decision to put it in the 'contrib' section.  That decision is unlikely
> to be revisited soon.

Thanks. I have searched for such a discussion but have not yet found
it. Do you have a link to the discussion? Did its conclusions
definitely cover source distribution, or only binaries?



> [...]
>> If it can't be disabled, then let me rephrase my earlier question.
>> Does Debian distribute a Debian Installer that will install only
>> DFSG-compliant software, to which the facility to compile and run ZFS
>> could in principle be added? (After all, everything in "contrib" is
>> DFSG-compliant,[1] including the ZFS-related packages.) If so, please
>> could you provide me with a link to it?
> [...]
>
> ZFSonLinux binaries are non-free.

Your remark is surprising to me. It seems to contradict the
information I have encountered. For example, see:

https://packages.debian.org/search?keywords=zfs=names=all=all

Some of the packages listed there are in main, and are therefore DFSG
compliant.[0] The remainder are marked "[contrib]", and so must also
be DFSG-compliant,[0] albeit not necessarily GPLv2-compatible. None of
them are marked "[non-free]".

Additionally:

- The FSF regards the CDDL as a free software license.[1]

- Debian regards the MPL (to which the CDDL is very similar[2]) as a
DFSG-compliant license.[3]

So, I am not sure why you say that ZFSonLinux binaries are non-free.
Please could you explain?



Thank you again for answering my questions and for helping to provide
clarity about the prospect of Debian distributing an Installer with
some kind of ZFS on root option.

[0] https://www.debian.org/doc/debian-policy/ch-archive.html
[1] https://www.gnu.org/licenses/license-list.html#CDDL
[2] 
http://www.crynwr.com/cgi-bin/ezmlm-cgi?3:mss:9125:200412:dmcacncfamieofeochbn
[3] https://wiki.debian.org/DFSGLicenses#Mozilla_Public_License_.28MPL.29

Re: Bug#861263: debian-installer: zfs support

[Ben Hutchings]

On Fri, 2017-05-05 at 20:27 +0100, Sam Kuper wrote:
> On 05/05/2017, Ben Hutchings  wrote:
> > On Fri, 2017-05-05 at 19:50 +0100, Sam Kuper wrote:
> > > 1. Move ZFS *source* into "main". Would this be possible without
> > > compromising Debian's "obviously prudent" arrangement?[1] Should I CC
> > > debian-legal?
> > 
> > This will not happen.
> 
> Forgive my ignorance, but why will it not happen?

The legal status of ZFSonLinux was discussed by the FTP team and DPL
over a long period, with input from legal counsel, resulting in a
decision to put it in the 'contrib' section.  That decision is unlikely
to be revisited soon.

[...]
> If it can't be disabled, then let me rephrase my earlier question.
> Does Debian distribute a Debian Installer that will install only
> DFSG-compliant software, to which the facility to compile and run ZFS
> could in principle be added? (After all, everything in "contrib" is
> DFSG-compliant,[1] including the ZFS-related packages.) If so, please
> could you provide me with a link to it?
[...]

ZFSonLinux binaries are non-free.

Ben.

-- 
Ben Hutchings
The program is absolutely right; therefore, the computer must be wrong.



signature.asc
Description: This is a digitally signed message part

Re: Bug#861263: debian-installer: zfs support

[Sam Kuper]

On 05/05/2017, Ben Hutchings  wrote:
>On Fri, 2017-05-05 at 19:50 +0100, Sam Kuper wrote:
>> 1. Move ZFS *source* into "main". Would this be possible without
>> compromising Debian's "obviously prudent" arrangement?[1] Should I CC
>> debian-legal?
>
> This will not happen.

Forgive my ignorance, but why will it not happen?

>> 2. Add ZFS to a Debian Installer that is not the *default* Debian
>> Installer. Does Debian distribute such an installer, to which the
>> facility to compile and run ZFS could be added?
>
> Yes, there is already an (officially unofficial) installer that
> includes non-free firmware.

Thanks for the information. Can the non-free aspect of that installer
be disabled by the user during installation? If not, then it would be
no use to anyone I know who would be interested in running ZFS under
Debian. That is because a key reason to use Debian in preference to
other distros is that Debian's blob-free kernel and DFSG-compliant
main and contrib repositories make it easy to avoid installing
non-free software. If a person doesn't mind the risk of installing
non-free firmware then they may as well just skip Debian and use
Ubuntu or FreeBSD instead, which ship with ZFS in the installer by
default.

If it can't be disabled, then let me rephrase my earlier question.
Does Debian distribute a Debian Installer that will install only
DFSG-compliant software, to which the facility to compile and run ZFS
could in principle be added? (After all, everything in "contrib" is
DFSG-compliant,[1] including the ZFS-related packages.) If so, please
could you provide me with a link to it?

Thanks again :)

Please CC me, as I am still not subscribed to the mailing list.

[1] https://www.debian.org/doc/debian-policy/ch-archive.html

Re: Bug#861263: debian-installer: zfs support

[Ben Hutchings]

On Fri, 2017-05-05 at 19:50 +0100, Sam Kuper wrote:
> > On 05/05/2017, Ben Hutchings  wrote:
> > On Fri, 2017-05-05 at 14:26 +0100, Sam Kuper wrote:
> > > On Wed, 2017-04-26 at 19:51:23 +0100, Ben Hutchings wrote:
> > > > On Wed, 2017-04-26 at 18:20 +0200, Timo Haas wrote:
> > > > > do you plan to support zfs as root filesystem in the installer?
> > > > 
> > > > ZFS binaries are not distributable due to the licence conflict, so this
> > > > is unlikely to happen.
> > > 
> > > If the Debian Installer were instead to ship with, or to download at
> > > runtime, the ZFS on Linux source code, would that be acceptable from a
> > > licensing standpoint?
> > 
> > I imagine this would be acceptable (though not in the default
> > installer, which only uses and installs packages from main).
> 
> Good point. Potential avenues:
> 
> 1. Move ZFS *source* into "main". Would this be possible without
> compromising Debian's "obviously prudent" arrangement?[1] Should I CC
> debian-legal?

This will not happen.

> 2. Add ZFS to a Debian Installer that is not the *default* Debian
> Installer. Does Debian distribute such an installer, to which the
> facility to compile and run ZFS could be added?

Yes, there is already an (officially unofficial) installer that
includes non-free firmware.

Ben.

> Thanks :)
> 
> (Please CC me, as I am still not subscribed to the mailing list.)
> 
> [1] https://softwarefreedom.org/resources/2016/linux-kernel-cddl.html
-- 
Ben Hutchings
The program is absolutely right; therefore, the computer must be wrong.



signature.asc
Description: This is a digitally signed message part

Re: Bug#861263: debian-installer: zfs support

[Sam Kuper]

On 05/05/2017, Ben Hutchings  wrote:
> On Fri, 2017-05-05 at 14:26 +0100, Sam Kuper wrote:
>> On Wed, 2017-04-26 at 19:51:23 +0100, Ben Hutchings wrote:
>> > On Wed, 2017-04-26 at 18:20 +0200, Timo Haas wrote:
>> > > do you plan to support zfs as root filesystem in the installer?
>> >
>> > ZFS binaries are not distributable due to the licence conflict, so this
>> > is unlikely to happen.
>>
>> If the Debian Installer were instead to ship with, or to download at
>> runtime, the ZFS on Linux source code, would that be acceptable from a
>> licensing standpoint?
>
> I imagine this would be acceptable (though not in the default
> installer, which only uses and installs packages from main).

Good point. Potential avenues:

1. Move ZFS *source* into "main". Would this be possible without
compromising Debian's "obviously prudent" arrangement?[1] Should I CC
debian-legal?

2. Add ZFS to a Debian Installer that is not the *default* Debian
Installer. Does Debian distribute such an installer, to which the
facility to compile and run ZFS could be added?

Thanks :)

(Please CC me, as I am still not subscribed to the mailing list.)

[1] https://softwarefreedom.org/resources/2016/linux-kernel-cddl.html

Re: Bug#861263: debian-installer: zfs support

[Ben Hutchings]

On Fri, 2017-05-05 at 14:26 +0100, Sam Kuper wrote:
> On Wed, 2017-04-26 at 19:51:23 +0100, Ben Hutchings wrote:
> > On Wed, 2017-04-26 at 18:20 +0200, Timo Haas wrote:
> > > Dear Maintainer,
> > > 
> > > do you plan to support zfs as root filesystem in the installer?
> > 
> > ZFS binaries are not distributable due to the licence conflict, so this
> > is unlikely to happen.
> 
> If the Debian Installer were instead to ship with, or to download at
> runtime, the ZFS on Linux source code, would that be acceptable from a
> licensing standpoint?

I imagine this would be acceptable (though not in the default
installer, which only uses and installs packages from main).

Ben.

> If so, then if the user were to instruct the Installer to use ZFS for
> some or all partitions, the Installer would (download and) compile and
> run the ZFS code appropriately.
> 
> Please correct me if I am mistaken about this being viable in principle.
> 
> Please CC me if you do so, as I am not subscribed to the mailing list.
> 
-- 
Ben Hutchings
The program is absolutely right; therefore, the computer must be wrong.



signature.asc
Description: This is a digitally signed message part

Bug#861263: debian-installer: zfs support

[Sam Kuper]

On Wed, 2017-04-26 at 19:51:23 +0100, Ben Hutchings wrote:
> On Wed, 2017-04-26 at 18:20 +0200, Timo Haas wrote:
>> Dear Maintainer,
>>
>> do you plan to support zfs as root filesystem in the installer?
>
> ZFS binaries are not distributable due to the licence conflict, so this
> is unlikely to happen.

If the Debian Installer were instead to ship with, or to download at
runtime, the ZFS on Linux source code, would that be acceptable from a
licensing standpoint?

If so, then if the user were to instruct the Installer to use ZFS for
some or all partitions, the Installer would (download and) compile and
run the ZFS code appropriately.

Please correct me if I am mistaken about this being viable in principle.

Please CC me if you do so, as I am not subscribed to the mailing list.

Bug#861263: debian-installer: zfs support

[Ben Hutchings]

Control: tag -1 wontfix

On Wed, 2017-04-26 at 18:20 +0200, Timo Haas wrote:
> Package: debian-installer
> Severity: wishlist
> Tags: d-i
> 
> Dear Maintainer,
> 
> do you plan to support zfs as root filesystem in the installer?

ZFS binaries are not distributable due to the licence conflict, so this
is unlikely to happen.

Ben.

-- 
Ben Hutchings
All extremists should be taken out and shot.



signature.asc
Description: This is a digitally signed message part

Processed: Re: Bug#861263: debian-installer: zfs support

[Debian Bug Tracking System]

Processing control commands:

> tag -1 wontfix
Bug #861263 [debian-installer] debian-installer: zfs support
Added tag(s) wontfix.

-- 
861263: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=861263
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Bug#861263: debian-installer: zfs support

[Timo Haas]

Package: debian-installer
Severity: wishlist
Tags: d-i

Dear Maintainer,

do you plan to support zfs as root filesystem in the installer?



-- System Information:
Debian Release: 9.0
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64
 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.9.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.utf8, LC_CTYPE=en_US.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)