Re: If Debian support OS certification?

[xavierautobot]

Wysłano z mojego smartfona Samsung Galaxy.

Re: If Debian support OS certification?

[Paul Wise]

On Fri, May 19, 2017 at 11:43 PM, Thomas Goirand wrote:

> The use of a live-cd wouldn't work, simply because we also may want to
> certify heterogeneous hardware (think about your USB missile launcher
> example). Therefore, a general case live-cd will not cover all cases. In
> such cases, self-certification wouldn't work.

There are options for solving this:

Install the full range of hardware support packages in the live-cd.

Install isenkram on the live-cd:

https://tracker.debian.org/pkg/isenkram
https://wiki.debian.org/AppStream/Guidelines#Announcing_supported_hardware

> Though the Debian organization and brand would be clearly improved.

Well, that depends which segment of our user base you are focused on.
If you care about corporate use of Debian, sure it definitely will. It
doesn't help regular users, for them the InstallingDebianOn and
ChrootOnAndroid are probably enough. Some may even be put off by
pandering to corporate interests, but we do a fair bit of that
already.

> Yes, we want openness and fairness in the process, even in the list of
> approved people doing the certification.

I'm having trouble thinking of an appropriate certifier approval
process, any thoughts?

> IMO, we should concentrate on having one first
> hardware maker certified through a 3rd party.

Quanta seems like the obvious choice, since they asked for it recently.

Is anyone aware of an appropriate 3rd party?

> Incrementally improving the process is the only realistic way to go.

It is also important to get the fundamentals correct.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: If Debian support OS certification?

[Thomas Goirand]

Paul,

The use of a live-cd wouldn't work, simply because we also may want to
certify heterogeneous hardware (think about your USB missile launcher
example). Therefore, a general case live-cd will not cover all cases. In
such cases, self-certification wouldn't work.

On 05/17/2017 03:50 PM, Paul Wise wrote:
> On Wed, May 17, 2017 at 9:28 PM, Ian Jackson wrote:
>> I suggest we would set the testing fee to cover the price we pay the
>> outsourced contractor, plus a 10% donation to Debian.
> 
> Debian is not a legal entity so we can't enter into contracts. We
> would have to ask our TOs if they can do this on our behalf.

The legal entity granting the use of the log would be SPI. Therefore,
SPI could have this role.

> Debian
> itself doesn't get improved when $giantserver is tested and shown to
> run perfectly.

The OS / project probably not. Though the Debian organization and brand
would be clearly improved.

> Selecting contractors only from existing contributors sounds like
> corruption to me. I also think we do not want a single entity having a
> monopoly on certification. We also have a lot of people already doing
> consulting around Debian, so it makes sense to at minimum notify them
> about the possibility of certification work, and direct vendors to
> them. If we do want to force exclusivity (not convinced we do) of
> people who can perform certification tests, a new list of 'certified'
> certification folks could be created (like we have for consultants,
> CD/preinstalled vendors etc), including any possible volunteers (some
> folks might be interested) and willing consultants, who agreed to play
> by the rules and only certify appropriate systems.
> 
> https://www.debian.org/consultants/

Yes, we want openness and fairness in the process, even in the list of
approved people doing the certification. Though IMO, this shouldn't be
our mail focus *for the moment*. IMO, we should concentrate on having
one first hardware maker certified through a 3rd party. Incrementally
improving the process is the only realistic way to go.

Cheers,

Thomas Goirand (zigo)

Re: If Debian support OS certification?

[Zlatan Todoric]

On 05/17/2017 03:49 PM, Holger Levsen wrote:
> On Wed, May 17, 2017 at 02:32:20PM +0100, Ian Jackson wrote:
>> A trustworthy certification report that said "this machine would have
>> passed the certification, except that the wifi card requires a
>> separately supplied firmware blob from Debian non-free" would be
>> extremely useful to many users and potential purchasers.
>  
> my usb controller uses a blob, my cpu uses a blob, my bios is collection of
> blobs and my bios-loader is a blob, [...] and my wireless card uses a blob
> from debian non-free and then the certification tells me it's all fine except
> for the wireless card. Is that really that useful or trustworthy?
>
>
Hi Qubes Holger :D

I assume the above Ian's assertion is still valid from Debian PoV -
Debian main will run fine on hardware with all those blobs except it
will not have wi-fi. If all those blobs (that don't stop Debian main
being perfectly usable) are the issue here than Debian is not good
enough for anything and I think on contrary.

Talking more about all the blobs is very helpful of course, but its not
something Debian will go and solve, and at state where Debian is
(compared to other OS certification) I am leaning towards Ian's
sentiment here.

Re: If Debian support OS certification?

[Paul Wise]

On Wed, May 17, 2017 at 9:32 PM, Ian Jackson wrote:

> A trustworthy certification report that said "this machine would have
> passed the certification, except that the wifi card requires a
> separately supplied firmware blob from Debian non-free" would be
> extremely useful to many users and potential purchasers.

It also needs "Also, you need to be aware of these other proprietary
components and what their capabilities are", as mentioned.

> I wonder if we could have a certification level (and associated name,
> logo, etc.) that specifically permits exactly this kind of deviation.

Consensus upthread seems to be no logo or "certification" for things
that don't qualify for the agreed-on top level. Still mention the
hardware, but don't "certify" it.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: If Debian support OS certification?

[Holger Levsen]

On Wed, May 17, 2017 at 02:32:20PM +0100, Ian Jackson wrote:
> A trustworthy certification report that said "this machine would have
> passed the certification, except that the wifi card requires a
> separately supplied firmware blob from Debian non-free" would be
> extremely useful to many users and potential purchasers.
 
my usb controller uses a blob, my cpu uses a blob, my bios is collection of
blobs and my bios-loader is a blob, [...] and my wireless card uses a blob
from debian non-free and then the certification tells me it's all fine except
for the wireless card. Is that really that useful or trustworthy?


-- 
cheers,
Holger

"[...]" is were I ommited 5-20 more blobs (TPM, graphics card, fingerprint 
reader,
SSD to name 4) on this machine which only needs one blob from Debian non-free 
:-)


signature.asc
Description: Digital signature

Re: If Debian support OS certification?

[Paul Wise]

On Wed, May 17, 2017 at 9:28 PM, Ian Jackson wrote:

> I would be in favour of asking for payment.

I think only optional donations are acceptable, I've written a few
mails upthread about that.

> The work of actually doing the testing is tedious.  If we think people
> should be doing that, they should be paid for their time.

They will be, either by their vendor employer or vendor clients or
people who use hardware they want certified. I guess, like the rest of
Debian, there could be volunteers doing certification too, if anyone
finds it interesting (some might like playing with new hardware a
lot).

> We shouldn't be expecting hardware vendors to self-certify.  That is
> an invitation to trouble.  We don't have the effort to properly audit
> such a thing.  (See above.)

I don't see a problem if it is mostly automated via a live CD.

> I think the only sensible structure is probably for us to outsource
> the certification testing, and administration.  There must be at least
> only mildly-useless corporations who could handle this for us.  Of
> course the Debian project could contract with some of our existing
> contributors, but I think we should engage in a proper supplier
> selection process.
>
> I suggest we would set the testing fee to cover the price we pay the
> outsourced contractor, plus a 10% donation to Debian.

Debian is not a legal entity so we can't enter into contracts. We
would have to ask our TOs if they can do this on our behalf.

I'm also not sure our donors would think this is a useful use of
Debian funds; it mostly benefits vendors and large organisations who
insist on someone else testing Debian instead of themselves. Debian
itself doesn't get improved when $giantserver is tested and shown to
run perfectly.

Selecting contractors only from existing contributors sounds like
corruption to me. I also think we do not want a single entity having a
monopoly on certification. We also have a lot of people already doing
consulting around Debian, so it makes sense to at minimum notify them
about the possibility of certification work, and direct vendors to
them. If we do want to force exclusivity (not convinced we do) of
people who can perform certification tests, a new list of 'certified'
certification folks could be created (like we have for consultants,
CD/preinstalled vendors etc), including any possible volunteers (some
folks might be interested) and willing consultants, who agreed to play
by the rules and only certify appropriate systems.

https://www.debian.org/consultants/

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: If Debian support OS certification?

[Ian Jackson]

Paul Wise writes ("Re: If Debian support OS certification?"):
> For Debian I expect your proposal "do not require loading externally
> supplied non-free firmware" is something that most of Debian can agree
> is a reasonable endorsement target for now.

Yes.

I think this is rather unfortunate for all the reasons you set out in
your mail, but I can't see a politically workable alternative bright
line.

> > Otherwise, we'll have to display different types of logo, like "works
> > with Debian ... but", and then that starts to confuse users, which is
> > counter-productive.
> 
> I think for hardware that doesn't support whatever criteria we come up
> with, we just wouldn't have a certification logo but would say "this
> hardware is *not* Debian certified because ..., but can run Debian if
> ...". For "certified" hardware we would include the logo and say "this
> hardware is Debian certified, but you need to be aware of these
> proprietary components and what their capabilities are".

I think this is a very good idea.

A trustworthy certification report that said "this machine would have
passed the certification, except that the wifi card requires a
separately supplied firmware blob from Debian non-free" would be
extremely useful to many users and potential purchasers.

I wonder if we could have a certification level (and associated name,
logo, etc.) that specifically permits exactly this kind of deviation.

Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: If Debian support OS certification?

[Ian Jackson]

Paul Wise writes ("Re: If Debian support OS certification?"):
> On Wed, May 17, 2017 at 12:18 AM, Thomas Goirand wrote:
> > If we made such a decision, I'd be very supportive of it. We could make
> > it in a "soft" way, ie tell that we accept some kind of (re-occurring?)
> > sponsorship, and providing a range of acceptable payment. We could make
> > such payment not completely mandatory, but strongly recommended. I'm
> > convinced that, with proper wording (meaning someone *else* than me
> > would attempt it, as I don't think I'd be the best person to write such
> > thing), it would work, and help the project to find more sponsors.
> 
> I think it would be acceptable for the certification process to
> promote donating to Debian and or becoming Debian partners, but not to
> make the certification conditional on payment. That seems to be what
> you are saying here.

I would be in favour of asking for payment.

The work of actually doing the testing is tedious.  If we think people
should be doing that, they should be paid for their time.

We shouldn't be expecting hardware vendors to self-certify.  That is
an invitation to trouble.  We don't have the effort to properly audit
such a thing.  (See above.)

I think the only sensible structure is probably for us to outsource
the certification testing, and administration.  There must be at least
only mildly-useless corporations who could handle this for us.  Of
course the Debian project could contract with some of our existing
contributors, but I think we should engage in a proper supplier
selection process.

I suggest we would set the testing fee to cover the price we pay the
outsourced contractor, plus a 10% donation to Debian.

Ian.

Re: If Debian support OS certification?

[Paul Wise]

On Wed, May 17, 2017 at 6:34 AM, Thomas Goirand wrote:

> I wonder what you call "everything". In the majority of the servers on
> which I have installed Debian, no non-free firmware were required.

That would be surprising to me, I imagine every one of those servers
was running non-free pre-installed firmware in multiple parts of the
machine. At minimum, every modern platform has a non-free boot ROM
(the first code run by the CPU, physically read-only, may do signature
checking). Then there are CPU microcode, Intel ME/AMD PSP, BIOS/UEFI,
BMC/IPMI/iLO/etc, hard drive or SSD firmware, NIC firmware, screen
firmware, keyboard firmware and so on.

I think what you meant to say was that the majority of the servers you
have run Debian on do not require loading externally supplied non-free
firmware before they will work normally.

I agree that as a practical measure, that is the certification target
that is most useful to end users and most appropriate for Debian right
now, at least until RISC-V/lowRISC hardware becomes widespread.

That said, from the things that the Debian Intel microcode package
maintainer says about microcode bugs on IRC, I'm not sure that it is a
good idea to recommend users run Debian systems without the updates to
the non-free CPU microcode provided by Debian.

>From a Software Freedom PoV though, "do not require loading externally
supplied non-free firmware" may be worse, since pre-installed firmware
is the elephant in the room (hello Intel AMT security bugs). It also
makes reverse engineering harder since pre-installed firmware is
harder to extract. Often there are zero mechanisms (or completely
proprietary ones) to update pre-installed firmware, which complicates
the reverse engineering process significantly and or prevents it for
all but the most sophisticated reverse engineers, for example to those
who can glitch signature checking code by altering power levels.

Only certifying hardware that does not require loading externally
supplied non-free firmware just *incentivises* vendors to just
pre-install their non-free firmware, which has the potential to
slightly reduce Software Freedom around firmware long-term.
So, I'd like us to counteract these incentives by encouraging hardware
vendors to support coreboot and other libre firmware projects and
exposing information to users and vendors about what proprietary
pre-installed software/firmware is present and how it could be
problematic.

> In my view, a certification Debian logo means we fully endorse.

For Debian I expect your proposal "do not require loading externally
supplied non-free firmware" is something that most of Debian can agree
is a reasonable endorsement target for now. We probably would require
a GR to make a decision about the target though. The FSF RYF
endorsement is approximately what you suggest:

https://www.fsf.org/resources/hw/endorsement/respects-your-freedom
https://libreplanet.org/wiki/Group:Hardware/Certification_criteria

> I do believe a vast majority of the Debian community do not fully
> endorse the requirement of non-free blobs.

I'm not sure that is the case, there appears to be significant support
for including NIC/WiFi firmware in the primary ISO that people
download from the Debian website (or also linking to the non-free ISO
from the front page), because the lack of it means it is harder to
install Debian on modern hardware. There also appears to be support
for installing CPU firmware by default.

> A certification is different from a compatibility checklist.
> Let's not confuse the 2.

Good point, I'd certainly made that mistake up till now.

> Otherwise, we'll have to display different types of logo, like "works
> with Debian ... but", and then that starts to confuse users, which is
> counter-productive.

I think for hardware that doesn't support whatever criteria we come up
with, we just wouldn't have a certification logo but would say "this
hardware is *not* Debian certified because ..., but can run Debian if
...". For "certified" hardware we would include the logo and say "this
hardware is Debian certified, but you need to be aware of these
proprietary components and what their capabilities are".

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: If Debian support OS certification?

[Paul Wise]

On Wed, May 17, 2017 at 12:18 AM, Thomas Goirand wrote:

> If we made such a decision, I'd be very supportive of it. We could make
> it in a "soft" way, ie tell that we accept some kind of (re-occurring?)
> sponsorship, and providing a range of acceptable payment. We could make
> such payment not completely mandatory, but strongly recommended. I'm
> convinced that, with proper wording (meaning someone *else* than me
> would attempt it, as I don't think I'd be the best person to write such
> thing), it would work, and help the project to find more sponsors.

I think it would be acceptable for the certification process to
promote donating to Debian and or becoming Debian partners, but not to
make the certification conditional on payment. That seems to be what
you are saying here.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: If Debian support OS certification?

[Thomas Goirand]

On 05/04/2017 01:56 AM, Paul Wise wrote:
> On Thu, May 4, 2017 at 12:17 AM, Ben Hutchings wrote:
> 
>> No, they should not, otherwise this certification becomes meaningless.
> 
> I see these certifications primarily as a service to Debian users and
> not as endorsements of vendors, but as statements of fact. The
> consequences to users should stated as part of the certification
> output. "This system can run Debian main", "This system is missing
> drivers for XYZ", "This system requires non-free firmware", "This
> system requires a custom bootloader", "This system requires a custom
> kernel", "This system requires a custom kernel and must use sysvinit",
> "This system requires an unofficial Debian port", "This system
> requires recompiling Debian from scratch" (CPU requirements bumps or
> CPU bugs). Basically, a more automated version of InstallingDebianOn.

IMO, that certification program should be the best place where to
promote the fact we do want all drivers to be free (ie: without non-free
firmware), with everything from main. It is my view that we should only
accept that a system is compatible with Debian in that case only. In
such case, we should claim compatibility with Debian 8 and above, for
example. In any other case, we may just deny displaying a Debian logo.

> If Debian only certifies systems installed using official d-i images
> then we won't be certifying much, since almost everything requires
> preinstalled or runtime-loaded non-free firmware for some part of the
> system.

I wonder what you call "everything". In the majority of the servers on
which I have installed Debian, no non-free firmware were required. If a
vendor decides to use a WiFi board that requires a non-free blob,
well... too bad for them, IMO, they don't deserve our endorsement.

In my view, a certification Debian logo means we fully endorse. I do
believe a vast majority of the Debian community do not fully endorse the
requirement of non-free blobs.

A certification is different from a compatibility checklist. Let's not
confuse the 2.

> Since we already need two tiers of certifications for main vs
> non-free, is it really that much of a problem to add some more as long
> as our users are informed of the issues they will face? Users are
> going to buy or acquire those problematic systems anyway, especially
> in areas where there are almost zero devices that Debian could be
> certified for (for eg mobile devices). If they do and then decide to
> run Debian, information about what the consequences are would be
> useful.

I agree that it is useful information. But that is not what the
certification program should be about. IMO, a certified hardware should
fully work from main, period.

Otherwise, we'll have to display different types of logo, like "works
with Debian ... but", and then that starts to confuse users, which is
counter-productive.

Very happy to share thoughts with you here,
Cheers,

Thomas Goirand (zigo)

Re: If Debian support OS certification?

[Thomas Goirand]

On 05/06/2017 03:54 AM, Luca Filipozzi wrote:
> On Fri, May 05, 2017 at 10:40:10PM +0100, Ben Hutchings wrote:
>> On Fri, 2017-05-05 at 16:54 +0200, Thomas Goirand wrote:
>>> On 05/02/2017 02:35 AM, Paul Wise wrote:
 With my DSA hat on, we don't like being guinea pigs for development
 boards and pre-release hardware. This kind of hardware tends to be
 unreliable and require too much hand-holding. That said, we definitely
 welcome hardware sponsorship and partners.
>>>
>>> Absolutely. However, you may know that commercial distros are making
>>> their certification program a non-free (as in: you must pay your beer)
>>> thing. I do believe it'd be a fair way to get free (as in free beer)
>>> hardware for the DSA team. It's up to us to define the terms.
>>
>> Free as in free kittens?
> 
> I'm not interested in hardware as payment for certification. It's too open to
> abuse. Charge a fee for certification testing; use the funds to buy hardware.

If we made such a decision, I'd be very supportive of it. We could make
it in a "soft" way, ie tell that we accept some kind of (re-occurring?)
sponsorship, and providing a range of acceptable payment. We could make
such payment not completely mandatory, but strongly recommended. I'm
convinced that, with proper wording (meaning someone *else* than me
would attempt it, as I don't think I'd be the best person to write such
thing), it would work, and help the project to find more sponsors.

Your thoughts?
Cheers,

Thomas Goirand (zigo)

Re: If Debian support OS certification?

[Paul Wise]

On Tue, May 16, 2017 at 2:09 PM, Ritesh Raj Sarraf wrote:

> I am not sure if this got a page added.

I didn't add one, so I think yours is the first.

> [1] https://wiki.debian.org/InstallingDebianOn/Certification

I've renamed the page into the Hardware/ namespace and made minor fixes:

https://wiki.debian.org/Hardware/Certification

I've also invited Debian derivatives to link to certification programs too:

https://wiki.debian.org/Derivatives/CensusTemplate?action=diff=85=86

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: If Debian support OS certification?

[Ritesh Raj Sarraf]

On Tue, May 2, 2017 at 11:44 PM, Ritesh Raj Sarraf  wrote:

> On Tue, 2017-05-02 at 08:35 +0800, Paul Wise wrote:
> > > What they are interested about, is having *us*, Debian, to certify that
> > > their hardware work on our system, so that their customer trust they
> can
> > > buy it to run Debian. It'd be a bit weird if they were certifying
> > > themselves.
> >
> > I think that Debian members/contributors do not and should not hold a
> > monopoly on verifying that Debian works on a particular piece of
> > hardware.
> >
>
> As members, we should come up with a "Certification Policy" guide. Which
> should
> define what constitutes a particular machine being marked certified. Then a
> testsuite could be built accordingly.
>


I am not sure if this got a page added. But I've created one at [1]. Please
feel free to add any other points I may have missed.

[1] https://wiki.debian.org/InstallingDebianOn/Certification


-- 
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."

Re: If Debian support OS certification?

[Tollef Fog Heen]

]] Thomas Goirand 

> I do believe it'd be a fair way to get free (as in free beer) hardware
> for the DSA team. It's up to us to define the terms.

It would mean we'd end up with a hodgepodge of different servers from
different vendors with no coherent OOB access methods, we'd need to
track a lot of different firmware versions and so on.  It's not
something we want.

-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are

Re: If Debian support OS certification?

[Luca Filipozzi]

On Fri, May 05, 2017 at 10:40:10PM +0100, Ben Hutchings wrote:
> On Fri, 2017-05-05 at 16:54 +0200, Thomas Goirand wrote:
> > On 05/02/2017 02:35 AM, Paul Wise wrote:
> > > With my DSA hat on, we don't like being guinea pigs for development
> > > boards and pre-release hardware. This kind of hardware tends to be
> > > unreliable and require too much hand-holding. That said, we definitely
> > > welcome hardware sponsorship and partners.
> > 
> > Absolutely. However, you may know that commercial distros are making
> > their certification program a non-free (as in: you must pay your beer)
> > thing. I do believe it'd be a fair way to get free (as in free beer)
> > hardware for the DSA team. It's up to us to define the terms.
> 
> Free as in free kittens?

I'm not interested in hardware as payment for certification. It's too open to
abuse. Charge a fee for certification testing; use the funds to buy hardware.

-- 
Luca Filipozzi
http://www.crowdrise.com/SupportDebian

Re: If Debian support OS certification?

[Ben Hutchings]

On Fri, 2017-05-05 at 16:54 +0200, Thomas Goirand wrote:
> On 05/02/2017 02:35 AM, Paul Wise wrote:
> > With my DSA hat on, we don't like being guinea pigs for development
> > boards and pre-release hardware. This kind of hardware tends to be
> > unreliable and require too much hand-holding. That said, we definitely
> > welcome hardware sponsorship and partners.
> 
> Absolutely. However, you may know that commercial distros are making
> their certification program a non-free (as in: you must pay your beer)
> thing. I do believe it'd be a fair way to get free (as in free beer)
> hardware for the DSA team. It's up to us to define the terms.

Free as in free kittens?

Ben.

-- 
Ben Hutchings
The program is absolutely right; therefore, the computer must be wrong.



signature.asc
Description: This is a digitally signed message part

Re: If Debian support OS certification?

[Thomas Goirand]

On 05/02/2017 02:35 AM, Paul Wise wrote:
> With my DSA hat on, we don't like being guinea pigs for development
> boards and pre-release hardware. This kind of hardware tends to be
> unreliable and require too much hand-holding. That said, we definitely
> welcome hardware sponsorship and partners.

Absolutely. However, you may know that commercial distros are making
their certification program a non-free (as in: you must pay your beer)
thing. I do believe it'd be a fair way to get free (as in free beer)
hardware for the DSA team. It's up to us to define the terms.

Cheers,

Thomas Goirand (zigo)

Re: If Debian support OS certification?

[Thomas Goirand]

On 05/02/2017 02:35 AM, Paul Wise wrote:
> On Tue, May 2, 2017 at 5:15 AM, Thomas Goirand wrote:
> 
>> While it is nice to answer the way you did, here, Debian is missing yet
>> another opportunity that other commercial distro would not. Maybe we
>> should have a BoF at debconf Montreal about this.
> 
> Please do register a BoF, I'd be happy to attend if I can.

I'm still not 100% sure if I can make it to Montreal, though I've never
the less registered such a BoF. The title is:

Vendor hardware certification BoF

Cheers,

Thomas Goirand (zigo)

Re: If Debian support OS certification?

[Paul Wise]

On Thu, May 4, 2017 at 8:23 AM, Ben Hutchings wrote:

> Well I already acknowledged that, didn't I?

Yes, I felt like re-stating it.

> My concern was that the bar you were setting was so low as to be
> useless for distinguishing systems that are well supported by Debian
> from those that are not.

That is definitely something we want to avoid. I guess we would want
the front page to be the devices certified to the best available
standard and things certified to lesser standards could be on other
pages.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: If Debian support OS certification?

[Paul Wise]

On Thu, May 4, 2017 at 8:03 AM, Steve McIntyre wrote:

> Are you really claiming that systems already shipped with *firmware
> included* can't be installed using d-i? That's rather bogus, if so.
> Please explain?

That was the result of writing mail too early in the morning.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: If Debian support OS certification?

[Ben Hutchings]

On Thu, 2017-05-04 at 07:56 +0800, Paul Wise wrote:
> On Thu, May 4, 2017 at 12:17 AM, Ben Hutchings wrote:
> 
> > No, they should not, otherwise this certification becomes meaningless.
> 
> I see these certifications primarily as a service to Debian users and
> not as endorsements of vendors, but as statements of fact. The
> consequences to users should stated as part of the certification
> output. "This system can run Debian main", "This system is missing
> drivers for XYZ", "This system requires non-free firmware", "This
> system requires a custom bootloader", "This system requires a custom
> kernel", "This system requires a custom kernel and must use sysvinit",
> "This system requires an unofficial Debian port", "This system
> requires recompiling Debian from scratch" (CPU requirements bumps or
> CPU bugs). Basically, a more automated version of InstallingDebianOn.

If we require that vendors make those caveats clear in any self-
certification, then I agree that this could be useful.

> If Debian only certifies systems installed using official d-i images
> then we won't be certifying much, since almost everything requires
> preinstalled or runtime-loaded non-free firmware for some part of the
> system. We would basically only be able to certify RYF devices and may
> as well just require FSF RYF certification up-front before a system
> can be certified for Debian use.

Well I already acknowledged that, didn't I?

> Since we already need two tiers of certifications for main vs
> non-free, is it really that much of a problem to add some more as long
> as our users are informed of the issues they will face?

My concern was that the bar you were setting was so low as to be
useless for distinguishing systems that are well supported by Debian
from those that are not.

> Users are
> going to buy or acquire those problematic systems anyway, especially
> in areas where there are almost zero devices that Debian could be
> certified for (for eg mobile devices). If they do and then decide to
> run Debian, information about what the consequences are would be
> useful.

Right.

Ben.

-- 
Ben Hutchings
If the facts do not conform to your theory, they must be disposed of.



signature.asc
Description: This is a digitally signed message part

Re: If Debian support OS certification?

[Steve McIntyre]

On Thu, May 04, 2017 at 07:56:45AM +0800, Paul Wise wrote:
>On Thu, May 4, 2017 at 12:17 AM, Ben Hutchings wrote:
>
>> No, they should not, otherwise this certification becomes meaningless.
>
>I see these certifications primarily as a service to Debian users and
>not as endorsements of vendors, but as statements of fact. The
>consequences to users should stated as part of the certification
>output. "This system can run Debian main", "This system is missing
>drivers for XYZ", "This system requires non-free firmware", "This
>system requires a custom bootloader", "This system requires a custom
>kernel", "This system requires a custom kernel and must use sysvinit",
>"This system requires an unofficial Debian port", "This system
>requires recompiling Debian from scratch" (CPU requirements bumps or
>CPU bugs). Basically, a more automated version of InstallingDebianOn.
>
>If Debian only certifies systems installed using official d-i images
>then we won't be certifying much, since almost everything requires
>preinstalled or runtime-loaded non-free firmware for some part of the
>system. We would basically only be able to certify RYF devices and may
>as well just require FSF RYF certification up-front before a system
>can be certified for Debian use.

Are you really claiming that systems already shipped with *firmware
included* can't be installed using d-i? That's rather bogus, if
so. Please explain?

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Mature Sporty Personal
  More Innovation More Adult
  A Man in Dandism
  Powered Midship Specialty

Re: If Debian support OS certification?

[Paul Wise]

On Thu, May 4, 2017 at 12:17 AM, Ben Hutchings wrote:

> No, they should not, otherwise this certification becomes meaningless.

I see these certifications primarily as a service to Debian users and
not as endorsements of vendors, but as statements of fact. The
consequences to users should stated as part of the certification
output. "This system can run Debian main", "This system is missing
drivers for XYZ", "This system requires non-free firmware", "This
system requires a custom bootloader", "This system requires a custom
kernel", "This system requires a custom kernel and must use sysvinit",
"This system requires an unofficial Debian port", "This system
requires recompiling Debian from scratch" (CPU requirements bumps or
CPU bugs). Basically, a more automated version of InstallingDebianOn.

If Debian only certifies systems installed using official d-i images
then we won't be certifying much, since almost everything requires
preinstalled or runtime-loaded non-free firmware for some part of the
system. We would basically only be able to certify RYF devices and may
as well just require FSF RYF certification up-front before a system
can be certified for Debian use.

Since we already need two tiers of certifications for main vs
non-free, is it really that much of a problem to add some more as long
as our users are informed of the issues they will face? Users are
going to buy or acquire those problematic systems anyway, especially
in areas where there are almost zero devices that Debian could be
certified for (for eg mobile devices). If they do and then decide to
run Debian, information about what the consequences are would be
useful.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: If Debian support OS certification?

[Andre Felipe Machado]

Hello,
We developed a very detailed sequence of compatibility and  disk and storage 
stress tests  , including nfsv4, to homologate disk and storage systems for 
government 
profile and scale IMAP loads on Debian systems.
Fio tests carefully modeled Cyrus IMAP real world behaviour at such scale, 
confirmed at cyrus project list.
Maybe one can find useful to ADAPT such tests as part of Debian certification. 
Almost all commands depends on available RAM and CPU count, bandwidth, etc. The 
test 
procedures were for stress storage systems not for Debian itself.
You could download PDF linked at page at
https://comunidadeexpresso.serpro.gov.br/mediawiki/index.php/Infra/DataStorageServers
Despite written in brazilian portuguese, the command lines listed are easily 
readable.
Regards.
Andre Felipe

Re: If Debian support OS certification?

[Ben Hutchings]

On Wed, 2017-05-03 at 16:55 +0800, Paul Wise wrote:
> On Tue, 2017-05-02 at 23:29 +0530, Ritesh Raj Sarraf wrote:
[...]
> > Like most other Enterprise Linux Distributions, Debian too picks a
> > particular kernel (stable- lts) and to some extent also backports
> > fixes into it. That makes it a completely unique kernel, against
> > which certification needs to be done.
> 
> It is true that we use a unique version of Linux/kFreeBSD/Hurd but I
> would advocate a different approach. There is a lot of hardware that
> will never run mainline Linux and will never be able to be fully
> supported by Debian. These systems should be able to be certified to
> work with Debian
[...]

No, they should not, otherwise this certification becomes meaningless. 
Basically any system using one of our supported architectures can run a
'Debian' system with some custom components added.  But that system is
unlikely to get prompt updates to fix kernel security bugs - or maybe
any updates at all, depending on how the vendor (mis)configured APT.

If the vendor (or their SoC supplier) chooses to fork and not to
contribute back to Linux, they must accept the consequences, and we
should not endorse that fork.

Certification should mean that you can use the Debian installer or an
official Debian image on the system.  If it actually requires a custom
installer or image created by the vendor, that is out of our control
and ability to support.

(I leave aside the question of whether 'Debian' would include the
contrib and non-free sections.  I think that realistically we would
have to add a second tier of certification for the vast majority of
systems that require installation of non-free firmware for important
components like the GPU or network interface.)

Ben.

-- 
Ben Hutchings
friends: People who know you well, but like you anyway.



signature.asc
Description: This is a digitally signed message part

Re: If Debian support OS certification?

[Paul Wise]

On Tue, 2017-05-02 at 23:29 +0530, Ritesh Raj Sarraf wrote:

> As members, we should come up with a "Certification Policy" guide. Which 
> should
> define what constitutes a particular machine being marked certified. Then a
> testsuite could be built accordingly.

Sounds good to me, would you mind starting a wiki page for this?

> It will have to go beyond the "does boot" scope, in my opinion.

Clearly, since each bit of hardware in each particular situation has a
probably unique set of features that need testing in their own way.
For example if there is a USB missile launcher attached, it should
definitely use isenkram install pymissile and ask the user to run the
tests for that.

> Like most other Enterprise Linux Distributions, Debian too picks a
> particular kernel (stable- lts) and to some extent also backports
> fixes into it. That makes it a completely unique kernel, against
> which certification needs to be done.

It is true that we use a unique version of Linux/kFreeBSD/Hurd but I
would advocate a different approach. There is a lot of hardware that
will never run mainline Linux and will never be able to be fully
supported by Debian. These systems should be able to be certified to
work with Debian but the certification would make it clear which
version of each component was used, including those that were not from
Debian. For example ARM systems will be able to have OpenGL but only
with the proprietary binary drivers. Other systems will be able to run
one release of Debian but not another (for example my MIPS router can
run jessie but not stretch because the CPU requirements changed).

> In all the certification tools I've worked with, rigorous stress
> tests are the most important part. For example, for file systems,
> doing large amounts of I/O with different chunks; Buffered and Direct
> I/O etc. Single queue, multi queue. WRITE_SAME and TRIM related HW
> Commands. CPU Burn, Memory tests, Network etc.

That sounds like a description of anarcat's stressant project.

https://gitlab.com/anarcat/stressant

> All core components of a server hardware needs tests to certify any
> server hardware.

I would strongly suggest *not* limiting this project to servers.
There are at least various types of cloud providers, laptops, desktops,
SBCs, routers, TVs etc that Debian can probably run on in some way.

> Yes. But I think we need to provide a tool, process and guideline for them to
> follow. So far, from what I've checked, not much engagement has been initiated
> from the hardware vendors.

I think instead of a tool, we want a framework for packages available
in Debian to provide both automatic and manual instructions for testing
things outside of the Debian system. Then we want a setup that can run
the automatic tests and provide the manual instructions to the user.
The process would then be: boot ISO, wait for auto tests, do manual
tests and enter results, click submit, take photo of certification and
or save any digital artefacts of the certification to external media.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise


signature.asc
Description: This is a digitally signed message part

Re: If Debian support OS certification?

[Ritesh Raj Sarraf]

On Tue, 2017-05-02 at 08:35 +0800, Paul Wise wrote:
> > What they are interested about, is having *us*, Debian, to certify that
> > their hardware work on our system, so that their customer trust they can
> > buy it to run Debian. It'd be a bit weird if they were certifying
> > themselves.
> 
> I think that Debian members/contributors do not and should not hold a
> monopoly on verifying that Debian works on a particular piece of
> hardware.
> 

As members, we should come up with a "Certification Policy" guide. Which should
define what constitutes a particular machine being marked certified. Then a
testsuite could be built accordingly.

> I think a better approach would be to produce a Debian Live image that
> on boot checks as much of the hardware as possible automatically and
> lists a checklist for verifying the rest of the hardware works. Anyone
> could run the image and the resulting report could be uploaded to
> hardware.d.o, where it would be displayed publicly and count as a
> "certification". This way users can trust Debian to run on the
> hardware and there is no monopoly on certification. ISTR Ubuntu's
> certification stuff works similarly except that only Ubuntu can give
> the certification mark, probably in exchange for money.
> 

It will have to go beyond the "does boot" scope, in my opinion. Like most other
Enterprise Linux Distributions, Debian too picks a particular kernel (stable-
lts) and to some extent also backports fixes into it.

That makes it a completely unique kernel, against which certification needs to
be done. In all the certification tools I've worked with, rigorous stress tests
are the most important part. For example, for file systems, doing large amounts
of I/O with different chunks; Buffered and Direct I/O etc. Single queue, multi
queue. WRITE_SAME and TRIM related HW Commands.

CPU Burn, Memory tests, Network etc. All core components of a server hardware
needs tests to certify any server hardware.

> In any case, hardware vendors are in a much better position to be able
> to certify that Debian runs on their hardware than we are. They know
> exactly what functionality should be present and have access to get
> more hardware in case running Debian bricks their devices.

Yes. But I think we need to provide a tool, process and guideline for them to
follow. So far, from what I've checked, not much engagement has been initiated
from the hardware vendors.

-- 
Given the large number of mailing lists I follow, I request you to CC
me in replies for quicker response

signature.asc
Description: This is a digitally signed message part

Re: If Debian support OS certification?

[Luca Filipozzi]

On Tue, May 02, 2017 at 08:35:07AM +0800, Paul Wise wrote:
> On Tue, May 2, 2017 at 5:15 AM, Thomas Goirand wrote:
> 
> > While it is nice to answer the way you did, here, Debian is missing yet
> > another opportunity that other commercial distro would not. Maybe we
> > should have a BoF at debconf Montreal about this.
> 
> Please do register a BoF, I'd be happy to attend if I can.

Me, also.

> > Quanta is a company shipping servers. If I'm not mistaking, they're
> > located in Shanghai. One thing they used to do (and probably continue to
> > do) is building servers matching open specifications from the "open
> > compute" project. That really appeals to Debian moral standards, IMO.
> 
> Thanks for the info.
> 
> > What they are interested about, is having *us*, Debian, to certify that
> > their hardware work on our system, so that their customer trust they can
> > buy it to run Debian. It'd be a bit weird if they were certifying
> > themselves.
> 
> I think that Debian members/contributors do not and should not hold a
> monopoly on verifying that Debian works on a particular piece of
> hardware.
> 
> I think a better approach would be to produce a Debian Live image that
> on boot checks as much of the hardware as possible automatically and
> lists a checklist for verifying the rest of the hardware works. Anyone
> could run the image and the resulting report could be uploaded to
> hardware.d.o, where it would be displayed publicly and count as a
> "certification". This way users can trust Debian to run on the
> hardware and there is no monopoly on certification. ISTR Ubuntu's
> certification stuff works similarly except that only Ubuntu can give
> the certification mark, probably in exchange for money.
> 
> In any case, hardware vendors are in a much better position to be able
> to certify that Debian runs on their hardware than we are. They know
> exactly what functionality should be present and have access to get
> more hardware in case running Debian bricks their devices.

Wearing my DSA hat: fully agree.

> > Now one idea: one way we could provide the certification would be asking
> > for hardware sponsorship. This way, we (ie: the DSA team) would get
> > "free" hardware, in exchange for a certification. Obviously, we'd need
> > to discuss this with the DSA.
> 
> With my DSA hat on, we don't like being guinea pigs for development
> boards and pre-release hardware. This kind of hardware tends to be
> unreliable and require too much hand-holding. That said, we definitely
> welcome hardware sponsorship and partners.

Wearing my DSA hat: fully agree. So tired of flakey hardware.

Wearing my Partners hat: what value a certification that was 'bought' by
donating hardware (or a variable amount of funding) to Debian. I'd prefer a
declared fee structure for the service, for transparency. That said, I'd far
prefer Paul's suggestion of a Live CD.

> > Then we'd need a kind of "Debian certified hardware" logo that we would
> > agree the certified company use for some hardware. This would need SPI
> > approval, since that's the entity owning the rights for the Debian logo.
> 
> I expect we can probably get a logo created by updating this:
> 
> https://wiki.debian.org/DebianArt/RequestArtwork
> 
> Often it takes some promotion for the right people to notice though.

-- 
Luca Filipozzi
http://www.crowdrise.com/SupportDebian

Re: If Debian support OS certification?

[Paul Wise]

On Wed, Mar 15, 2017 at 8:35 AM, Paul Wise wrote:
> On Tue, Mar 14, 2017 at 12:17 PM, Paul Wise wrote:
>
>> At this time, Debian does not have a formal hardware certification program.
>
> I forgot to mention that we have experimental service called LAVA for
> automated hardware testing using Debian. Quanta could create a local
> hardware lab that would submit test results to Debian, please see the
> wiki page for more information about that:

I forgot to mention that the upstream Linux kernel community has a
similar service called kernelci that is also based on LAVA. If you do
setup a hardware lab it would be a good idea to have it send test
results to kernelci too:

https://kernelci.org/
https://kernelci.org/faq/

Some articles that mention it:

https://lwn.net/Articles/662882/
https://lwn.net/Articles/717221/

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

RE: If Debian support OS certification?

[賴裕文]

Hi Paul,

Thanks for your information.

I will discuss with my team internally. If have further question/problem, will 
let you know.

Best Regards,
Eric


-Original Message-
From: paul.is.w...@gmail.com [mailto:paul.is.w...@gmail.com] On Behalf Of Paul 
Wise
Sent: Wednesday, March 15, 2017 8:36 AM
To: Eric Lai (賴裕文)
Cc: debian-project@lists.debian.org
Subject: Re: If Debian support OS certification?

On Tue, Mar 14, 2017 at 12:17 PM, Paul Wise wrote:

> At this time, Debian does not have a formal hardware certification program.

I forgot to mention that we have experimental service called LAVA for automated 
hardware testing using Debian. Quanta could create a local hardware lab that 
would submit test results to Debian, please see the wiki page for more 
information about that:

https://wiki.debian.org/LAVA
https://lava.debian.net/

--
bye,
pabs

https://wiki.debian.org/PaulWise

Re: If Debian support OS certification?

[Paul Wise]

On Tue, Mar 14, 2017 at 12:17 PM, Paul Wise wrote:

> At this time, Debian does not have a formal hardware certification program.

I forgot to mention that we have experimental service called LAVA for
automated hardware testing using Debian. Quanta could create a local
hardware lab that would submit test results to Debian, please see the
wiki page for more information about that:

https://wiki.debian.org/LAVA
https://lava.debian.net/

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: If Debian support OS certification?

[Paul Wise]

On Tue, Mar 14, 2017 at 10:27 AM, Eric Lai (賴裕文) wrote:

> This is Eric from Quanta Cloud Technology, Taiwan.
> I am in charge of server hardware certification.
> We would like to know if Debian can perform hardware certification as well.
> If support, please advise the process and any document we can refer it.

At this time, Debian does not have a formal hardware certification program.

If you are interested in checking how Debian works with your hardware,
you could have your developers do the work, or hire consultants who
are familiar with Debian to test compatibility with your hardware.

https://www.debian.org/distrib/
https://www.debian.org/consultants/
https://lists.debian.org/debian-consultants/

We have a section on the wiki where users can report their hardware experiences:

https://wiki.debian.org/InstallingDebianOn

There is a similar service called h-node that is for all libre Linux distros:

https://h-node.org/

We have a couple of lists of hardware that ships with Debian out of
the box. If Quanta are shipping Debian on your hardware, feel free to
register an account on our wiki and edit the ShippingWithDebian page.

https://wiki.debian.org/Hardware/ShippingWithDebian
https://wiki.debian.org/DebianHardware

Ultimately, Debian relies on the upstream Linux kernel community for
most of our hardware support, so getting any needed drivers or patches
included upstream will mean that Debian supports your hardware.

https://www.kernel.org/
https://kernelnewbies.org/UpstreamMerge

Please also consider adding support for your servers to the coreboot
firmware project:

http://coreboot.org/

PS: in 2018 the annual Debian conference will be held in Hsinchu,
Taiwan. It would be great if Quanta could help fund DebConf18 and
Quanta developers could attend DebConf18.

https://wiki.debconf.org/wiki/DebConf18

Sponsorship information for 2017 is listed here:

https://debconf17.debconf.org/sponsors/become-a-sponsor/

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

If Debian support OS certification?

[賴裕文]

Hello,

This is Eric from Quanta Cloud Technology, Taiwan.

I am in charge of server hardware certification.

Currently, we have capability of Windows, RHEL, SLES, Ubuntu cert, etc…

We would like to know if Debian can perform hardware certification as well.

If support, please advise the process and any document we can refer it.

Thanks.
Best Regards,
Eric