Re: Realizing Good Ideas with Debian Money

[Thomas Goirand]

On 6/2/19 3:39 PM, Ben Hutchings wrote:
> On Fri, 2019-05-31 at 21:04 +, Luca Filipozzi wrote:
> [...]
>> However, without an HPE donation or discount, we are much more likely to
>> follow a less expensive approach: pairs of 2U servers with local
>> storage, etc. Still not cheap but not multiples of 100k.
>>
>> If a hardware vendor happens to offer a discounts, then we can stretch
>> the dollars further.
> [...]
> 
> As I understand it, list prices for "enterprise" hardware are set with
> the assumption that customers will negotiate a 50% or higher discount.
> If that's right, we should expect and ask for discounts, regardless of
> whether the vendor is interested in being a sponsor.
> 
> Ben.
> 

Oh, Ben... You don't know how much that's truth.

We got had a vendor (that I will not name) to lower his quote some N
amount of network cards from 13k to 5k, just because we told him we
would buy more and that we felt it was too expensive (sorry, I don't
think my employer would be happy if I was disclosing more details of who
and what...).

So very much, when purchasing hardware, negotiating is mandatory. Asking
2 vendors at once, comparing, let them know one has quoted for less, is
also super important. This is secret to no-one doing hardware purchase.

Cheers,

Thomas Goirand (zigo)

Re: Realizing Good Ideas with Debian Money

[Ben Hutchings]

On Fri, 2019-05-31 at 21:04 +, Luca Filipozzi wrote:
[...]
> However, without an HPE donation or discount, we are much more likely to
> follow a less expensive approach: pairs of 2U servers with local
> storage, etc. Still not cheap but not multiples of 100k.
> 
> If a hardware vendor happens to offer a discounts, then we can stretch
> the dollars further.
[...]

As I understand it, list prices for "enterprise" hardware are set with
the assumption that customers will negotiate a 50% or higher discount.
If that's right, we should expect and ask for discounts, regardless of
whether the vendor is interested in being a sponsor.

Ben.

-- 
Ben Hutchings
Unix is many things to many people,
but it's never been everything to anybody.




signature.asc
Description: This is a digitally signed message part

Re: Realizing Good Ideas with Debian Money

[Russ Allbery]

"G. Branden Robinson"  writes:

> My two cents[4] is that DSA should make its purchasing and hardware
> solicitation decisions with the architectural security issue fairly far
> down the priority list.  It saddens me to say that, but this new class
> of exploits, what van Schaik et al. call "microarchitectural data
> sampling" (MDS), is a playground for security researchers right now; a
> big rock has been turned over and bugs are erupting from the soil in a
> squamous frenzy.  It will take months or years for the situation to
> settle down.

> To acquire hardware based on what is known today is to risk buyer's
> remorse.  Plan on inescapable remorse later; every chip vendor will let
> us down until corporate managers learn to treat confidentiality and
> integrity as feature rather than cost centers.  (And count on them to
> forget what they've learned after a few quarters pass without
> embarassing headlines.)

+1 to this.  So far as I can tell, about the only thing that seems to
correlate with being less likely to have side-channel attacks is less
sophisticated scheduling pipelines and processor architecture (read:
simpler, slower processors).  And this area of security research is
changing very rapidly.  I would expect several more novel attacks to
surface.

Processors that don't have a bunch of non-free, unauditable bullshit as a
proprietary control plane would obviously be better, but you'd be paying a
prohibitive performance price (not to mention other issues).  There just
aren't any good options right now.  Buy (or accept donations of) whatever
makes sense for other reasons, and expect there to be mandatory microcode
updates, kernel and virtualization workarounds, and security bugs.

-- 
Russ Allbery (r...@debian.org)   

Re: Realizing Good Ideas with Debian Money

[G. Branden Robinson]

At 2019-06-01T09:04:39+0200, Philipp Kern wrote:
> Are we then looking more closely at AMD-based machines given that
> those had less problems around speculative attacks?

To borrow a phrase from Christopher Hitchens, this comment gives a
hostage to fortune.

My team at work closely follows (and part of it contributes to) the
research in microarchitectural timing-channel attacks; we just covered
the white paper on one of the three new attacks (RIDL)[1] on Friday.

I'll say this now because I don't know of anything embargoed that could
get me into trouble: don't count on AMD's good smell just this second to
last.  Remember that the previous round of embarrassments
(Spectre/Meltdown) didn't entirely spare AMD and ARM, and we haven't yet
seen any ground-up reimplementations of CPU cores with publically
auditable, formally-verified proofs of immunity to microarchitectural
timing channel attacks.

I see no reason to reward AMD with purchases based on what may be an
accidental and temporary lack of egg on the face.  This is the same firm
that followed Intel into the land of unauditable system management
firmware[2] and acquired ATI and shut down the information channels
enabling good free video drivers to be developed[3].

My two cents[4] is that DSA should make its purchasing and hardware
solicitation decisions with the architectural security issue fairly far
down the priority list.  It saddens me to say that, but this new class
of exploits, what van Schaik et al. call "microarchitectural data
sampling" (MDS), is a playground for security researchers right now; a
big rock has been turned over and bugs are erupting from the soil in a
squamous frenzy.  It will take months or years for the situation to
settle down.

To acquire hardware based on what is known today is to risk buyer's
remorse.  Plan on inescapable remorse later; every chip vendor will let
us down until corporate managers learn to treat confidentiality and
integrity as feature rather than cost centers.  (And count on them to
forget what they've learned after a few quarters pass without
embarassing headlines.)

Some day, perhaps, if the universe is less than maximally cruel, we'll
have the option of server-class RISC-V systems with fully-documented,
formally-verified designs.  But that day is not yet here.

In the meantime, always keep a fork with some cooked crow on it ready to
hand, so that the next time you run into one of the many "pragmatic"
people in our community who puffed and blew about how we didn't "really
need" open hardware, you can invite them to eat the stuff and so be
silent.

One wonders how pragmatic they'll feel when it's _their_ private data
being exfiltrated.

[1] https://mdsattacks.com/files/ridl.pdf
[2] https://libreboot.org/faq.html#amd
[3] I don't have a good cite handy for this, but Michel Dänzer can
doubtless tell the story with more accuracy and precision than I
can.
[4] ...further discounted reflecting my rather low level of project
activity.


signature.asc
Description: PGP signature

Re: Realizing Good Ideas with Debian Money

[Philipp Kern]

On 5/31/2019 11:04 PM, Luca Filipozzi wrote:
> Before you ask: an insecure hypervisor is an insecure buildd.

Are we then looking more closely at AMD-based machines given that those
had less problems around speculative attacks?

Kind regards
Philipp Kern

Re: Realizing Good Ideas with Debian Money

[Luca Filipozzi]

On Sat, Jun 01, 2019 at 01:50:25AM +0300, Adrian Bunk wrote:
> On Fri, May 31, 2019 at 09:04:24PM +, Luca Filipozzi wrote:
> >...
> > When we last crunched the numbers, maintaining a 5y refresh (to stay in
> > warranty, etc.) would require $75k-100k/yr. We've avoided that level of
> > annual expenditure because we are keeping hardware longer than 5y and
> > we've had amazing hardware [donations][1].
> >...
> 
> For me this implies that Debian should aim at having at least US$500k 
> reserves, to be prepared if there is no large donation coming for a 
> future refresh.

Plus another $300k in reserves for DebConf in case those donations don't
come through.

-- 
Luca Filipozzi

Re: Realizing Good Ideas with Debian Money

[Adrian Bunk]

On Fri, May 31, 2019 at 09:04:24PM +, Luca Filipozzi wrote:
>...
> When we last crunched the numbers, maintaining a 5y refresh (to stay in
> warranty, etc.) would require $75k-100k/yr. We've avoided that level of
> annual expenditure because we are keeping hardware longer than 5y and
> we've had amazing hardware [donations][1].
>...

For me this implies that Debian should aim at having at least US$500k 
reserves, to be prepared if there is no large donation coming for a 
future refresh.

> Luca Filipozzi

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: Realizing Good Ideas with Debian Money

[Luca Filipozzi]

On Fri, May 31, 2019 at 11:32:42PM +0300, Adrian Bunk wrote:
> On Wed, May 29, 2019 at 07:49:25AM -0400, Sam Hartman wrote:
> > So, there were two $300k donations in the last year.
> > One of these was earmarked for a DSA equipment upgrade.
> > DSA has a couple of options to pursue, but it's possible they may
> > actually spend $400k on an equipment refresh.
>
> The information required for an informed discussion on this topic
> is missing.

If we (DSA) elect to puchase HPE equipment for EU that is similar to the
NA equipment that was donated by HPE (blade enclosure, server blades,
10GE switches, FC storage, FC switches - thank you HPE!), then the
(list) costs are as per above.

However, without an HPE donation or discount, we are much more likely to
follow a less expensive approach: pairs of 2U servers with local
storage, etc. Still not cheap but not multiples of 100k.

If a hardware vendor happens to offer a discounts, then we can stretch
the dollars further. We would love to have HPE or Dell or Lenovo become
an ongoing hardware partner; hmu if you can facilitate.

When we last crunched the numbers, maintaining a 5y refresh (to stay in
warranty, etc.) would require $75k-100k/yr. We've avoided that level of
annual expenditure because we are keeping hardware longer than 5y and
we've had amazing hardware [donations][1].

Before you ask: an insecure hypervisor is an insecure buildd.

[1]: https://www.debian.org/News/2016/20161003.

-

Personally speaking, I would prefer to keep Debian a volunteer
organization.

-

-- 
Luca Filipozzi

Re: Realizing Good Ideas with Debian Money

[Adrian Bunk]

On Wed, May 29, 2019 at 07:49:25AM -0400, Sam Hartman wrote:
> 
> [moving a discussion from -devel to -project where it belongs]
> 
> > "Mo" == Mo Zhou  writes:
> 
> Mo> Hi,
> Mo> On 2019-05-29 08:38, Raphael Hertzog wrote:
> >> Use the $300,000 on our bank accounts?
> 
> So, there were two $300k donations in the last year.
> One of these was earmarked for a DSA equipment upgrade.
> DSA has a couple of options to pursue, but it's possible they may
> actually spend $400k on an equipment refresh.
> 
> $200k doesn't really go that far in terms of big infrastructure projects
> like bikeshed or similar.
> 
> I'm looking for someone who would be willing to guide a discussion of
> the Money issues Martin brought up in his campaign.  I don't have time
> to guide that effor myself.  Real thought needs to be put into it; it
> will be at least as much work as the discussions I'm leading on
> packaging practices and git if done correctly.
> 
> However it could be very valuable for the project.

The information required for an informed discussion on this topic
is missing.

What is really missing in Debian is an annual report from the
treasurer team covering all trusted organizations, listing the
accounts of all income and expenses as well as the reserves.

Some people are suggesting to spend 6 digit US$ amounts on whatever they 
consider important, while other people are spending their precious 
Debian time on getting mere 4 or 5 digit amounts of sponsorship for
a DebConf or MiniDebConf.

I don't see how these could both make sense at the same time.

Just from looking at the SPI part I would say that Debian has some 
reserves that could be used if needed, but new substantial recurring
commitments would not be reasonable since the long-term situation
is that there are usually < US$ 100k per year in both regular income
and expenses (excluding Debconf earmarks).

Other trusted organizations might show a similar or a completely 
different picture - it is impossible to start the budgetary
discussion you are asking for without the status quo of the
Debian finances as a basis.

> --Sam

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Realizing Good Ideas with Debian Money

[Sam Hartman]

[moving a discussion from -devel to -project where it belongs]

> "Mo" == Mo Zhou  writes:

Mo> Hi,
Mo> On 2019-05-29 08:38, Raphael Hertzog wrote:
>> Use the $300,000 on our bank accounts?

So, there were two $300k donations in the last year.
One of these was earmarked for a DSA equipment upgrade.
DSA has a couple of options to pursue, but it's possible they may
actually spend $400k on an equipment refresh.

$200k doesn't really go that far in terms of big infrastructure projects
like bikeshed or similar.

I'm looking for someone who would be willing to guide a discussion of
the Money issues Martin brought up in his campaign.  I don't have time
to guide that effor myself.  Real thought needs to be put into it; it
will be at least as much work as the discussions I'm leading on
packaging practices and git if done correctly.

However it could be very valuable for the project.

--Sam