Re: Debian built from non-Debian sources

[Alexey Salmin]

On Wed, Jul 19, 2017 at 3:59 PM, Holger Levsen  wrote:
> I dont even see how there would
> be much delay, if we had the policy of say requiring a debian-cd upload
> (to sid) with the changes made to create the stable upload.

Sid likely won't work here because both sid and testing may get ahead
of the version used to build stable image. And even if sources match,
packages won't because of the build environment. The backports repo
looks more suitable, but again at some point you may want to see a
backported testing version of utilities there, not patched stable.

> And maybe (probably?) there is a case for the Stretch images where one tool
wasnt uploaded yet. Seems like a bug to me too (if thats the case), but not
really a reason for a flamewar nor calling out trolling.

In the compilers development a failure to build itself is normally
considered a critical compiler bug that blocks the release. Even
though bootstrapping is not the main purpose of compiler existence.
Similarly, a failure to produce an install image of debian on a system
running the same version of debian sounds like a perfectly valid
ground the delay the release.

I guess when the next release approaches someone may want to set up
nightly image builds of testing using clean testing environment and
report bugs in the building process (if any) as RC. I know the
debian-cd team already have weekly builds of testing, but I assume
they run the patched toolchain from alioth repo. I'll see if I can
arrange a self-build testing framework for x86 and amd64, but I'll
definitely won't be able to cover all archs.

Re: Debian built from non-Debian sources

[Ian Jackson]

Holger Levsen writes ("Re: Debian built from non-Debian sources"):
> On Tue, Jul 18, 2017 at 09:50:48PM +0200, Bernd Zeimetz wrote:
> > So do I understand it right that you are actively going to test the CD
> > build process in a long enough time before the release and send patches
> > in time to make sure the changes will be part of the release?
> > 
> > If not - please stop trolling…
>  
> I dont understand why this is seen as trolling. We are willing to delay our
> release for (certain) serious bugs and here I dont even see how there would 
> be much delay, if we had the policy of say requiring a debian-cd upload
> (to sid) with the changes made to create the stable upload.
> 
> I also think this is basically happening, just not per policy but per Sledge
> just doing that work.

IMO a better arrangement would be something that
  - did not involve any manual intervention by the image preparation team
  - ensured that the actual source code was published _somewhere_
  - automatically ensured that the generated image farm contains
enough information to always reliably find the corresponding version

Even if the place where the files end up is not the Debian archive.

To look at this another way:

I think publishing the source is the vitally important part.  That
allows anyone else to reproduce what we have done.

Publishing the source _as a package in the Debian archive_ involves
additional release management work, and other additional
complications.  If the source code actually used is published, that
extra work does not need to be done by the image preparation team.  It
can be done by *anyone* who can get a sponsor for their upload.

It is great if the image preparation team do this work too.  But I
don't want them to feel they have to.  I also don't want a situation
to arise where, because that work hasn't been done, the source is
completely unavailable.

I'm afraid I'm not offering to write the image-script-autopublication
patch.  But those who are complaining about lack of timely provision
of source code should write that patch, rather than moaning.

Those who are complaining about the packages in the archive being out
of date should step forward and volunteer to take over doing the
uploads.

Ian.

Re: Debian built from non-Debian sources

[Holger Levsen]

On Tue, Jul 18, 2017 at 09:50:48PM +0200, Bernd Zeimetz wrote:
> So do I understand it right that you are actively going to test the CD
> build process in a long enough time before the release and send patches
> in time to make sure the changes will be part of the release?
> 
> If not - please stop trolling…
 
I dont understand why this is seen as trolling. We are willing to delay our
release for (certain) serious bugs and here I dont even see how there would 
be much delay, if we had the policy of say requiring a debian-cd upload
(to sid) with the changes made to create the stable upload.

I also think this is basically happening, just not per policy but per Sledge
just doing that work.

And maybe (probably?) there is a case for the Stretch images where one tool
wasnt uploaded yet. Seems like a bug to me too (if thats the case), but not
really a reason for a flamewar nor calling out trolling.

(So also thanks to Jonas *and* Sledge from me, for keeping this thread
constructive for the most part!)


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: Debian built from non-Debian sources

[Ian Jackson]

Steve McIntyre writes ("Re: Debian built from non-Debian sources"):
> There are several repos involved. They're described at the DebianCD
> team wiki page
>   https://wiki.debian.org/Teams/DebianCd
> which is itself linked from the top-level of our download area at
>   https://get.debian.org/images/

Thanks for the links.  Also, thanks for producing constructive and
informative mails even though you feel attacked.  (And of course for
the work behind those links and behind the images.)

Regards,
Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Debian built from non-Debian sources

[Steve McIntyre]

On Tue, Jul 18, 2017 at 07:15:47PM +0100, Ian Jackson wrote:
>Steve McIntyre writes ("Re: Debian built from non-Debian sources"):
>> For the record, all the changes I'm talking about go into public git
>> with all of the configuration and scripting that we use on our image
>> building machine.
>
>For me, this is the important part.  (There should be a pointer to it
>somewhere, ideally.)
>
>Since some people apparently want to improve it, then perhaps you
>could provide the url for the repo and they could send you patches.

There are several repos involved. They're described at the DebianCD
team wiki page

  https://wiki.debian.org/Teams/DebianCd

which is itself linked from the top-level of our download area at

  https://get.debian.org/images/

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
Who needs computer imagery when you've got Brian Blessed?

Re: Debian built from non-Debian sources

[Alexey Salmin]

On Tue, Jul 18, 2017 at 9:15 PM, Ian Jackson
<ijack...@chiark.greenend.org.uk> wrote:
> Steve McIntyre writes ("Re: Debian built from non-Debian sources"):
>> For the record, all the changes I'm talking about go into public git
>> with all of the configuration and scripting that we use on our image
>> building machine.
>
> For me, this is the important part.  (There should be a pointer to it
> somewhere, ideally.)
>
> Since some people apparently want to improve it, then perhaps you
> could provide the url for the repo and they could send you patches.

I assume it's the repo on alioth [1].

[1] 
https://alioth.debian.org/plugins/scmgit/cgi-bin/gitweb.cgi?a=project_list;pf=debian-cd

Re: Debian built from non-Debian sources

[Bernd Zeimetz]

On 07/18/2017 03:12 AM, Jonas Smedegaard wrote:
> I do not say you are doing it wrong.
> 
> I say we(!) can do better.  I can do better if I can repreat your work 
> using Debian components.  I can more confidently verify that I did not 
> mess things up if I can mimic with a higher degree your work. [...]

So do I understand it right that you are actively going to test the CD
build process in a long enough time before the release and send patches
in time to make sure the changes will be part of the release?

If not - please stop trolling the people who are actually doing awesome
work to ensure we can ship a release.


-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F



signature.asc
Description: OpenPGP digital signature

Re: Debian built from non-Debian sources

[Ian Jackson]

Steve McIntyre writes ("Re: Debian built from non-Debian sources"):
> For the record, all the changes I'm talking about go into public git
> with all of the configuration and scripting that we use on our image
> building machine.

For me, this is the important part.  (There should be a pointer to it
somewhere, ideally.)

Since some people apparently want to improve it, then perhaps you
could provide the url for the repo and they could send you patches.

Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Debian built from non-Debian sources

[Don Armstrong]

On Tue, 18 Jul 2017, Alexey Salmin wrote:
> No one can volunteer to fix it until it's acknowledged this is worth
> fixing.

If you think it's worth fixing, you can volunteer yourself to fix it.
That would look something like:

"I'm interested in helping make sure that THINGY happens. Would
patches which helped do THINGY be acceptable in principle? If not, is
there an approach which would accomplish near-THINGY that would be?"

and then show that you're actually invested in THINGY and the underlying
project by participating.

Otherwise, you can just raise it as a wishlist bug, make the best case
you can for it in one message, and hope for the best.

-- 
Don Armstrong  https://www.donarmstrong.com

Rule 6: "If violence wasn't your last resort, you failed to resort to
enough of it."
  -- Howard Tayler _Schlock Mercenary_  March 13th, 2005
 http://www.schlockmercenary.com/d/20050313.html

Re: Debian built from non-Debian sources

[Steve McIntyre]

alexey.sal...@gmail.com wrote:
>On Tue, Jul 18, 2017 at 2:09 AM, Steve McIntyre  wrote:
>> Making images often requires tweaks to the build script at/near
>> release time. The archive continues to be a moving target until very
>> close to that time. More than once we've fixed things or added
>> workarounds in the image generation scripts *on release day*. I'm not
>> going to remove the ability to do that and make working images to
>> pander to your ideals here.
>
>This is very understandable. The thing is however, exact same
>arguments can be applied to justify e.g. usage of proprietary software
>to produce Debian images. In both cases tools used to reproduce an
>image are not available to the community and it doesn't make much
>difference whether these tools are completely unavailable or just
>incorporate some non-public patches and tweaks. This is where people
>get worried.

Comparing my work here to proprietary software is *massively*
insulting and demotivating. Thanks *so* much for that. It really
helps.

For the record, all the changes I'm talking about go into public git
with all of the configuration and scripting that we use on our image
building machine.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"You can't barbecue lettuce!" -- Ellie Crane

Re: Debian built from non-Debian sources

[Alexey Salmin]

On Tue, Jul 18, 2017 at 4:51 PM, Scott Kitterman  wrote:
>
>
> On July 18, 2017 8:41:43 AM EDT, Alexey Salmin  
> wrote:
>>On Tue, Jul 18, 2017 at 2:09 AM, Steve McIntyre 
>>wrote:
>>> Making images often requires tweaks to the build script at/near
>>> release time. The archive continues to be a moving target until very
>>> close to that time. More than once we've fixed things or added
>>> workarounds in the image generation scripts *on release day*. I'm not
>>> going to remove the ability to do that and make working images to
>>> pander to your ideals here.
>>
>>This is very understandable. The thing is however, exact same
>>arguments can be applied to justify e.g. usage of proprietary software
>>to produce Debian images. In both cases tools used to reproduce an
>>image are not available to the community and it doesn't make much
>>difference whether these tools are completely unavailable or just
>>incorporate some non-public patches and tweaks. This is where people
>>get worried.
>
> Until someone volunteers to help, I think this is just a bunch of nit picking 
> that the people who are doing the work should actively ignore.  Personally, I 
> think sniping about how people who are doing the work are doing it "wrong" by 
> people who aren't is one of the most demotivating things that happens in 
> Debian.
>
> Is there anyone who cares about this that is willing to help?
>
> Scott K
>

No one can volunteer to fix it until it's acknowledged this is worth
fixing. So far it's maintained that the current approach is the right
one and should not be changed "to pander to smb's ideals". This is
something I disagree with. I write that reproducibility of images is a
valuable property and if current processes don't ensure it, then it's
reasonable to improve these processes. Now, if we manage to agree on
that -- then yes, volunteers are welcome to implement the actual
change. At the moment they're clearly not.

Re: Debian built from non-Debian sources

[Scott Kitterman]

On July 18, 2017 8:41:43 AM EDT, Alexey Salmin  wrote:
>On Tue, Jul 18, 2017 at 2:09 AM, Steve McIntyre 
>wrote:
>> Making images often requires tweaks to the build script at/near
>> release time. The archive continues to be a moving target until very
>> close to that time. More than once we've fixed things or added
>> workarounds in the image generation scripts *on release day*. I'm not
>> going to remove the ability to do that and make working images to
>> pander to your ideals here.
>
>This is very understandable. The thing is however, exact same
>arguments can be applied to justify e.g. usage of proprietary software
>to produce Debian images. In both cases tools used to reproduce an
>image are not available to the community and it doesn't make much
>difference whether these tools are completely unavailable or just
>incorporate some non-public patches and tweaks. This is where people
>get worried.

Until someone volunteers to help, I think this is just a bunch of nit picking 
that the people who are doing the work should actively ignore.  Personally, I 
think sniping about how people who are doing the work are doing it "wrong" by 
people who aren't is one of the most demotivating things that happens in Debian.

Is there anyone who cares about this that is willing to help?

Scott K

Re: Debian built from non-Debian sources

[Alexey Salmin]

On Tue, Jul 18, 2017 at 2:09 AM, Steve McIntyre  wrote:
> Making images often requires tweaks to the build script at/near
> release time. The archive continues to be a moving target until very
> close to that time. More than once we've fixed things or added
> workarounds in the image generation scripts *on release day*. I'm not
> going to remove the ability to do that and make working images to
> pander to your ideals here.

This is very understandable. The thing is however, exact same
arguments can be applied to justify e.g. usage of proprietary software
to produce Debian images. In both cases tools used to reproduce an
image are not available to the community and it doesn't make much
difference whether these tools are completely unavailable or just
incorporate some non-public patches and tweaks. This is where people
get worried.

Re: Debian built from non-Debian sources

[Ian Jackson]

Steve McIntyre writes ("Re: Debian built from non-Debian sources"):
> Making images often requires tweaks to the build script at/near
> release time. The archive continues to be a moving target until very
> close to that time. More than once we've fixed things or added
> workarounds in the image generation scripts *on release day*. I'm not
> going to remove the ability to do that and make working images to
> pander to your ideals here.

Perhaps it would be a good idea to arrange that we at least
automatically distribute the source code to the image ?

If you frequently need to edit the build script at the last moment,
can the build script copy itself into its output ?  It wouldn't have
to be in the actual iso; it could live next to it on the mirrors.

Forgive me if we already do that.

Ian.

Re: Debian built from non-Debian sources

[Paul Wise]

On Tue, Jul 18, 2017 at 9:09 AM, Steve McIntyre wrote:

> Making images often requires tweaks to the build script at/near
> release time. The archive continues to be a moving target until very
> close to that time. More than once we've fixed things or added
> workarounds in the image generation scripts *on release day*.

Does the freeze have an affect on this situation?

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Debian built from non-Debian sources

[Jonas Smedegaard]

Quoting Steve McIntyre (2017-07-18 02:32:11)
> On Tue, Jul 18, 2017 at 02:04:04AM +0200, Jonas Smedegaard wrote:
> >Quoting Steve McIntyre (2017-07-18 01:09:43)
> >> I've corrected several of your incorrect assertions already. I'm bored 
> >> of this.
> >
> >Your arrogant attitude hurts!
> 
> I'm *sorry* if you're hurt. Look at it from my point of view. You've
> come in with complaints about stuff I've spent a very long time
> working on. You've made several factual mistakes amongst those
> complaints, and you've continued to imply that I'm doing it wrong,
> showing little apparent understanding or experience of the problem
> space.

I do not say you are doing it wrong.

I say we(!) can do better.  I can do better if I can repreat your work 
using Debian components.  I can more confidently verify that I did not 
mess things up if I can mimic with a higher degree your work.  When I am 
more confident in testing that my setup fails not because I am stupid 
but there is some likelihood that it is a more general problem, I am 
more likely to dare raise it as a bugreport.  Yes, I know that you 
welcome my asking on the mailinglist - but I know that I can have 
trouble explaining myself, so I can be shy.  I don't think I am alone.

I try argue that treating build tools used for producing Debian make 
sense to get packaged for Debian.  Could be after the fact.

I say your looking at services and build tools together and concluding 
the task is too big might change by looking at a smaller task.

This is not an attack on your work.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: signature

Re: Debian built from non-Debian sources

[Steve McIntyre]

On Tue, Jul 18, 2017 at 02:04:04AM +0200, Jonas Smedegaard wrote:
>Quoting Steve McIntyre (2017-07-18 01:09:43)
>> I've corrected several of your incorrect assertions already. I'm bored 
>> of this.
>
>Your arrogant attitude hurts!

I'm *sorry* if you're hurt. Look at it from my point of view. You've
come in with complaints about stuff I've spent a very long time
working on. You've made several factual mistakes amongst those
complaints, and you've continued to imply that I'm doing it wrong,
showing little apparent understanding or experience of the problem
space.

This does not make for a productive conversation.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"When C++ is your hammer, everything looks like a thumb." -- Steven M. Haflich

Re: Debian built from non-Debian sources

[Jonas Smedegaard]

Quoting Steve McIntyre (2017-07-18 01:09:43)
> I've corrected several of your incorrect assertions already. I'm bored 
> of this.

Your arrogant attitude hurts!


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: signature

Re: Debian built from non-Debian sources

[Steve McIntyre]

Jonas wrote:
>Quoting Steve McIntyre (2017-07-17 02:00:25)
>
>> But a *lot* of the infrastructure we use to run Debian is not exactly 
>> what's been packaged, as already mentioned. Look at dak. debian-cd, 
>> live-wrapper et al *are* packaged, but we're not *necessarily* using 
>> the exact code that's in the stable archive at any point. We're 
>> typically using code from git on the build machines, to allow for more 
>> flexibility in terms of changes to build scripts as problems arise. We 
>> release things to the archive periodically as a convenience to users, 
>> but serious use often necessitates using git too. This isn't going to 
>> change any time soon.
>
>Sure it would be ideal to keep track of *everything* we do, including 
>how we run services.  But as mentioned above I distinguish between 
>services and things directly affecting our product.  Would you agree 
>that at first limiting the task to covering only the tools directly 
>affecting our product (e.g. debian-cd, liver-wrapper, libisofs) is more 
>realistic than tracking also e.g. dak and Alioth?
>
>For starters, I believe they all exist as packages in Debian, it is 
>"only" a matter of releasing into Debian the specific version used in 
>production.
>
>...but since they seemingly are excempt from Debian Policy exactly 
>because the code used is not packaged code, we cannot track this issue 
>in the same way we track issues with packages.  We can "ask on the 
>list"...

I've corrected several of your incorrect assertions already. I'm bored
of this.

Making images often requires tweaks to the build script at/near
release time. The archive continues to be a moving target until very
close to that time. More than once we've fixed things or added
workarounds in the image generation scripts *on release day*. I'm not
going to remove the ability to do that and make working images to
pander to your ideals here.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"Because heaters aren't purple!" -- Catherine Pitt

Re: Debian built from non-Debian sources

[Tollef Fog Heen]

]] Ian Jackson 

> Russ Allbery writes ("Re: Debian built from non-Debian sources"):
> > I think it would be interesting to strive for making available all Debian
> > infrastructure in our archives (although I think you may find that you'll
> > need a separate archive that doesn't correspond to stable or unstable,
> > based on having done similar things in the past), but it would be
> > premature to put a requirement into Policy until we actually *did* decide
> > to do that.  Which would affect a ton of different teams, and would be
> > quite a bit of work.
> 
> As a practical matter, complex bespoke services are much easier to run
> directly out of their vcs trees.

I've been toying with the idea of running those services from
containers.  That would at least get us a runnable artifact, even if it
wasn't purely generated from the archive.  (Yes, we'd need to publish
them somewhere and record where they came from and there's a lot of
practical questions.)

-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are

Re: Debian built from non-Debian sources

[Russ Allbery]

Jonas Smedegaard  writes:

> I still believe that libisofs are closer tied to our product than to our
> surrounding services: We (or derivatives) may upgrade/replace/skip
> Apache or dak or Alioth and still deliver an identical product, but
> upgrading libisofs may directly cause an image to fail or succeed.
> Isn't that exactly the reason you have chosen to not rely on Debian
> packages but stayed in close contact with upstream and custom compiled
> versions for use in the release process?

Maybe it just depends on one's perspective, but with my Debian user hat
on, I interact with the output of dak and its integrations with apt and
debootstrap and the like *way* more often than I interact with the
installer.  The output of libisofs is something I use every year or two;
the output of dak is something I consume multiple times per day.

Not arguing that you're wrong about the close link between libisofs and
our product, but I do think that any argument you can make about it
applies about equally well to dak.  They're both "just" ways of shipping a
set of Debian packages in a particular consumable configuration used by
key components of our distribution.

-- 
Russ Allbery (r...@debian.org)   

Re: Debian built from non-Debian sources

[Jonas Smedegaard]

Quoting Steve McIntyre (2017-07-17 02:00:25)
> Jonas wrote:
> >Quoting Steve McIntyre (2017-07-16 22:14:29)
> >> Jonas wrote:
> >> >Quoting Andrey Rahmatullin (2017-07-16 19:28:06)
> >> >> On Sun, Jul 16, 2017 at 07:05:10PM +0200, Jonas Smedegaard 
> >> >> wrote:
> >> >> > Is our install images excepmt from our Policy that all 
> >> >> > dependencies must be in Debian, or am I mistaken that we have 
> >> >> > such Policy?
> >> >> Do we?  The Debian Policy covers only debs.
> >> >> Also, dak isn't in the archive either.
> >> >
> >> >I thought Policy covered what we distribute - which excludes dak 
> >> >but includes libisofs code embedded in installer images.
> >> 
> >> Can you identify any code at all from libisofs which is embedded in 
> >> the images? I'm honestly not aware of any.
> >
> >I believe the embedded MBR is part of the libisofs project.
> 
> No, the MBR is generated from the isolinux/syslinux packages. xorriso 
> (libisofs) just updates some pointers in there to point at the El 
> Torito bootable images, to add a fake partition table, etc.

Ok, so I stand corrected in that libisofs do not provide code (like e.g. 
a library).  Thanks for clarifying.

I still believe that libisofs are closer tied to our product than to our 
surrounding services: We (or derivatives) may upgrade/replace/skip 
Apache or dak or Alioth and still deliver an identical product, but 
upgrading libisofs may directly cause an image to fail or succeed.  
Isn't that exactly the reason you have chosen to not rely on Debian 
packages but stayed in close contact with upstream and custom compiled 
versions for use in the release process?

I mean, if it were packages, then a new version of the libisofs package 
would likely result in a binNMU of the image packages.  An updated 
version of Debian _services_ like apache or Alioth or dak would unlikely 
lead to binNMUs of any packages.


> >My concern is the ability to replicate and derive the least possible 
> >from Debian resources like the install images.
> 
> ACK, understood.
> 
> >Concretely The Debian derivative PureOS is having trouble booting their 
> >homemade live image on some hardware, but boots fine on both Debian 
> >netinst image and Debian live image.  Looking at the properly working 
> >images I noticed that the live image for stable was produced using 
> >newer-than-stable libisofs,
> 
> Sorry, wrong. It was built using the standard xorriso and libisofs
> versions in stretch. From the stretch-based VM used to build it:

Whoops - you are right.  My eyes are apparently too used to see equal 
versions in tracker.d.o as meaning unstable+testing, not the rare 
unstable+testing+stable that we have now.  Sorry!


> If the PureOS folks are having problems, they could ask on the 
> debian-cd list?

Sure there is always the option to simply ask.

I mentioned the concrete problem as an *example* of a more general 
concern: Whether or not it is even *possible* for anyone outside Debian 
to replicate the Debian product.  We have Debian Policy governing (i.e. 
_documenting_ for outsiders) how we create packages, but apparently we 
have nothing governing the final procedures of how we create our images.


> >and that the stable netinst image was produced using a
> >never-in-Debian release of libisofs.
> 
> Right, I'll give you that one. But there's *seriously* nothing special 
> there any more than what you'd find in any backport to jessie.

I am talking about _avoiding_ uncertainty: Irrelevant to compare with 
other ways of introducing variability to a Debian system!


> But a *lot* of the infrastructure we use to run Debian is not exactly 
> what's been packaged, as already mentioned. Look at dak. debian-cd, 
> live-wrapper et al *are* packaged, but we're not *necessarily* using 
> the exact code that's in the stable archive at any point. We're 
> typically using code from git on the build machines, to allow for more 
> flexibility in terms of changes to build scripts as problems arise. We 
> release things to the archive periodically as a convenience to users, 
> but serious use often necessitates using git too. This isn't going to 
> change any time soon.

Sure it would be ideal to keep track of *everything* we do, including 
how we run services.  But as mentioned above I distinguish between 
services and things directly affecting our product.  Would you agree 
that at first limiting the task to covering only the tools directly 
affecting our product (e.g. debian-cd, liver-wrapper, libisofs) is more 
realistic than tracking also e.g. dak and Alioth?

For starters, I believe they all exist as packages in Debian, it is 
"only" a matter of releasing into Debian the specific version used in 
production.

...but since they seemingly are excempt from Debian Policy exactly 
because the code used is not packaged code, we cannot track this issue 
in the same way we track issues with packages.  We can "ask on the 
list"...


 - Jonas

-- 
 * Jonas Smedegaard - idealist & 

Re: Debian built from non-Debian sources

[Holger Levsen]

On Mon, Jul 17, 2017 at 12:09:30AM +0200, Christian Seiler wrote:
> [...] downstreams should be able to reproduce Debian images.
> In the short term only in functionality and not bit-by-bit, but I
> would consider reproducible image builds a worthwhile long-term goal
> after fully reproducible package builds throughout the archive have
> been achieved.
 
fully reproducible images are possible with unreproducible packages,
those two are related+similar but don't depend on each other.

(of course with unreproducible packages in reproducible images you
still need to believe the providers of those unreproducible packages
that they contain what they should contain, but at least the image
can be reproducible…)


-- 
cheers,
Holger


signature.asc
Description: Digital signature

Re: Debian built from non-Debian sources

[Ian Jackson]

Russ Allbery writes ("Re: Debian built from non-Debian sources"):
> I think it would be interesting to strive for making available all Debian
> infrastructure in our archives (although I think you may find that you'll
> need a separate archive that doesn't correspond to stable or unstable,
> based on having done similar things in the past), but it would be
> premature to put a requirement into Policy until we actually *did* decide
> to do that.  Which would affect a ton of different teams, and would be
> quite a bit of work.

As a practical matter, complex bespoke services are much easier to run
directly out of their vcs trees.

I think a better engineering approach to allowing others to share our
infrastructure code is to ensure that every service we run can
provide, automatically and at runtime, its own source code.

Ian.

-- 
Ian Jackson <ijack...@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Re: Debian built from non-Debian sources

[Russ Allbery]

Steve McIntyre  writes:
> Jonas wrote:

>> It was just an example, however, and my real question was generally
>> what governs code we distribute outside packages - i.e. our install
>> images, if Debian Policy covers only packages.

> Go argue with the Policy folks, I guess.

Speaking as one of the Policy folks

> But a *lot* of the infrastructure we use to run Debian is not exactly
> what's been packaged, as already mentioned. Look at dak. debian-cd,
> live-wrapper et al *are* packaged, but we're not *necessarily* using
> the exact code that's in the stable archive at any point.

...this has been the position of the project for as long as I've been part
of it, and it's not the place of Policy to change that position, only to
reflect project consensus.

I think it would be interesting to strive for making available all Debian
infrastructure in our archives (although I think you may find that you'll
need a separate archive that doesn't correspond to stable or unstable,
based on having done similar things in the past), but it would be
premature to put a requirement into Policy until we actually *did* decide
to do that.  Which would affect a ton of different teams, and would be
quite a bit of work.

-- 
Russ Allbery (r...@debian.org)   

Re: Debian built from non-Debian sources

[Steve McIntyre]

Jonas wrote:
>Quoting Steve McIntyre (2017-07-16 22:14:29)
>> Jonas wrote:
>> >Quoting Andrey Rahmatullin (2017-07-16 19:28:06)
>> >> On Sun, Jul 16, 2017 at 07:05:10PM +0200, Jonas Smedegaard wrote:
>> >> > Is our install images excepmt from our Policy that all dependencies 
>> >> > must 
>> >> > be in Debian, or am I mistaken that we have such Policy?
>> >> Do we?  The Debian Policy covers only debs.
>> >> Also, dak isn't in the archive either.
>> >
>> >I thought Policy covered what we distribute - which excludes dak but 
>> >includes libisofs code embedded in installer images.
>> 
>> Can you identify any code at all from libisofs which is embedded in
>> the images? I'm honestly not aware of any.
>
>I believe the embedded MBR is part of the libisofs project.

No, the MBR is generated from the isolinux/syslinux packages. xorriso
(libisofs) just updates some pointers in there to point at the El
Torito bootable images, to add a fake partition table, etc.

>> I've been using upstream versions of xorriso on-and-off over the last 
>> few years, accepting assistance from the (very helpful!) upstream 
>> author Thomas Schmitt when trying to debug thorny issues like hybrid 
>> BIOS/UEFI booting.
>> 
>> The exact behaviour that we've worked out in some cases has needed 
>> upstream tweaks, and we've needed those tweaks from time to time in 
>> what we've released.
>
>No doubt the particular version of libisofs was used for good reasons.
>
>My concern is the ability to replicate and derive the least possible 
>from Debian resources like the install images.

ACK, understood.

>Concretely The Debian derivative PureOS is having trouble booting their 
>homemade live image on some hardware, but boots fine on both Debian 
>netinst image and Debian live image.  Looking at the properly working 
>images I noticed that the live image for stable was produced using 
>newer-than-stable libisofs,

Sorry, wrong. It was built using the standard xorriso and libisofs
versions in stretch. From the stretch-based VM used to build it:

root@STRETCH-debian-live-builder:~# dpkg -s xorriso | grep Version
Version: 1.4.6-1+b1
root@STRETCH-debian-live-builder:~# dpkg -s libisofs6 | grep Version
Version: 1.4.6-1

If the PureOS folks are having problems, they could ask on the
debian-cd list?

>and that the stable netinst image was produced using a
>never-in-Debian release of libisofs.

Right, I'll give you that one. But there's *seriously* nothing special
there any more than what you'd find in any backport to jessie.

>A related issue is bug#807168.
>
>> >Do we have any Policy on installer images?  If e.g. our netinst or 
>> >live images contain non-DFSG-but-free-to-distribute code would that 
>> >be only a wishlist bug, not a Policy violation?
>> 
>> That would be a serious bug IMHO - please raise any as you find them.
>
>Thanks for clarifying the severity of such theoretic bug specifically.
>
>It was just an example, however, and my real question was generally what 
>governs code we distribute outside packages - i.e. our install images, 
>if Debian Policy covers only packages.

Go argue with the Policy folks, I guess.

But a *lot* of the infrastructure we use to run Debian is not exactly
what's been packaged, as already mentioned. Look at dak. debian-cd,
live-wrapper et al *are* packaged, but we're not *necessarily* using
the exact code that's in the stable archive at any point. We're
typically using code from git on the build machines, to allow for more
flexibility in terms of changes to build scripts as problems arise. We
release things to the archive periodically as a convenience to users,
but serious use often necessitates using git too. This isn't going to
change any time soon.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"Because heaters aren't purple!" -- Catherine Pitt

Re: Debian built from non-Debian sources

[Christian Seiler]

On 07/16/2017 11:12 PM, Jonas Smedegaard wrote:
> It was just an example, however, and my real question was generally what 
> governs code we distribute outside packages - i.e. our install images, 
> if Debian Policy covers only packages.

I don't know if this is actually in Policy or not, but in my opinion
any tool used to create any official Debian images should also be
part of Debian itself - for precisely the reasons you were mentioning
in your email: downstreams should be able to reproduce Debian images.
In the short term only in functionality and not bit-by-bit, but I
would consider reproducible image builds a worthwhile long-term goal
after fully reproducible package builds throughout the archive have
been achieved.

Actually, I'm extremely surprised that the tools used to create the
official Debian images are not part of the corresponding Debian
stable version. I don't doubt there's a good reason for using these
specific versions of those tools, but in that case they should also
be packaged for the release in question.

Regards,
Christian

Re: Debian built from non-Debian sources

[Jonas Smedegaard]

Quoting Steve McIntyre (2017-07-16 22:14:29)
> Jonas wrote:
> >Quoting Andrey Rahmatullin (2017-07-16 19:28:06)
> >> On Sun, Jul 16, 2017 at 07:05:10PM +0200, Jonas Smedegaard wrote:
> >> > Is our install images excepmt from our Policy that all dependencies must 
> >> > be in Debian, or am I mistaken that we have such Policy?
> >> Do we?  The Debian Policy covers only debs.
> >> Also, dak isn't in the archive either.
> >
> >I thought Policy covered what we distribute - which excludes dak but 
> >includes libisofs code embedded in installer images.
> 
> Can you identify any code at all from libisofs which is embedded in
> the images? I'm honestly not aware of any.

I believe the embedded MBR is part of the libisofs project.


> I've been using upstream versions of xorriso on-and-off over the last 
> few years, accepting assistance from the (very helpful!) upstream 
> author Thomas Schmitt when trying to debug thorny issues like hybrid 
> BIOS/UEFI booting.
> 
> The exact behaviour that we've worked out in some cases has needed 
> upstream tweaks, and we've needed those tweaks from time to time in 
> what we've released.

No doubt the particular version of libisofs was used for good reasons.

My concern is the ability to replicate and derive the least possible 
from Debian resources like the install images.

Concretely The Debian derivative PureOS is having trouble booting their 
homemade live image on some hardware, but boots fine on both Debian 
netinst image and Debian live image.  Looking at the properly working 
images I noticed that the live image for stable was produced using 
newer-than-stable libisofs, and that the stable netinst image was 
produced using a never-in-Debian release of libisofs.

A related issue is bug#807168.


> >Do we have any Policy on installer images?  If e.g. our netinst or 
> >live images contain non-DFSG-but-free-to-distribute code would that 
> >be only a wishlist bug, not a Policy violation?
> 
> That would be a serious bug IMHO - please raise any as you find them.

Thanks for clarifying the severity of such theoretic bug specifically.

It was just an example, however, and my real question was generally what 
governs code we distribute outside packages - i.e. our install images, 
if Debian Policy covers only packages.


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: signature

Re: Debian built from non-Debian sources

[Steve McIntyre]

Jonas wrote:
>Quoting Andrey Rahmatullin (2017-07-16 19:28:06)
>> On Sun, Jul 16, 2017 at 07:05:10PM +0200, Jonas Smedegaard wrote:
>> > Is our install images excepmt from our Policy that all dependencies must 
>> > be in Debian, or am I mistaken that we have such Policy?
>> Do we?  The Debian Policy covers only debs.
>> Also, dak isn't in the archive either.
>
>I thought Policy covered what we distribute - which excludes dak but 
>includes libisofs code embedded in installer images.

Can you identify any code at all from libisofs which is embedded in
the images? I'm honestly not aware of any.

I've been using upstream versions of xorriso on-and-off over the last
few years, accepting assistance from the (very helpful!) upstream
author Thomas Schmitt when trying to debug thorny issues like hybrid
BIOS/UEFI booting.

The exact behaviour that we've worked out in some cases has needed
upstream tweaks, and we've needed those tweaks from time to time in
what we've released.

>Do we have any Policy on installer images?  If e.g. our netinst or live 
>images contain non-DFSG-but-free-to-distribute code would that be only a 
>wishlist bug, not a Policy violation?

That would be a serious bug IMHO - please raise any as you find them.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"Because heaters aren't purple!" -- Catherine Pitt

Re: Debian built from non-Debian sources

[Andrey Rahmatullin]

On Sun, Jul 16, 2017 at 08:12:55PM +0200, Jonas Smedegaard wrote:
> > > Is our install images excepmt from our Policy that all dependencies must 
> > > be in Debian, or am I mistaken that we have such Policy?
> > Do we?  The Debian Policy covers only debs.
> > Also, dak isn't in the archive either.
> 
> I thought Policy covered what we distribute - which excludes dak but 
> includes libisofs code embedded in installer images.
https://www.debian.org/doc/debian-policy/ch-archive.html#s-sections
Note that it speaks about the archive.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Debian built from non-Debian sources

[Jonas Smedegaard]

Quoting Andrey Rahmatullin (2017-07-16 19:28:06)
> On Sun, Jul 16, 2017 at 07:05:10PM +0200, Jonas Smedegaard wrote:
> > Is our install images excepmt from our Policy that all dependencies must 
> > be in Debian, or am I mistaken that we have such Policy?
> Do we?  The Debian Policy covers only debs.
> Also, dak isn't in the archive either.

I thought Policy covered what we distribute - which excludes dak but 
includes libisofs code embedded in installer images.

Do we have any Policy on installer images?  If e.g. our netinst or live 
images contain non-DFSG-but-free-to-distribute code would that be only a 
wishlist bug, not a Policy violation?


 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/

 [x] quote me freely  [ ] ask before reusing  [ ] keep private


signature.asc
Description: signature

Re: Debian built from non-Debian sources

[Andrey Rahmatullin]

On Sun, Jul 16, 2017 at 07:05:10PM +0200, Jonas Smedegaard wrote:
> Is our install images excepmt from our Policy that all dependencies must 
> be in Debian, or am I mistaken that we have such Policy?
Do we?  The Debian Policy covers only debs.
Also, dak isn't in the archive either.

-- 
WBR, wRAR


signature.asc
Description: PGP signature