Re: Do the file timestamps in ISOs matter ?

2016-09-12 Thread Thomas Schmitt
Hi,

by the narrow margin of one vote, the proposal "set_to_mtime" wins.
Committed as http://libburnia-project.org/changeset/5757

man 1 xorrisofs of the next release will state:

  SOURCE_DATE_EPOCH  belongs to the specs of reproducible-builds.org.  It
  is supposed to be either undefined or to contain a decimal number which
  tells the seconds since january 1st 1970. If it contains a number, then
  it is used as time value to set the default of --modification-date= and
  --gpt_disk_guid.The   default   of   --set_all_file_dates  is  then
  "set_to_mtime".
  Startup  files  and  program  options  can  override  the   effect   of
  SOURCE_DATE_EPOCH.

This refers to the description of two new options:

  --gpt_disk_guid value
  Control  whether  an emerging GPT shall get a randomly generated
  disk GUID or whether the GUID is supplied by  the  user.   Value
  "random"  is  default.  Value "modification-date" produces a low
  quality GUID from the value set by option --modification-date=.
  [...]

  --set_all_file_dates timestring
  [...]
  If the timestring is "set_to_mtime", then the atime and ctime of
  each file and directory get set to  the  value  found  in  their
  mtime.
  [...]


I am now beginning final regression tests for the release of xorriso-1.4.6.


Have a nice day :)

Thomas


___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds


Re: Do the file timestamps in ISOs matter ?

2016-09-02 Thread Thomas Schmitt
Hi,

i wrote:
> > All three possible behaviors lead to reproducibility if the
> > input trees of the ISO production runs are sufficiently
> > similar.

Sam Hartman wrote:
> So, are you using a definition of reproducible different than the
> resulting iso will have the same SHA-1 hash?

I mean identical results. Bit by bit.

The question to decide is:
What is considered a sufficiently similar input tree in respect to
timestamps ?

Or in the context of ISO producers:
How much flattening (non-injectivity) is tolerable. How much fidelity
(non-aliasing) is needed ?

The answer depends on the expectations of the programs inside the ISO
or expectations of programs and informations which get imported from
elsewhere by the programs in the ISO. (The expectations of already
installed software on the target system are restricted to those of
boot firmware, i assume.)

For example, assume that "make" is supposed to act on ISO content.
In that case, high timestamp fidelity would be needed.


Have a nice day :)

Thomas


___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds


Re: Do the file timestamps in ISOs matter ?

2016-09-02 Thread Thomas Schmitt
Hi,

Chris Lamb wrote:
> I just don't see this usecase of being "partly" reproducible being remotely
> useful to anyone, ever. I'm probably misunderstanding something, however.

All three possible behaviors lead to reproducibility if the input trees
of the ISO production runs are sufficiently similar.

The undecided question is, as the subject says, how much similarity is
most likely needed. The more - the harder to reproduce.

- If ctime and/or atime of files in the input tree matter, then they
  should not be overridden by SOURCE_DATE_EPOCH.
  I.e.:  --set_all_file_dates  "default"

  Reproducibility would break quite easily unless the input filesystem
  is read-only.

- If mtime stamps of input tree matter, then they should not be overridden
  by SOURCE_DATE_EPOCH.
  I.e.:  --set_all_file_dates  "set_to_mtime"

  Reproducibility would survive input tree operations like cp -a.

- File timestamps in the ISO do not matter and rather are just a potential
  obstacle for reproducibility, then SOURCE_DATE_EPOCH should override them.
  I.e.:  --set_all_file_dates  ="$SOURCE_DATE_EPOCH"

  Reproducibility would survive any operation that preserves name tree,
  ownership, permissions, and file content.
  (Ownership and permissions could be flattened by xorrisofs option -r.)

The latter two are what i would consider reasonable for general purposes.
I myself with my purpose of repacking existing ISOs and checking for
regressions could even live with the first one.


> Therefore folding it all into inheriting {a,c}time from mtime iff S_D_E
> is the logical conclusion,

If "iff" means "if and only if", then i have to contradict:
SOURCE_DATE_EPOCH is not the only way to achieve mtime to ctime,atime copying.
That's a consequence of how i implement it.
So it's only "if", not "iff".


> there is need to for people who "just"
> want a reproducible ISO to tediously scour the documentation to discover
> they need to set --set_all_file_dates.

No. SOURCE_DATE_EPOCH makes the ISO reproducible if it does not get
overriden by non-constant parameters of the program options which one
would have to look for (after man page section ENVIRONMENT listed them
as explanation of SOURCE_DATE_EPOCH).

Another precondition is that the input trees are sufficiently similar,
of course.

The proposal to set all file timestamps to SOURCE_DATE_EPOCH would actually
"just" ease the production of reproducible ISOs.


> > grub-mkrescue
> They won't be run with SOURCE_DATE_EPOCH exported.

When xorriso-1.4.6 is out i plan to ask grub-devel to introduce an option
in grub-mkrescue which overrides the "now" time when the argument for
xorrisofs option --modification-date= is determined.
Maybe i can talk them into deducing the --modification-date= string from
SOURCE_DATE_EPOCH.
(If "set_to_mtime" wins the election, then the actual time value of
 SOURCE_DATE_EPOCH will get overridden by --modification-date= completely.)


Have a nice day :)

Thomas


___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds


Re: Do the file timestamps in ISOs matter ?

2016-09-02 Thread Chris Lamb

> > I believe [--set_all_file_dates] is entirely unnecessary
> 
> if SOURCE_DATE_EPOCH sets the default, there must be some option to set
> non-default values, especially the value which is default without
> SOURCE_DATE_EPOCH.

Consider the state space:

If someone does not set SOURCE_DATE_EPOCH, they are going to get an
unreproducible regardless. Therefore, whichever set_all_file_dates option
they choose is entirely meaningless and thus xorriso can simply inherit
{a,c,m}time just as before.

If someone does set SOURCE_DATE_EPOCH, they want a reproducible image by
definition. Thus, inheriting {a,c}time from the filesystem makes no sense
as their image will then not be reproducible and setting {a,c,m}time to a
specific date is not necessary as that can be trivially done pre-build.

Therefore folding it all into inheriting {a,c}time from mtime iff S_D_E
is the logical conclusion, from the angles of complicating xorriso itself
and complicating the end-user UI for someone who wants a reproducible ISO.

Putting it another way, the "semi-reproducible" scenario seems so unlikely
we can entirely discount it and thus there is need to for people who "just"
want a reproducible ISO to tediously scour the documentation to discover they
need to set --set_all_file_dates.

> I decided to let SOURCE_DATE_EPOCH only set the defaults of existing
> and new options, in order to protect ISO producers like grub-mkrescue from
> negative effects by overriding their program options.

They won't be run with SOURCE_DATE_EPOCH exported.

> Not to forget that it offers those users a workaround, who do not agree
> with the final decision about the question posed here, but want to use
> SOURCE_DATE_EPOCH nevertheless.

I just don't see this usecase of being "partly" reproducible being remotely
useful to anyone, ever. I'm probably misunderstanding something, however.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds


Re: Do the file timestamps in ISOs matter ?

2016-09-02 Thread Thomas Schmitt
Hi,

i wrote:
> > new program option:  --set_all_file_dates [...]

Chris Lamb wrote:
> I believe this option is entirely unnecessary. No need to
> complicate things.

It is architecturally needed, because if SOURCE_DATE_EPOCH sets the
default, there must be some option to set non-default values, especially
the value which is default without SOURCE_DATE_EPOCH.

I decided to let SOURCE_DATE_EPOCH only set the defaults of existing
and new options, in order to protect ISO producers like grub-mkrescue from
negative effects by overriding their program options.
(grub-mkrescue uses --modification-date= to set the text in ISO 9660 field
 Volume Modification Date, so that GRUB command
   search --fs-uuid --set -MM-DD-hh-mm-ss-cc
 can look it up.)

Not to forget that it offers those users a workaround, who do not agree
with the final decision about the question posed here, but want to use
SOURCE_DATE_EPOCH nevertheless.


Have a nice day :)

Thomas


___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds


Re: Do the file timestamps in ISOs matter ?

2016-09-01 Thread Chris Lamb
Hi Thomas,

> Still undecided is whether [SOURCE_DATE_EPOCH] shall by default override
> all timestamps inside the ISO

I believe this steps outside the scope of what exporting SOURCE_DATE_EPOCH
should do, both philosophically, æsthetically, and practically as it
prevents ISO producers from having differing mtimes in files.

It's easy enough to clamp (or simply override) the mtimes prior to calling
xorriso anyway…


> [SOURCE_DATE_EPOCH] shall only copy each file's mtime to the same file's
> ctime and atime (to ease reproduction of input file trees).

+1 for this.


> xorriso will support all three models by allowing to override 
> SOURCE_DATE_EPOCH
> by one of three parameters to a new program option:
>   --set_all_file_dates  default
>   --set_all_file_dates  set_to_mtime
>   --set_all_file_dates  ="$SOURCE_DATE_EPOCH"

Given that there is no believable use-case where one would be exporting
SOURCE_DATE_EPOCH but somehow desire the atime and ctime to vary but have a
fixed mtime, I believe this option is entirely unnecessary. No need to
complicate things.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds