Bug#894476: RCC: provide option to encode EPOCH timestamp

[Chris Lamb]

Hi Thomas,

> Maybe the solution is then not in rcc but in whatever generate the files 
> that the qrc that rcc processes refer to. For instance in the case of 
> ultracopier lrelease could have a mean if generating .qm files with the 
> same modified timestamp as the .ts file it processes

That sounds workable..


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#894476: RCC: provide option to encode EPOCH timestamp

[Thomas Preud'homme]

On April 3, 2018 10:15:42 PM GMT+01:00, "Lisandro Damián Nicanor Pérez Meyer" 
 wrote:
>El mar., 3 de abr. de 2018 16:42, Sune Vuorela 
>escribió:
>
>> On Tuesday, April 3, 2018 9:24:58 PM CEST Chris Lamb wrote:
>> > I'm not /entirely/ sure what the difference is as I'm not in the
>> > Qt/RCC world too often these days (alas !), but why just not use
>> > SOURCE_DATE_EPOCH *iff* it is exported?
>> >
>> > Normal systems simply do not have this envvar set, so there is
>really
>> > no danger of it leaking elsewhere or causing unintended
>side-effects.
>>
>> We don't know how application users are using this.
>>
>> The following is some theoretical example, that I'm quite sure could
>be
>> used
>> in some real world applications:
>>
>> QFileInfo embeddedFile(":/foo/data.xml"); // fallback downloaded in
>the
>> ancient past
>> QFileInfo systemfile("/usr/share/foo/data.xml");
>>
>> if (systemfile.lastModified() > embeddedFile.lastModified())
>> {
>>data = readFromFile(systemFile);
>> }
>> else
>> {
>>data = readFromFile(embeddedFile);
>> }
>>
>> If S_D_E gets used, rather than the data.xml modified date in the
>source,
>> this
>> will end up using the wrong file if S_D_E is newer than the system
>copy of
>> the
>> file.
>>
>> This is not a unused databit, but a fully available piece of metadata
>for
>> the
>> files.
>
>
>I'm afraid Sune is right here, it might be used that way. Questionable?
>Sure thing, but still valid code.
>
>But after all the same can be said from embedding translations into the
>binary itself.
>
>So yes, we will need help from upstream with this one.
>
>>
>>

Maybe the solution is then not in rcc but in whatever generate the files that 
the qrc that rcc processes refer to. For instance in the case of ultracopier 
lrelease could have a mean if generating .qm files with the same modified 
timestamp as the .ts file it processes. This would guarantee stable output for 
a stable source and this later on rcc would also generate stable output.

Thoughts?

Best regards,

Thomas

Bug#894476: RCC: provide option to encode EPOCH timestamp

[Lisandro Damián Nicanor Pérez Meyer]

El mar., 3 de abr. de 2018 16:42, Sune Vuorela  escribió:

> On Tuesday, April 3, 2018 9:24:58 PM CEST Chris Lamb wrote:
> > I'm not /entirely/ sure what the difference is as I'm not in the
> > Qt/RCC world too often these days (alas !), but why just not use
> > SOURCE_DATE_EPOCH *iff* it is exported?
> >
> > Normal systems simply do not have this envvar set, so there is really
> > no danger of it leaking elsewhere or causing unintended side-effects.
>
> We don't know how application users are using this.
>
> The following is some theoretical example, that I'm quite sure could be
> used
> in some real world applications:
>
> QFileInfo embeddedFile(":/foo/data.xml"); // fallback downloaded in the
> ancient past
> QFileInfo systemfile("/usr/share/foo/data.xml");
>
> if (systemfile.lastModified() > embeddedFile.lastModified())
> {
>data = readFromFile(systemFile);
> }
> else
> {
>data = readFromFile(embeddedFile);
> }
>
> If S_D_E gets used, rather than the data.xml modified date in the source,
> this
> will end up using the wrong file if S_D_E is newer than the system copy of
> the
> file.
>
> This is not a unused databit, but a fully available piece of metadata for
> the
> files.


I'm afraid Sune is right here, it might be used that way. Questionable?
Sure thing, but still valid code.

But after all the same can be said from embedding translations into the
binary itself.

So yes, we will need help from upstream with this one.

>
>

Bug#894476: RCC: provide option to encode EPOCH timestamp

[Sune Vuorela]

On Tuesday, April 3, 2018 9:24:58 PM CEST Chris Lamb wrote:
> I'm not /entirely/ sure what the difference is as I'm not in the
> Qt/RCC world too often these days (alas !), but why just not use
> SOURCE_DATE_EPOCH *iff* it is exported?
> 
> Normal systems simply do not have this envvar set, so there is really
> no danger of it leaking elsewhere or causing unintended side-effects.

We don't know how application users are using this.

The following is some theoretical example, that I'm quite sure could be used 
in some real world applications:

QFileInfo embeddedFile(":/foo/data.xml"); // fallback downloaded in the 
ancient past
QFileInfo systemfile("/usr/share/foo/data.xml");

if (systemfile.lastModified() > embeddedFile.lastModified())
{
   data = readFromFile(systemFile);
}
else
{
   data = readFromFile(embeddedFile);
}

If S_D_E gets used, rather than the data.xml modified date in the source, this 
will end up using the wrong file if S_D_E is newer than the system copy of the 
file.

This is not a unused databit, but a fully available piece of metadata for the 
files.

/Sune
-- 
I didn’t stop pretending when I became an adult, it’s just that when I was a 
kid I was pretending that I fit into the rules and structures of this world. 
And now that I’m an adult, I pretend that those rules and structures exist.
   - zefrank

Bug#894476: RCC: provide option to encode EPOCH timestamp

[Thomas Preud'homme]

On April 3, 2018 8:20:22 PM GMT+01:00, Sune Vuorela  wrote:
>On Tuesday, April 3, 2018 9:14:23 PM CEST Chris Lamb wrote:
>> Hi Sune!
>> 
>> > I don't think honouring SOURCE_DATE_EPOCH is the right idea under
>normal
>> > circumstances
>> 
>> Can you elaborate on what you mean by "normal circumstances"? :)
>
>"normal circumstances" is rcc on a source file, as opposed to an
>autogenerated 
>file.
>
>> How about e use S_D_E if it is setup/exported, otherwise use the
>mtime
>> of the file as before?
>
>I think that using S_D_E only makes sense if rcc is run on an
>autogenerated 
>file, but I do think that most rcc runs is run on existing source
>files.
>
>
>I don't have a good idea to differentiate those.
>
>/Sune
>-- 
>I didn’t stop pretending when I became an adult, it’s just that when I
>was a 
>kid I was pretending that I fit into the rules and structures of this
>world. 
>And now that I’m an adult, I pretend that those rules and structures
>exist.
>   - zefrank

One small clarification: in my case rcc *is* run on a nongenerated resource 
file. It's some of the files that the resource file list that are generated and 
whose timestamp end up in the cpp file generated by rcc.

In other hand:

foo.qrc mentions bar.qm which is generated from bar.ts.

rcc foo.qrc generates resource_bar.cpp which contains constant data that 
encodes bar.qm timestamp and this create different resource_bar.o at every 
build.

Best regards,

Thomas

Bug#894476: RCC: provide option to encode EPOCH timestamp

[Chris Lamb]

Hi Sune,

> "normal circumstances" is rcc on a source file, as opposed
> to an autogenerated file

I'm not /entirely/ sure what the difference is as I'm not in the
Qt/RCC world too often these days (alas !), but why just not use
SOURCE_DATE_EPOCH *iff* it is exported?

Normal systems simply do not have this envvar set, so there is really
no danger of it leaking elsewhere or causing unintended side-effects.

It would also have the benefit of not having to differentiate between
the two... :)


Best wishes,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#894476: RCC: provide option to encode EPOCH timestamp

[Sune Vuorela]

On Tuesday, April 3, 2018 9:14:23 PM CEST Chris Lamb wrote:
> Hi Sune!
> 
> > I don't think honouring SOURCE_DATE_EPOCH is the right idea under normal
> > circumstances
> 
> Can you elaborate on what you mean by "normal circumstances"? :)

"normal circumstances" is rcc on a source file, as opposed to an autogenerated 
file.

> How about e use S_D_E if it is setup/exported, otherwise use the mtime
> of the file as before?

I think that using S_D_E only makes sense if rcc is run on an autogenerated 
file, but I do think that most rcc runs is run on existing source files.


I don't have a good idea to differentiate those.

/Sune
-- 
I didn’t stop pretending when I became an adult, it’s just that when I was a 
kid I was pretending that I fit into the rules and structures of this world. 
And now that I’m an adult, I pretend that those rules and structures exist.
   - zefrank

Bug#894476: RCC: provide option to encode EPOCH timestamp

[Chris Lamb]

Hi Sune!

> I don't think honouring SOURCE_DATE_EPOCH is the right idea under normal 
> circumstances

Can you elaborate on what you mean by "normal circumstances"? :)

How about e use S_D_E if it is setup/exported, otherwise use the mtime
of the file as before?

(This is the pattern/logic used extensively elsewhere in toolchains for
this very exact purpose; it would not be unique to rcc)


Best wishes,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#894476: RCC: provide option to encode EPOCH timestamp

[Sune Vuorela]

On Tuesday, April 3, 2018 8:46:51 PM CEST Thomas Preud'homme wrote:
> The problematic files are Qt message files (ie .qm files) generated at build
> time via lrelease from translation files (ie .ts files). Therefore two

Right. I didn't think about autogenerated files during build.

> I think it would be nice if there was a way for
> rcc to avoid doing that manually. I agree with Chris that honouring
> SOURCE_DATE_EPOCH in rcc would be a nice solution.

I don't think honouring SOURCE_DATE_EPOCH is the right idea under normal 
circumstances, given that I do think that having the actual timestamp of the 
non-generated files makes sense.

I have reached out to upstream about it.

/Sune
-- 
I didn’t stop pretending when I became an adult, it’s just that when I was a 
kid I was pretending that I fit into the rules and structures of this world. 
And now that I’m an adult, I pretend that those rules and structures exist.
   - zefrank

Bug#894476: RCC: provide option to encode EPOCH timestamp

[Sune Vuorela]

On Saturday, March 31, 2018 12:05:02 PM CEST Chris Lamb wrote:
> > While investigating ultracopier's lack of build reproducibility, I found
> > out that rcc encodes the timestamp of the files the QRC file being
> > compiled references

I don't actually see why this should be a problem. If input changes, output 
changes.
I do think that using touch(1) on the input should allow different output.

It is quite easy to reproduce:

qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../
qimage.qrc -o 1
qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../
qimage.qrc -o 2
qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../
qimage.qrc -o 3

Gives same output.
Even adding in touch ../qimage.qrc keeps the same output.

qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../
qimage.qrc -o 4

touch ../images/image.bmp

qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version 2 ../
qimage.qrc -o 5

is needed to get different output

14ef6dae8e4992ce907948c1c4af272b  1
14ef6dae8e4992ce907948c1c4af272b  2
14ef6dae8e4992ce907948c1c4af272b  3
14ef6dae8e4992ce907948c1c4af272b  4
54c6f8c09a347955ae2f36e68bbd2539  5


So. What touches the files?

/Sune
-- 
I didn’t stop pretending when I became an adult, it’s just that when I was a 
kid I was pretending that I fit into the rules and structures of this world. 
And now that I’m an adult, I pretend that those rules and structures exist.
   - zefrank

Bug#894476: RCC: provide option to encode EPOCH timestamp

[Thomas Preud'homme]

On April 3, 2018 7:25:03 PM GMT+01:00, Sune Vuorela  wrote:
>On Saturday, March 31, 2018 12:05:02 PM CEST Chris Lamb wrote:
>> > While investigating ultracopier's lack of build reproducibility, I
>found
>> > out that rcc encodes the timestamp of the files the QRC file being
>> > compiled references
>
>I don't actually see why this should be a problem. If input changes,
>output 
>changes.
>I do think that using touch(1) on the input should allow different
>output.
>
>It is quite easy to reproduce:
>
>qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version
>2 ../
>qimage.qrc -o 1
>qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version
>2 ../
>qimage.qrc -o 2
>qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version
>2 ../
>qimage.qrc -o 3
>
>Gives same output.
>Even adding in touch ../qimage.qrc keeps the same output.
>
>qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version
>2 ../
>qimage.qrc -o 4
>
>touch ../images/image.bmp
>
>qtbase/tests/auto/gui/image/qimage/foo (dev $%=)$ rcc --format-version
>2 ../
>qimage.qrc -o 5
>
>is needed to get different output
>
>14ef6dae8e4992ce907948c1c4af272b  1
>14ef6dae8e4992ce907948c1c4af272b  2
>14ef6dae8e4992ce907948c1c4af272b  3
>14ef6dae8e4992ce907948c1c4af272b  4
>54c6f8c09a347955ae2f36e68bbd2539  5
>
>
>So. What touches the files?
>
>/Sune
>-- 
>I didn’t stop pretending when I became an adult, it’s just that when I
>was a 
>kid I was pretending that I fit into the rules and structures of this
>world. 
>And now that I’m an adult, I pretend that those rules and structures
>exist.
>   - zefrank

Hi Sune,

The problematic files are Qt message files (ie .qm files) generated at build 
time via lrelease from translation files (ie .ts files). Therefore two 
different builds will generate those .qm files at different times which will 
end up with different cpp files generated by rcc. Currently I'm working around 
it by setting the modified time of those .qm files to EPOCH after they are 
generated. I think it would be nice if there was a way for rcc to avoid doing 
that manually. I agree with Chris that honouring SOURCE_DATE_EPOCH in rcc would 
be a nice solution.

Best regards,

Thomas

Bug#894476: RCC: provide option to encode EPOCH timestamp

[Lisandro Damián Nicanor Pérez Meyer]

Thanks for the bug! And sorry for the to posting :-(

I'll try to take a look into this.

El sáb., 31 de mar. de 2018 07:09, Chris Lamb  escribió:

> Hi,
>
> > While investigating ultracopier's lack of build reproducibility, I found
> > out that rcc encodes the timestamp of the files the QRC file being
> > compiled references
>
> Indeed, we've been tracking this for a while in the Reproducible Builds
> project [0] but, without a patch, we did not wish to bother you with a
> bug. :)
>
> It is affecting at least 4 or 5 other packages. I had previously tracked
> it down to [1].
>
> Regarding the solution, I think I would actually prefer to see something
> consuming SOURCE_DATE_EPOCH[1] rather than adding an option to set a
> specific timestamp.
>
>   [0] https://reproducible-builds.org/
>   [1]
> https://sources.debian.org/src/qtbase-opensource-src/5.9.2+dfsg-12/src/tools/rcc/rcc.cpp/#L207-L212
>   [2] https://reproducible-builds.org/specs/source-date-epoch/
>
>
> Best wishes,
>
> --
>   ,''`.
>  : :'  : Chris Lamb
>  `. `'`  la...@debian.org / chris-lamb.co.uk
>`-
>
>

Bug#894476: RCC: provide option to encode EPOCH timestamp

[Chris Lamb]

Hi,

> While investigating ultracopier's lack of build reproducibility, I found
> out that rcc encodes the timestamp of the files the QRC file being
> compiled references

Indeed, we've been tracking this for a while in the Reproducible Builds
project [0] but, without a patch, we did not wish to bother you with a
bug. :)

It is affecting at least 4 or 5 other packages. I had previously tracked
it down to [1].

Regarding the solution, I think I would actually prefer to see something
consuming SOURCE_DATE_EPOCH[1] rather than adding an option to set a
specific timestamp.

  [0] https://reproducible-builds.org/
  [1] 
https://sources.debian.org/src/qtbase-opensource-src/5.9.2+dfsg-12/src/tools/rcc/rcc.cpp/#L207-L212
  [2] https://reproducible-builds.org/specs/source-date-epoch/


Best wishes,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

Bug#894476: RCC: provide option to encode EPOCH timestamp

[Thomas Preud'homme]

Package: qtbase5-dev-tools
Version: 5.9.2+dfsg-12
Severity: wishlist
Tags: upstream
User: reproducible-bui...@lists.alioth.debian.org
Usertags: timestamps toolchain

Hi,

While investigating ultracopier's lack of build reproducibility, I found
out that rcc encodes the timestamp of the files the QRC file being
compiled references (see end of RCCFileInfo::writeDataInfo()). This
becomes a problem for generated files because the output of rcc is then
different in 2 different builds. It would be nice if rcc had an option
to encode a stable timestamp, eg. EPOCH.

Best regards,

Thomas

-- System Information:
Debian Release: buster/sid
  APT prefers unstable
  APT policy: (900, 'unstable'), (500, 'unstable-debug'), (500, 
'stable-updates'), (500, 'testing'), (500, 'stable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.15.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), 
LANGUAGE=fr_FR:fr (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages qtbase5-dev-tools depends on:
ii  libc62.27-2
ii  libgcc1  1:8-20180321-1
ii  libqt5core5a [qtbase-abi-5-9-2]  5.9.2+dfsg-12
ii  libqt5dbus5  5.9.2+dfsg-12
ii  libstdc++6   8-20180321-1
ii  perl 5.26.1-5
ii  qtchooser64-ga1b6736-5
ii  zlib1g   1:1.2.8.dfsg-5

qtbase5-dev-tools recommends no packages.

qtbase5-dev-tools suggests no packages.

-- no debconf information