[jira] [Commented] (MSOURCES-120) Reproducible Builds: make entries in output jar files reproducible (order + timestamp)
[ https://issues.apache.org/jira/browse/MSOURCES-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961558#comment-16961558 ] Hudson commented on MSOURCES-120: - Build failed in Jenkins: Maven TLP » maven-source-plugin » master #25 See https://builds.apache.org/job/maven-box/job/maven-source-plugin/job/master/25/ > Reproducible Builds: make entries in output jar files reproducible (order + > timestamp) > -- > > Key: MSOURCES-120 > URL: https://issues.apache.org/jira/browse/MSOURCES-120 > Project: Maven Source Plugin > Issue Type: New Feature >Affects Versions: 3.0.1 >Reporter: Herve Boutemy >Priority: Major > > since a jar file is a zip file, entries order and timestamp are a natural > source of non Reproducible Builds: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MSOURCES-120) Reproducible Builds: make entries in output jar files reproducible (order + timestamp)
[ https://issues.apache.org/jira/browse/MSOURCES-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961239#comment-16961239 ] Hudson commented on MSOURCES-120: - Build failed in Jenkins: Maven TLP » maven-source-plugin » master #24 See https://builds.apache.org/job/maven-box/job/maven-source-plugin/job/master/24/ > Reproducible Builds: make entries in output jar files reproducible (order + > timestamp) > -- > > Key: MSOURCES-120 > URL: https://issues.apache.org/jira/browse/MSOURCES-120 > Project: Maven Source Plugin > Issue Type: New Feature >Affects Versions: 3.0.1 >Reporter: Herve Boutemy >Priority: Major > > since a jar file is a zip file, entries order and timestamp are a natural > source of non Reproducible Builds: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (MSOURCES-120) Reproducible Builds: make entries in output jar files reproducible (order + timestamp)
[ https://issues.apache.org/jira/browse/MSOURCES-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925109#comment-16925109 ] Hudson commented on MSOURCES-120: - Build failed in Jenkins: Maven TLP » maven-source-plugin » MSOURCES-120 #22 See https://builds.apache.org/job/maven-box/job/maven-source-plugin/job/MSOURCES-120/22/ > Reproducible Builds: make entries in output jar files reproducible (order + > timestamp) > -- > > Key: MSOURCES-120 > URL: https://issues.apache.org/jira/browse/MSOURCES-120 > Project: Maven Source Plugin > Issue Type: New Feature >Affects Versions: 3.0.1 >Reporter: Hervé Boutemy >Priority: Major > > since a jar file is a zip file, entries order and timestamp are a natural > source of non Reproducible Builds: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (MSOURCES-120) Reproducible Builds: make entries in output jar files reproducible (order + timestamp)
[ https://issues.apache.org/jira/browse/MSOURCES-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925108#comment-16925108 ] Hervé Boutemy commented on MSOURCES-120: PR 121 created https://github.com/codehaus-plexus/plexus-archiver/pull/121 > Reproducible Builds: make entries in output jar files reproducible (order + > timestamp) > -- > > Key: MSOURCES-120 > URL: https://issues.apache.org/jira/browse/MSOURCES-120 > Project: Maven Source Plugin > Issue Type: New Feature >Affects Versions: 3.0.1 >Reporter: Hervé Boutemy >Priority: Major > > since a jar file is a zip file, entries order and timestamp are a natural > source of non Reproducible Builds: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (MSOURCES-120) Reproducible Builds: make entries in output jar files reproducible (order + timestamp)
[
https://issues.apache.org/jira/browse/MSOURCES-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924814#comment-16924814
]
Hervé Boutemy commented on MSOURCES-120:
{quote}I think would be nice if the UTC logic is in Plexus Archiver so it can
be used by other plugins as well.{quote}
sure, I intend to test on Maven Assembly Plugin before doing releases, to check
the logic against other archive formats, like tar, to be sure to get adapted
API to plexus-archiver
{quote}I guess right now you just testing in the source plugin{quote}
Yes, I discovered in the past that working on Reproducible Builds by fixing
every issue separately, leading to plugins releases, just does not work: it's
when you run the plugin that you discover which is the next issue to fix.
This time, I'm doing a PoC at plugin level before dispatching code where it
will finally belong
> Reproducible Builds: make entries in output jar files reproducible (order +
> timestamp)
> --
>
> Key: MSOURCES-120
> URL: https://issues.apache.org/jira/browse/MSOURCES-120
> Project: Maven Source Plugin
> Issue Type: New Feature
>Affects Versions: 3.0.1
>Reporter: Hervé Boutemy
>Priority: Major
>
> since a jar file is a zip file, entries order and timestamp are a natural
> source of non Reproducible Builds:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (MSOURCES-120) Reproducible Builds: make entries in output jar files reproducible (order + timestamp)
[ https://issues.apache.org/jira/browse/MSOURCES-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924785#comment-16924785 ] Michael Osipov commented on MSOURCES-120: - I agree with Plamen, Plexus Archiver shall receive a timezone object per archive. > Reproducible Builds: make entries in output jar files reproducible (order + > timestamp) > -- > > Key: MSOURCES-120 > URL: https://issues.apache.org/jira/browse/MSOURCES-120 > Project: Maven Source Plugin > Issue Type: New Feature >Affects Versions: 3.0.1 >Reporter: Hervé Boutemy >Priority: Major > > since a jar file is a zip file, entries order and timestamp are a natural > source of non Reproducible Builds: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (MSOURCES-120) Reproducible Builds: make entries in output jar files reproducible (order + timestamp)
[ https://issues.apache.org/jira/browse/MSOURCES-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924761#comment-16924761 ] Plamen Totev commented on MSOURCES-120: --- [~hboutemy] I think would be nice if the UTC logic is in Plexus Archiver so it can be used by other plugins as well. I guess right now you just testing in the source plugin, but wanted to mention it just in case. > Reproducible Builds: make entries in output jar files reproducible (order + > timestamp) > -- > > Key: MSOURCES-120 > URL: https://issues.apache.org/jira/browse/MSOURCES-120 > Project: Maven Source Plugin > Issue Type: New Feature >Affects Versions: 3.0.1 >Reporter: Hervé Boutemy >Priority: Major > > since a jar file is a zip file, entries order and timestamp are a natural > source of non Reproducible Builds: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (MSOURCES-120) Reproducible Builds: make entries in output jar files reproducible (order + timestamp)
[ https://issues.apache.org/jira/browse/MSOURCES-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924736#comment-16924736 ] Hudson commented on MSOURCES-120: - Build failed in Jenkins: Maven TLP » maven-source-plugin » MSOURCES-120 #21 See https://builds.apache.org/job/maven-box/job/maven-source-plugin/job/MSOURCES-120/21/ > Reproducible Builds: make entries in output jar files reproducible (order + > timestamp) > -- > > Key: MSOURCES-120 > URL: https://issues.apache.org/jira/browse/MSOURCES-120 > Project: Maven Source Plugin > Issue Type: New Feature >Affects Versions: 3.0.1 >Reporter: Hervé Boutemy >Priority: Major > > since a jar file is a zip file, entries order and timestamp are a natural > source of non Reproducible Builds: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (MSOURCES-120) Reproducible Builds: make entries in output jar files reproducible (order + timestamp)
[ https://issues.apache.org/jira/browse/MSOURCES-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924630#comment-16924630 ] Hudson commented on MSOURCES-120: - Build failed in Jenkins: Maven TLP » maven-source-plugin » MSOURCES-120 #19 See https://builds.apache.org/job/maven-box/job/maven-source-plugin/job/MSOURCES-120/19/ > Reproducible Builds: make entries in output jar files reproducible (order + > timestamp) > -- > > Key: MSOURCES-120 > URL: https://issues.apache.org/jira/browse/MSOURCES-120 > Project: Maven Source Plugin > Issue Type: New Feature >Affects Versions: 3.0.1 >Reporter: Hervé Boutemy >Priority: Major > > since a jar file is a zip file, entries order and timestamp are a natural > source of non Reproducible Builds: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (MSOURCES-120) Reproducible Builds: make entries in output jar files reproducible (order + timestamp)
[ https://issues.apache.org/jira/browse/MSOURCES-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923947#comment-16923947 ] Hervé Boutemy commented on MSOURCES-120: sure, serializing UTC value is what I had in mind when describing in a more complex way "trick the Java timestamp using local timezone" :) defining the extra time field could be a nice addition also > Reproducible Builds: make entries in output jar files reproducible (order + > timestamp) > -- > > Key: MSOURCES-120 > URL: https://issues.apache.org/jira/browse/MSOURCES-120 > Project: Maven Source Plugin > Issue Type: New Feature >Affects Versions: 3.0.1 >Reporter: Hervé Boutemy >Priority: Major > > since a jar file is a zip file, entries order and timestamp are a natural > source of non Reproducible Builds: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (MSOURCES-120) Reproducible Builds: make entries in output jar files reproducible (order + timestamp)
[ https://issues.apache.org/jira/browse/MSOURCES-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923718#comment-16923718 ] Michael Osipov commented on MSOURCES-120: - As far as I understand the format, there is no timezone notion. I assume that we have to provide two options: local timezone and UTC. [~hboutemy], why not serialize UTC value? > Reproducible Builds: make entries in output jar files reproducible (order + > timestamp) > -- > > Key: MSOURCES-120 > URL: https://issues.apache.org/jira/browse/MSOURCES-120 > Project: Maven Source Plugin > Issue Type: New Feature >Affects Versions: 3.0.1 >Reporter: Hervé Boutemy >Priority: Major > > since a jar file is a zip file, entries order and timestamp are a natural > source of non Reproducible Builds: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (MSOURCES-120) Reproducible Builds: make entries in output jar files reproducible (order + timestamp)
[
https://issues.apache.org/jira/browse/MSOURCES-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923638#comment-16923638
]
Hervé Boutemy commented on MSOURCES-120:
thank you for the hint: I added some details to the output to confirm/show:
on my machine, with Paris timezone UTC+2:
{noformat}reproducible-1.0-sources.jar sha1 =
7bd062f9b45afb7423f3c004c8080ebad4e98754
encoding: UTF8
timezone offset (minutes): -120
M size (cmp) crc java time date time zip time mode name
-comment; extra
8 25 ( 27) ee027fb2 1566419332000 2019-08-21 22:28:52 1326822298 100644
META-INF/MANIFEST.MF ; 0{noformat}
on ASF CI server, with UTC configuration:
{noformat}reproducible-1.0-sources.jar sha1 =
acf461ff37ddc3c44c620770a73d9cf42f7ca429
encoding: UTF8
timezone offset (minutes): 0
M size (cmp) crc java time date time zip time mode name
-comment; extra
8 25 ( 27) ee027fb2 1566419332000 2019-08-21 20:28:52 1326818202 100644
META-INF/MANIFEST.MF ; 0{noformat}
as you can see from the "zip time field", which is the long value that is
really stored in the zip stream, we get 1326822298 for a machine in UTC+2 but
1326818202 for a machine in UTC
now, I'll update the code to trick the Java timestamp using local timezone:
this will give unexpected different Dates in Java (should I say incorrect?),
but from a zip stream perspective, we'll get the same value on every timezone
Zip dates are tricky :)
> Reproducible Builds: make entries in output jar files reproducible (order +
> timestamp)
> --
>
> Key: MSOURCES-120
> URL: https://issues.apache.org/jira/browse/MSOURCES-120
> Project: Maven Source Plugin
> Issue Type: New Feature
>Affects Versions: 3.0.1
>Reporter: Hervé Boutemy
>Priority: Major
>
> since a jar file is a zip file, entries order and timestamp are a natural
> source of non Reproducible Builds:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (MSOURCES-120) Reproducible Builds: make entries in output jar files reproducible (order + timestamp)
[
https://issues.apache.org/jira/browse/MSOURCES-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921017#comment-16921017
]
Plamen Totev commented on MSOURCES-120:
---
I think I found the root cause. In order to set the Zip entry timestamp Plexus
Archiver calls \{{java.util.zip.ZipEntry#setTime}}. If you follow the method
logic you'll see it actually calls {{java.util.zip.ZipUtils#javaToDosTime}}
{code:java}
Instant instant = Instant.ofEpochMilli(time);
LocalDateTime ldt = LocalDateTime.ofInstant(
instant, ZoneId.systemDefault());
{code}
So actually the resulting ZIP archive depends on the build machine time zone.
> Reproducible Builds: make entries in output jar files reproducible (order +
> timestamp)
> --
>
> Key: MSOURCES-120
> URL: https://issues.apache.org/jira/browse/MSOURCES-120
> Project: Maven Source Plugin
> Issue Type: New Feature
>Affects Versions: 3.0.1
>Reporter: Hervé Boutemy
>Priority: Major
>
> since a jar file is a zip file, entries order and timestamp are a natural
> source of non Reproducible Builds:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (MSOURCES-120) Reproducible Builds: make entries in output jar files reproducible (order + timestamp)
[
https://issues.apache.org/jira/browse/MSOURCES-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920627#comment-16920627
]
Plamen Totev commented on MSOURCES-120:
---
I've tried to build {{a734f2739b4d13c3dab437a2c8f38607e8c325f5}} locally and it
failed. Here is the debug output if it helps:
{code:java}
reproducible-1.0-sources.jar sha1 = acf461ff37ddc3c44c620770a73d9cf42f7ca429
encoding: UTF8
M
size (cmp) crc time mode name -comment; extra
8
25 ( 27) ee027fb2 1566419332000 100644 META-INF/MANIFEST.MF ; 0
00
( 0)0 1566419332000 40755 META-INF/ ; 0
00 (
0)0 1566419332000 40755 dir-A/ ; 0
00 ( 0)
0 1566419332000 40755 dir-C/ ; 0
00 ( 0)
0 1566419332000 40755 dir-b/ ; 0
00 ( 0)
0 1566419332000 40755 dir-b/B2/ ; 0
00 ( 0)0
1566419332000 40755 dir-b/B4/ ; 0
00 ( 0)0
1566419332000 40755 dir-b/b1/ ; 0
00 ( 0)0
1566419332000 40755 dir-b/b3/ ; 0
00 ( 0)0
1566419332000 40755 dir-d/ ; 0
00 ( 0)0
1566419332000 40755 META-INF/maven/ ; 0
00 ( 0)0
1566419332000 40755 META-INF/maven/org.apache.maven.its/ ; 0
00 ( 0)0
1566419332000 40755 META-INF/maven/org.apache.maven.its/reproducible/ ; 0
8 788 (453) 598eee6e
1566419332000 100644 Uppercase.txt ; 0
8 788 (453) 598eee6e
1566419332000 100644 dir-A/A2.txt ; 0
8 788 (453) 598eee6e
1566419332000 100644
[jira] [Commented] (MSOURCES-120) Reproducible Builds: make entries in output jar files reproducible (order + timestamp)
[ https://issues.apache.org/jira/browse/MSOURCES-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915690#comment-16915690 ] Hudson commented on MSOURCES-120: - Build failed in Jenkins: Maven TLP » maven-source-plugin » MSOURCES-120 #3 See https://builds.apache.org/job/maven-box/job/maven-source-plugin/job/MSOURCES-120/3/ > Reproducible Builds: make entries in output jar files reproducible (order + > timestamp) > -- > > Key: MSOURCES-120 > URL: https://issues.apache.org/jira/browse/MSOURCES-120 > Project: Maven Source Plugin > Issue Type: New Feature >Affects Versions: 3.0.1 >Reporter: Hervé Boutemy >Priority: Major > > since a jar file is a zip file, entries order and timestamp are a natural > source of non Reproducible Builds: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (MSOURCES-120) Reproducible Builds: make entries in output jar files reproducible (order + timestamp)
[ https://issues.apache.org/jira/browse/MSOURCES-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913231#comment-16913231 ] Hudson commented on MSOURCES-120: - Build failed in Jenkins: Maven TLP » maven-source-plugin » MSOURCES-120 #2 See https://builds.apache.org/job/maven-box/job/maven-source-plugin/job/MSOURCES-120/2/ > Reproducible Builds: make entries in output jar files reproducible (order + > timestamp) > -- > > Key: MSOURCES-120 > URL: https://issues.apache.org/jira/browse/MSOURCES-120 > Project: Maven Source Plugin > Issue Type: New Feature >Affects Versions: 3.0.1 >Reporter: Hervé Boutemy >Priority: Major > > since a jar file is a zip file, entries order and timestamp are a natural > source of non Reproducible Builds: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318 -- This message was sent by Atlassian Jira (v8.3.2#803003)
