adoroszlai commented on PR #6856: URL: https://github.com/apache/ozone/pull/6856#issuecomment-2408966860
Thanks @jojochuang for the questions. > Have we verified it does create reproducible builds? Yes. The PR adds a new CI check (`repro`) to rebuild Ozone and compare artifacts against the reference created by the `build` check. Comparison is performed by Maven Artifact Plugin's `compare` goal, which reports differences like: ``` [ERROR] size mismatch hdds-common-2.0.0-SNAPSHOT.jar: investigate with diffoscope hadoop-hdds/common/target/reference/org.apache.ozone/hdds-common-2.0.0-SNAPSHOT.jar hadoop-hdds/common/target/hdds-common-2.0.0-SNAPSHOT.jar [ERROR] Reproducible Build output summary: 6 files ok, 1 different [ERROR] see diff hadoop-hdds/common/target/reference/hdds-common-2.0.0-SNAPSHOT.buildinfo hadoop-hdds/common/target/hdds-common-2.0.0-SNAPSHOT.buildinfo [ERROR] see also https://maven.apache.org/guides/mini/guide-reproducible-builds.html ``` When looking this up to answer your question, I just found that `repro` check was passing despite the error. This is now fixed, check is [failing properly](https://github.com/adoroszlai/ozone/actions/runs/11312548693/job/31460267951) when mismatch is found. The check also has a follow-up step to display the root cause of the difference, using `diffoscope`. [Example](https://github.com/adoroszlai/ozone/actions/runs/11312548693/job/31460267951#step:10:12): ``` --- /home/runner/.m2/repository/org/apache/ozone/hdds-common/2.0.0-SNAPSHOT/hdds-common-2.0.0-SNAPSHOT.jar +++ hadoop-hdds/common/target/hdds-common-2.0.0-SNAPSHOT.jar ├── zipinfo /dev/stdin │ @@ -1,8 +1,8 @@ │ -Zip file size: 705376 bytes, number of entries: 430 │ +Zip file size: 705373 bytes, number of entries: 430 │ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 META-INF/ │ -rw-r--r-- 2.0 unx 81 b- defN 23-Jan-01 00:00 META-INF/MANIFEST.MF │ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 com/ │ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 com/google/ │ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 com/google/protobuf/ │ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 org/ │ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 org/apache/ │ @@ -68,15 +68,15 @@ │ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 META-INF/maven/ │ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 META-INF/maven/org.apache.ozone/ │ drwxr-xr-x 2.0 unx 0 b- stor 23-Jan-01 00:00 META-INF/maven/org.apache.ozone/hdds-common/ │ -rw-r--r-- 2.0 unx [12](https://github.com/adoroszlai/ozone/actions/runs/11312548693/job/31460267951#step:10:13)660 b- defN 23-Jan-01 00:00 META-INF/LICENSE.txt │ -rw-r--r-- 2.0 unx 1493 b- defN 23-Jan-01 00:00 META-INF/NOTICE.txt │ -rw-r--r-- 2.0 unx 628 b- defN 23-Jan-01 00:00 com/google/protobuf/Proto2Utils.class │ -rw-r--r-- 2.0 unx 3847 b- defN 23-Jan-01 00:00 hdds-common.classpath │ --rw-r--r-- 2.0 unx 1089 b- defN 23-Jan-01 00:00 hdds-version-info.properties │ +-rw-r--r-- 2.0 unx 1078 b- defN 23-Jan-01 00:00 hdds-version-info.properties │ -rw-r--r-- 2.0 unx 3176 b- defN 23-Jan-01 00:00 network-topology-default.xml │ -rw-r--r-- 2.0 unx 1689 b- defN 23-Jan-01 00:00 network-topology-default.yaml │ -rw-r--r-- 2.0 unx 3380 b- defN 23-Jan-01 00:00 network-topology-nodegroup.xml │ -rw-r--r-- 2.0 unx 210 b- defN 23-Jan-01 00:00 org/apache/hadoop/hdds/ComponentVersion.class │ -rw-r--r-- 2.0 unx 2076 b- defN 23-Jan-01 00:00 org/apache/hadoop/hdds/DFSConfigKeysLegacy.class │ -rw-r--r-- 2.0 unx 3698 b- defN 23-Jan-01 00:00 org/apache/hadoop/hdds/DatanodeVersion.class │ -rw-r--r-- 2.0 unx 12469 b- defN 23-Jan-01 00:00 org/apache/hadoop/hdds/HddsConfigKeys.class │ @@ -425,8 +425,8 @@ │ -rw-r--r-- 2.0 unx 9599 b- defN 23-Jan-01 00:00 org/apache/hadoop/ozone/util/ShutdownHookManager.class │ -rw-r--r-- 2.0 unx 1414 b- defN 23-Jan-01 00:00 org/apache/hadoop/ozone/util/StringWithByteString.class │ -rw-r--r-- 2.0 unx 1423 b- defN 23-Jan-01 00:00 org/apache/hadoop/ozone/util/UUIDUtil.class │ -rw-r--r-- 2.0 unx 11999 b- defN 23-Jan-01 00:00 ozone-default-generated.xml │ -rw-r--r-- 2.0 unx 163523 b- defN 23-Jan-01 00:00 ozone-default.xml │ -rw-r--r-- 2.0 unx 1[13](https://github.com/adoroszlai/ozone/actions/runs/11312548693/job/31460267951#step:10:14)99 b- defN 23-Jan-01 00:00 META-INF/maven/org.apache.ozone/hdds-common/pom.xml │ -rw-r--r-- 2.0 unx 71 b- defN 23-Jan-01 00:00 META-INF/maven/org.apache.ozone/hdds-common/pom.properties │ -430 files, 1670413 bytes uncompressed, 626710 bytes compressed: 62.5% │ +430 files, 1670402 bytes uncompressed, 626707 bytes compressed: 62.5% ├── hdds-version-info.properties │ @@ -[14](https://github.com/adoroszlai/ozone/actions/runs/11312548693/job/31460267951#step:10:15),14 +14,14 @@ │ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. │ # See the License for the specific language governing permissions and │ # limitations under the License. │ # │ │ version=2.0.0-SNAPSHOT │ revision=c0b94741e9e[17](https://github.com/adoroszlai/ozone/actions/runs/11312548693/job/31460267951#step:10:18)3206855148885afdde62fe8a1e1 │ -branch=(HEAD detached at c0b9474) │ +branch=HDDS-11057-test │ url=https://github.com/adoroszlai/ozone │ srcChecksum=2e[19](https://github.com/adoroszlai/ozone/actions/runs/11312548693/job/31460267951#step:10:20)8fe64d737c693739f29336d2341 │ hadoopProtoc2Version=2.5.0 │ hadoopProtoc3Version=3.7.1 │ grpcProtocVersion=3.19.6 │ compilePlatform=linux-x86_64 ``` > Supposedly two builds made at different time results in the same tarball checksum? Currently we only check at the jar level. Tarballs are not yet reproducible due to timestamps at least. I'll check separately if they can be fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
