Re: [pacman-dev] [PATCH] makepkg: introduce SOURCE_DATE_EPOCH

[Allan McRae]

On 17/04/17 23:34, Andrew Gregory wrote:
> On 04/17/17 at 08:41pm, Allan McRae wrote:
>> This patch introduces the SOURCE_DATE_EPOCH environmental variable.  All 
>> files
>> in a package are adjusted to have their modification dates set to the value
>> of SOURCE_DATE_EPOCH, which defaults to "date +%s".
>>
>> Setting this variable allows a package that is built twice in the same
>> environment to be (potentially) reproducible in that the checksum of the
>> generated package file will be the same.
>>
>> Signed-off-by: Allan McRae 
> 
> I'm of the opinion that makepkg is the wrong place to work on
> reproducible builds.  We could probably take care of the low-hanging
> fruit directly in makepkg, but a number of packages are going to
> require more find-grained control over the environment then I think we
> should be putting in makepkg.  If you look at `perl -V`, for instance,
> it embeds the output of `uname -a` and a timestamp directly in the
> executable.  I suspect that any effort we put into reproducible builds
> with makepkg would eventually have to be duplicated with a more
> powerful wrapper script in order to handle packages like perl that
> record more of their environment than we should be manipulating in
> makepkg.

I agree that makepkg is not the place for much of this.  However, the
SOURCE_DATE_EPOCH variable is a standard and we require makepkg to
understand it and make a few other minor changes for any tool to have a
chance of recreating a package from its PKGBUILD and .BUILDINFO file.  I
am not looking to extend the changes beyond this initial patchset.

Allan

Re: [pacman-dev] [PATCH] makepkg: introduce SOURCE_DATE_EPOCH

[Allan McRae]

On 17/04/17 23:37, Andrew Gregory wrote:
> On 04/17/17 at 10:04pm, Allan McRae wrote:
>> On 17/04/17 20:41, Allan McRae wrote:
>>> +   # ensure all elements of the package have the same mtime
>>> +   find . -exec touch -d @$SOURCE_DATE_EPOCH {} \;
>>> +
>>> msg2 "$(gettext "Generating .MTREE file...")"
>>> -   list_package_files | LANG=C bsdtar -cnzf .MTREE --format=mtree \
>>> +   list_package_files | LANG=C bsdtar -cnf - --format=mtree \
>>> 
>>> --options='!all,use-set,type,uid,gid,mode,time,size,md5,sha256,link' \
>>> -   --null --files-from - --exclude .MTREE
>>> +   --null --files-from - --exclude .MTREE | gzip -c -f -n > .MTREE
>>> +   touch -d @$SOURCE_DATE_EPOCH .MTREE
>>>  
>>> msg2 "$(gettext "Compressing package...")"
>>> # TODO: Maybe this can be set globally for robustness
>>>
>>
>> These touch commands have had a -h added.
> 
> touch -h and date %s are not POSIX, are they available everywhere we
> support?
> 

touch -h is in BSDs.  date +%s is mentioned in the FreeBSD man page, so
I assume it works.

A

Re: [pacman-dev] [PATCH] makepkg: introduce SOURCE_DATE_EPOCH

[Allan McRae]

On 17/04/17 23:37, Andrew Gregory wrote:
> On 04/17/17 at 10:04pm, Allan McRae wrote:
>> On 17/04/17 20:41, Allan McRae wrote:
>>> +   # ensure all elements of the package have the same mtime
>>> +   find . -exec touch -d @$SOURCE_DATE_EPOCH {} \;
>>> +
>>> msg2 "$(gettext "Generating .MTREE file...")"
>>> -   list_package_files | LANG=C bsdtar -cnzf .MTREE --format=mtree \
>>> +   list_package_files | LANG=C bsdtar -cnf - --format=mtree \
>>> 
>>> --options='!all,use-set,type,uid,gid,mode,time,size,md5,sha256,link' \
>>> -   --null --files-from - --exclude .MTREE
>>> +   --null --files-from - --exclude .MTREE | gzip -c -f -n > .MTREE
>>> +   touch -d @$SOURCE_DATE_EPOCH .MTREE
>>>  
>>> msg2 "$(gettext "Compressing package...")"
>>> # TODO: Maybe this can be set globally for robustness
>>>
>>
>> These touch commands have had a -h added.
> 
> touch -h and date %s are not POSIX, are they available everywhere we
> support?
> 
> Why the change to gzip for .MTREE?
> 

A timestamp is embed in a gz file unless gzip -n is used.

A

Re: [pacman-dev] [PATCH] makepkg: introduce SOURCE_DATE_EPOCH

[Levente Polyak]

On 04/17/2017 08:42 PM, Andrew Gregory wrote:
> I have no problem with making makepkg's own output more controllable
> (e.g.  allowing builddate to be set rather than using the current
> time).  But, a lot of the time, reproducing an identical package is
> going to require a very precise environment, especially for compiled
> software.  The environmental factors that influence the built software
> vary from project to project and can get their values from a variety
> of locations.  I think that trying to manage all of that from makepkg
> would be a mistake if it would even be possible.  Some things, like
> building in a chroot for software that embeds the build directory,
> would almost certainly be easier from a script that wraps makepkg.
> I would prefer to see effort be put toward such a script rather than
> have it go into makepkg only to have to be moved to a separate script
> later.
> 
> apg
> 


I fully agree with your points... actually exactly that is the plan and
the reason the .BUILDINFO file exists -- to be able to recreate the very
precise environment that was used to build a package. This is of cause
needed, as you mentioned, for things like some binary software (gcc
version)... but we actually include the .BUILDINFO file into the package
itself. This has IMO a lot of advantages but that already declares the
requirement to have an exact identical environment to be reproducible.

The current set of adjustments are needed for makepkg itself. I'm sure
nobody intends to go lot further and include environment recreation
things or explicit software dependent stuff (like PERL_BUILD_DATE).

makechrootpkg and things like that are project (like Arch) specific.
Surely there will be the need of a wrapper around it to recreate an
identical environment from the .BUILDINFO file to be able to reproduce a
package beyond invoking it twice (something like makerepropkg).
On top of that, there will always be some need to add some things to
PKGBUILD files that are software dependent. An example would be to
define PERL_BUILD_DATE="${SOURCE_DATE_EPOCH}" and i agree that something
like PERL_BUILD_DATE is not to be included in makepkg itself.

I hope i could settle some of your concerns :)

cheers,
Levente



signature.asc
Description: OpenPGP digital signature

Re: [pacman-dev] [PATCH] makepkg: introduce SOURCE_DATE_EPOCH

[Andrew Gregory]

On 04/17/17 at 03:53pm, Levente Polyak wrote:
> On 04/17/2017 03:34 PM, Andrew Gregory wrote:
> > On 04/17/17 at 08:41pm, Allan McRae wrote:
> >> This patch introduces the SOURCE_DATE_EPOCH environmental variable.  All 
> >> files
> >> in a package are adjusted to have their modification dates set to the value
> >> of SOURCE_DATE_EPOCH, which defaults to "date +%s".
> >>
> >> Setting this variable allows a package that is built twice in the same
> >> environment to be (potentially) reproducible in that the checksum of the
> >> generated package file will be the same.
> >>
> >> Signed-off-by: Allan McRae 
> > 
> > I'm of the opinion that makepkg is the wrong place to work on
> > reproducible builds.  We could probably take care of the low-hanging
> > fruit directly in makepkg, but a number of packages are going to
> > require more find-grained control over the environment then I think we
> > should be putting in makepkg.  If you look at `perl -V`, for instance,
> > it embeds the output of `uname -a` and a timestamp directly in the
> > executable.  I suspect that any effort we put into reproducible builds
> > with makepkg would eventually have to be duplicated with a more
> > powerful wrapper script in order to handle packages like perl that
> > record more of their environment than we should be manipulating in
> > makepkg.
> > 
> > apg
> > 
> 
> Makepkg is the place that we control and need to work on to make
> packages created by makepkg reproducible. Currently they are not exactly
> because of the reasons these patches address and there is literally no
> way to get reproducible package artifacts without these patches.
> Especially the deterministic way to pass in SOURCE_DATE_EPOCH is a
> requirement for cases you mentioned and downstream projects using dates
> in any produced artifacts should implement SOURCE_DATE_EPOCH. An
> incredible high amount of projects already do so and more and more adopt
> as this is getting infacto a standard (actually it already is).
> No complex wrapper scripts should be needed at any place to achieve
> reproducibility.
> 
> cheers,
> Levente

I have no problem with making makepkg's own output more controllable
(e.g.  allowing builddate to be set rather than using the current
time).  But, a lot of the time, reproducing an identical package is
going to require a very precise environment, especially for compiled
software.  The environmental factors that influence the built software
vary from project to project and can get their values from a variety
of locations.  I think that trying to manage all of that from makepkg
would be a mistake if it would even be possible.  Some things, like
building in a chroot for software that embeds the build directory,
would almost certainly be easier from a script that wraps makepkg.
I would prefer to see effort be put toward such a script rather than
have it go into makepkg only to have to be moved to a separate script
later.

apg

Re: [pacman-dev] [PATCH] makepkg: introduce SOURCE_DATE_EPOCH

[Andrew Gregory]

On 04/17/17 at 10:04pm, Allan McRae wrote:
> On 17/04/17 20:41, Allan McRae wrote:
> > +   # ensure all elements of the package have the same mtime
> > +   find . -exec touch -d @$SOURCE_DATE_EPOCH {} \;
> > +
> > msg2 "$(gettext "Generating .MTREE file...")"
> > -   list_package_files | LANG=C bsdtar -cnzf .MTREE --format=mtree \
> > +   list_package_files | LANG=C bsdtar -cnf - --format=mtree \
> > 
> > --options='!all,use-set,type,uid,gid,mode,time,size,md5,sha256,link' \
> > -   --null --files-from - --exclude .MTREE
> > +   --null --files-from - --exclude .MTREE | gzip -c -f -n > .MTREE
> > +   touch -d @$SOURCE_DATE_EPOCH .MTREE
> >  
> > msg2 "$(gettext "Compressing package...")"
> > # TODO: Maybe this can be set globally for robustness
> > 
> 
> These touch commands have had a -h added.

touch -h and date %s are not POSIX, are they available everywhere we
support?

Why the change to gzip for .MTREE?

apg

Re: [pacman-dev] [PATCH] makepkg: introduce SOURCE_DATE_EPOCH

[Andrew Gregory]

On 04/17/17 at 08:41pm, Allan McRae wrote:
> This patch introduces the SOURCE_DATE_EPOCH environmental variable.  All files
> in a package are adjusted to have their modification dates set to the value
> of SOURCE_DATE_EPOCH, which defaults to "date +%s".
> 
> Setting this variable allows a package that is built twice in the same
> environment to be (potentially) reproducible in that the checksum of the
> generated package file will be the same.
> 
> Signed-off-by: Allan McRae 

I'm of the opinion that makepkg is the wrong place to work on
reproducible builds.  We could probably take care of the low-hanging
fruit directly in makepkg, but a number of packages are going to
require more find-grained control over the environment then I think we
should be putting in makepkg.  If you look at `perl -V`, for instance,
it embeds the output of `uname -a` and a timestamp directly in the
executable.  I suspect that any effort we put into reproducible builds
with makepkg would eventually have to be duplicated with a more
powerful wrapper script in order to handle packages like perl that
record more of their environment than we should be manipulating in
makepkg.

apg

Re: [pacman-dev] [PATCH 4/4] [RFC] makepkg: unify times for generated files in srcdir before packaging

[Dave Reisner]

On Mon, Apr 17, 2017 at 10:03:03PM +1000, Allan McRae wrote:
> From: Levente Polyak 
> 
> Signed-off-by: Allan McRae 
> ---
> 
> [Allan] I'm told his is useful for some python packages that generate pyo/pyc
> files during package...  I am undecided about its suitability for inclusion
> in makepkg yet.
> 
>  scripts/makepkg.sh.in | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
> index df4d6a06..84b83e7d 100644
> --- a/scripts/makepkg.sh.in
> +++ b/scripts/makepkg.sh.in
> @@ -493,6 +493,8 @@ run_package() {
>   pkgfunc="package_$1"
>   fi
>  
> + # unify source times before package for reproducibility
> + find "$srcdir" -exec touch -h -d "@${SOURCE_DATE_EPOCH}" {} \;

Same as 3/4 -- prefer {} +.

If we accept this patch, the commit message should include an
explanation as to why this is useful.

>   run_function_safe "$pkgfunc"
>  }
>  
> -- 
> 2.12.0

Re: [pacman-dev] [PATCH 3/4] makepkg: unify source file times for improved build reproducibility

[Dave Reisner]

On Mon, Apr 17, 2017 at 10:03:02PM +1000, Allan McRae wrote:
> From: Levente Polyak 
> 
> Signed-off-by: Allan McRae 
> ---
>  scripts/makepkg.sh.in | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
> index 7692ade5..df4d6a06 100644
> --- a/scripts/makepkg.sh.in
> +++ b/scripts/makepkg.sh.in
> @@ -475,6 +475,9 @@ run_prepare() {
>  }
>  
>  run_build() {
> + # unify source times before building for reproducibility
> + find "$srcdir" -exec touch -h -d "@${SOURCE_DATE_EPOCH}" {} \;
> +

I'd use the '{} +' form of find here to avoid excessive forking.

>   run_function_safe "build"
>  }
>  
> -- 
> 2.12.0

Re: [pacman-dev] [PATCH 1/4] makepkg: extract parts of the write_pkginfo for use elsewhere

[Dave Reisner]

On Mon, Apr 17, 2017 at 10:03:00PM +1000, Allan McRae wrote:
> From: Levente Polyak 
> 
> Signed-off-by: Allan McRae 
> ---

Sorry, a lot of these comments are irrelevant to the actual patch, but I
couldn't help pointing them out...

>  scripts/makepkg.sh.in | 42 ++
>  1 file changed, 26 insertions(+), 16 deletions(-)
> 
> diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
> index 42a76004..d61c7fff 100644
> --- a/scripts/makepkg.sh.in
> +++ b/scripts/makepkg.sh.in
> @@ -608,6 +608,15 @@ find_libprovides() {
>   (( ${#libprovides[@]} )) && printf '%s\n' "${libprovides[@]}"
>  }
>  
> +get_packager() {
> + if [[ -n $PACKAGER ]]; then
> + local packager="$PACKAGER"
> + else
> + local packager="Unknown Packager"
> + fi
> + printf "%s\n" "$packager"

I was going to suggest that we simply make this:

  printf '%s\n' "${PACKAGER:-Unknown Packager}"

But then it occurred to me that if we just set this default value up
front, we don't need to treat this var as special...

Actually relevant to this patch, why not define this as
'write_kv_packager' to match other functions here, like
'write_kv_pkgname' and 'write_kv_pkgver'?

> +}
> +
>  write_kv_pair() {
>   local key="$1"
>   shift
> @@ -621,13 +630,22 @@ write_kv_pair() {
>   done
>  }
>  
> -write_pkginfo() {
> - if [[ -n $PACKAGER ]]; then
> - local packager="$PACKAGER"
> - else
> - local packager="Unknown Packager"
> +write_kv_pkgname() {
> + write_kv_pair "pkgname" "$pkgname"
> + if (( SPLITPKG )) || [[ "$pkgbase" != "$pkgname" ]]; then
> + write_kv_pair "pkgbase" "$pkgbase"
> + fi

Wouldn't it be nice if we just *always* wrote the pkgbase?

> +}
> +
> +write_kv_pkgver() {
> + local fullver=$(get_full_version)
> + write_kv_pair "pkgver" "$fullver"
> + if [[ "$fullver" != "$basever" ]]; then
> + write_kv_pair "basever" "$basever"
>   fi

Since 8a02abcf19, disallow pkgver overrides in package functions.
Therefore, I'm unclear on when we'd ever emit this basever attr.

> +}
>  
> +write_pkginfo() {
>   local size="$(@DUPATH@ @DUFLAGS@)"
>   size="$(( ${size%%[^0-9]*} * 1024 ))"
>  
> @@ -637,16 +655,8 @@ write_pkginfo() {
>   printf "# Generated by makepkg %s\n" "$makepkg_version"
>   printf "# using %s\n" "$(fakeroot -v)"
>  
> - write_kv_pair "pkgname" "$pkgname"
> - if (( SPLITPKG )) || [[ "$pkgbase" != "$pkgname" ]]; then
> - write_kv_pair "pkgbase" "$pkgbase"
> - fi
> -
> - local fullver=$(get_full_version)
> - write_kv_pair "pkgver" "$fullver"
> - if [[ "$fullver" != "$basever" ]]; then
> - write_kv_pair "basever" "$basever"
> - fi
> + write_kv_pkgname
> + write_kv_pkgver
>  
>   # TODO: all fields should have this treatment
>   local spd="${pkgdesc//+([[:space:]])/ }"
> @@ -656,7 +666,7 @@ write_pkginfo() {
>   write_kv_pair "pkgdesc" "$spd"
>   write_kv_pair "url" "$url"
>   write_kv_pair "builddate" "$SOURCE_DATE_EPOCH"
> - write_kv_pair "packager" "$packager"
> + write_kv_pair "packager" "$(get_packager)"
>   write_kv_pair "size" "$size"
>   write_kv_pair "arch" "$pkgarch"
>  
> -- 
> 2.12.0

Re: [pacman-dev] [PATCH] makepkg: introduce SOURCE_DATE_EPOCH

[Allan McRae]

On 17/04/17 20:41, Allan McRae wrote:
> + # ensure all elements of the package have the same mtime
> + find . -exec touch -d @$SOURCE_DATE_EPOCH {} \;
> +
>   msg2 "$(gettext "Generating .MTREE file...")"
> - list_package_files | LANG=C bsdtar -cnzf .MTREE --format=mtree \
> + list_package_files | LANG=C bsdtar -cnf - --format=mtree \
>   
> --options='!all,use-set,type,uid,gid,mode,time,size,md5,sha256,link' \
> - --null --files-from - --exclude .MTREE
> + --null --files-from - --exclude .MTREE | gzip -c -f -n > .MTREE
> + touch -d @$SOURCE_DATE_EPOCH .MTREE
>  
>   msg2 "$(gettext "Compressing package...")"
>   # TODO: Maybe this can be set globally for robustness
> 

These touch commands have had a -h added.

A

[pacman-dev] [PATCH 4/4] [RFC] makepkg: unify times for generated files in srcdir before packaging

[Allan McRae]

From: Levente Polyak 

Signed-off-by: Allan McRae 
---

[Allan] I'm told his is useful for some python packages that generate pyo/pyc
files during package...  I am undecided about its suitability for inclusion
in makepkg yet.

 scripts/makepkg.sh.in | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
index df4d6a06..84b83e7d 100644
--- a/scripts/makepkg.sh.in
+++ b/scripts/makepkg.sh.in
@@ -493,6 +493,8 @@ run_package() {
pkgfunc="package_$1"
fi
 
+   # unify source times before package for reproducibility
+   find "$srcdir" -exec touch -h -d "@${SOURCE_DATE_EPOCH}" {} \;
run_function_safe "$pkgfunc"
 }
 
-- 
2.12.0

[pacman-dev] [PATCH 3/4] makepkg: unify source file times for improved build reproducibility

[Allan McRae]

From: Levente Polyak 

Signed-off-by: Allan McRae 
---
 scripts/makepkg.sh.in | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
index 7692ade5..df4d6a06 100644
--- a/scripts/makepkg.sh.in
+++ b/scripts/makepkg.sh.in
@@ -475,6 +475,9 @@ run_prepare() {
 }
 
 run_build() {
+   # unify source times before building for reproducibility
+   find "$srcdir" -exec touch -h -d "@${SOURCE_DATE_EPOCH}" {} \;
+
run_function_safe "build"
 }
 
-- 
2.12.0

[pacman-dev] [PATCH 2/4] makepkg: add more information to .BUILDINFO

[Allan McRae]

From: Levente Polyak 

The .BUILDINFO file should retain all the information needed to reproducibly
build a package.  Add some extra information to the file and also provide a
version number to keep track of future changes.

Signed-off-by: Allan McRae 
---
 scripts/makepkg.sh.in | 8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
index d61c7fff..7692ade5 100644
--- a/scripts/makepkg.sh.in
+++ b/scripts/makepkg.sh.in
@@ -688,13 +688,17 @@ write_pkginfo() {
 write_buildinfo() {
msg2 "$(gettext "Generating %s file...")" ".BUILDINFO"
 
-   write_kv_pair "builddir"  "${BUILDDIR}"
+   write_kv_pair "format" "1"
+   write_kv_pkgname
+   write_kv_pkgver
 
local sum="$(sha256sum "${BUILDFILE}")"
sum=${sum%% *}
-
write_kv_pair "pkgbuild_sha256sum" $sum
 
+   write_kv_pair "packager" "$(get_packager)"
+   write_kv_pair "builddate" "${SOURCE_DATE_EPOCH}"
+   write_kv_pair "builddir"  "${BUILDDIR}"
write_kv_pair "buildenv" "${BUILDENV[@]}"
write_kv_pair "options" "${OPTIONS[@]}"
 
-- 
2.12.0

[pacman-dev] [PATCH 1/4] makepkg: extract parts of the write_pkginfo for use elsewhere

[Allan McRae]

From: Levente Polyak 

Signed-off-by: Allan McRae 
---
 scripts/makepkg.sh.in | 42 ++
 1 file changed, 26 insertions(+), 16 deletions(-)

diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
index 42a76004..d61c7fff 100644
--- a/scripts/makepkg.sh.in
+++ b/scripts/makepkg.sh.in
@@ -608,6 +608,15 @@ find_libprovides() {
(( ${#libprovides[@]} )) && printf '%s\n' "${libprovides[@]}"
 }
 
+get_packager() {
+   if [[ -n $PACKAGER ]]; then
+   local packager="$PACKAGER"
+   else
+   local packager="Unknown Packager"
+   fi
+   printf "%s\n" "$packager"
+}
+
 write_kv_pair() {
local key="$1"
shift
@@ -621,13 +630,22 @@ write_kv_pair() {
done
 }
 
-write_pkginfo() {
-   if [[ -n $PACKAGER ]]; then
-   local packager="$PACKAGER"
-   else
-   local packager="Unknown Packager"
+write_kv_pkgname() {
+   write_kv_pair "pkgname" "$pkgname"
+   if (( SPLITPKG )) || [[ "$pkgbase" != "$pkgname" ]]; then
+   write_kv_pair "pkgbase" "$pkgbase"
+   fi
+}
+
+write_kv_pkgver() {
+   local fullver=$(get_full_version)
+   write_kv_pair "pkgver" "$fullver"
+   if [[ "$fullver" != "$basever" ]]; then
+   write_kv_pair "basever" "$basever"
fi
+}
 
+write_pkginfo() {
local size="$(@DUPATH@ @DUFLAGS@)"
size="$(( ${size%%[^0-9]*} * 1024 ))"
 
@@ -637,16 +655,8 @@ write_pkginfo() {
printf "# Generated by makepkg %s\n" "$makepkg_version"
printf "# using %s\n" "$(fakeroot -v)"
 
-   write_kv_pair "pkgname" "$pkgname"
-   if (( SPLITPKG )) || [[ "$pkgbase" != "$pkgname" ]]; then
-   write_kv_pair "pkgbase" "$pkgbase"
-   fi
-
-   local fullver=$(get_full_version)
-   write_kv_pair "pkgver" "$fullver"
-   if [[ "$fullver" != "$basever" ]]; then
-   write_kv_pair "basever" "$basever"
-   fi
+   write_kv_pkgname
+   write_kv_pkgver
 
# TODO: all fields should have this treatment
local spd="${pkgdesc//+([[:space:]])/ }"
@@ -656,7 +666,7 @@ write_pkginfo() {
write_kv_pair "pkgdesc" "$spd"
write_kv_pair "url" "$url"
write_kv_pair "builddate" "$SOURCE_DATE_EPOCH"
-   write_kv_pair "packager" "$packager"
+   write_kv_pair "packager" "$(get_packager)"
write_kv_pair "size" "$size"
write_kv_pair "arch" "$pkgarch"
 
-- 
2.12.0

[pacman-dev] [PATCH] makepkg: introduce SOURCE_DATE_EPOCH

[Allan McRae]

This patch introduces the SOURCE_DATE_EPOCH environmental variable.  All files
in a package are adjusted to have their modification dates set to the value
of SOURCE_DATE_EPOCH, which defaults to "date +%s".

Setting this variable allows a package that is built twice in the same
environment to be (potentially) reproducible in that the checksum of the
generated package file will be the same.

Signed-off-by: Allan McRae 
---
 scripts/makepkg.sh.in | 13 +
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
index c019ae3b..529b51f7 100644
--- a/scripts/makepkg.sh.in
+++ b/scripts/makepkg.sh.in
@@ -87,6 +87,8 @@ SPLITPKG=0
 SOURCEONLY=0
 VERIFYSOURCE=0
 
+SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-$(date +%s)}
+
 PACMAN_OPTS=()
 
 shopt -s extglob
@@ -620,7 +622,6 @@ write_kv_pair() {
 }
 
 write_pkginfo() {
-   local builddate=$(date -u "+%s")
if [[ -n $PACKAGER ]]; then
local packager="$PACKAGER"
else
@@ -654,7 +655,7 @@ write_pkginfo() {
 
write_kv_pair "pkgdesc" "$spd"
write_kv_pair "url" "$url"
-   write_kv_pair "builddate" "$builddate"
+   write_kv_pair "builddate" "$SOURCE_DATE_EPOCH"
write_kv_pair "packager" "$packager"
write_kv_pair "size" "$size"
write_kv_pair "arch" "$pkgarch"
@@ -738,10 +739,14 @@ create_package() {
[[ -f $pkg_file ]] && rm -f "$pkg_file"
[[ -f $pkg_file.sig ]] && rm -f "$pkg_file.sig"
 
+   # ensure all elements of the package have the same mtime
+   find . -exec touch -d @$SOURCE_DATE_EPOCH {} \;
+
msg2 "$(gettext "Generating .MTREE file...")"
-   list_package_files | LANG=C bsdtar -cnzf .MTREE --format=mtree \
+   list_package_files | LANG=C bsdtar -cnf - --format=mtree \

--options='!all,use-set,type,uid,gid,mode,time,size,md5,sha256,link' \
-   --null --files-from - --exclude .MTREE
+   --null --files-from - --exclude .MTREE | gzip -c -f -n > .MTREE
+   touch -d @$SOURCE_DATE_EPOCH .MTREE
 
msg2 "$(gettext "Compressing package...")"
# TODO: Maybe this can be set globally for robustness
-- 
2.12.0

[pacman-dev] [PATCH] makepkg: remove build date from .PKGINFO header

[Allan McRae]

This information is duplicated (in less friendly format) in the "builddate"
entry and removing it improves reproducible packaging.

Signed-off-by: Allan McRae 
---
 scripts/makepkg.sh.in | 1 -
 1 file changed, 1 deletion(-)

diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
index 43c7e328..c019ae3b 100644
--- a/scripts/makepkg.sh.in
+++ b/scripts/makepkg.sh.in
@@ -635,7 +635,6 @@ write_pkginfo() {
msg2 "$(gettext "Generating %s file...")" ".PKGINFO"
printf "# Generated by makepkg %s\n" "$makepkg_version"
printf "# using %s\n" "$(fakeroot -v)"
-   printf "# %s\n" "$(LC_ALL=C date -u)"
 
write_kv_pair "pkgname" "$pkgname"
if (( SPLITPKG )) || [[ "$pkgbase" != "$pkgname" ]]; then
-- 
2.12.0

Re: [pacman-dev] [PATCH] be_sync: error out if a db cannot be parsed

[Allan McRae]

On 17/04/17 13:28, Andrew Gregory wrote:
> Signed-off-by: Andrew Gregory 
> ---
> 
> See FS#49342 for an example of how to break a database in a way that was
> previously silently ignored.  The linked github issue includes a copy of an
> actual broken db.
> 

Looks like the ideal solution requires libarchive changes.  This is the
best we can do at our end for the time being.

Thanks,
Allan

[pacman-dev] [PATCH 1/2] makepkg: do not create symlinks in build directory

[Allan McRae]

Setting PKGDEST and friends enables us to keep all built packages in a single
location.  Symlinking these files into the build directory creates unnecessary
clutter and requires clean-up in multiple places when removing old version.

Signed-off-by: Allan McRae 
---

Does anyone use these symlinks?   Are they needed?
Removing this makes the next patch much easier...

 scripts/makepkg.sh.in | 28 
 1 file changed, 28 deletions(-)

diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
index 0218e13b..be0ea72e 100644
--- a/scripts/makepkg.sh.in
+++ b/scripts/makepkg.sh.in
@@ -771,20 +771,6 @@ create_package() {
fi
 
create_signature "$pkg_file"
-
-   if (( ! ret )) && [[ ! "$PKGDEST" -ef "${startdir}" ]]; then
-   rm -f "${pkg_file/$PKGDEST/$startdir}"
-   ln -s "${pkg_file}" "${pkg_file/$PKGDEST/$startdir}"
-   ret=$?
-   if [[ -f $pkg_file.sig ]]; then
-   rm -f "${pkg_file/$PKGDEST/$startdir}.sig"
-   ln -s "$pkg_file.sig" 
"${pkg_file/$PKGDEST/$startdir}.sig"
-   fi
-   fi
-
-   if (( ret )); then
-   warning "$(gettext "Failed to create symlink to package file.")"
-   fi
 }
 
 create_debug_package() {
@@ -884,20 +870,6 @@ create_srcpackage() {
 
create_signature "$pkg_file"
 
-   if [[ ! "$SRCPKGDEST" -ef "${startdir}" ]]; then
-   rm -f "${pkg_file/$SRCPKGDEST/$startdir}"
-   ln -s "${pkg_file}" "${pkg_file/$SRCPKGDEST/$startdir}"
-   ret=$?
-   if [[ -f $pkg_file.sig ]]; then
-   rm -f "${pkg_file/$SRCPKGDEST/$startdir}.sig"
-   ln -s "$pkg_file.sig" 
"${pkg_file/$SRCPKGDEST/$startdir}.sig"
-   fi
-   fi
-
-   if (( ret )); then
-   warning "$(gettext "Failed to create symlink to source package 
file.")"
-   fi
-
cd_safe "${startdir}"
rm -rf "${srclinks}"
 }
-- 
2.12.0

[pacman-dev] [PATCH 2/2] makepkg: create signature files outside of fakeroot

[Allan McRae]

With recent version of gpg, signing within fakeroot works on the first
invocation, but fails on later runs.  Sign all packages outside of fakeroot
to avoid this issue.

Fixes FS#49946.

Signed-off-by: Allan McRae 
---
 .../libmakepkg/integrity/generate_signature.sh.in  | 34 --
 scripts/makepkg.sh.in  |  9 +++---
 2 files changed, 36 insertions(+), 7 deletions(-)

diff --git a/scripts/libmakepkg/integrity/generate_signature.sh.in 
b/scripts/libmakepkg/integrity/generate_signature.sh.in
index 060ae344..6d65d82d 100644
--- a/scripts/libmakepkg/integrity/generate_signature.sh.in
+++ b/scripts/libmakepkg/integrity/generate_signature.sh.in
@@ -27,11 +27,10 @@ source "$LIBRARY/util/message.sh"
 
 create_signature() {
if [[ $SIGNPKG != 'y' ]]; then
-   return
+   return 0
fi
local ret=0
local filename="$1"
-   msg "$(gettext "Signing package...")"
 
local SIGNWITHKEY=""
if [[ -n $GPGKEY ]]; then
@@ -42,8 +41,37 @@ create_signature() {
 
 
if (( ! ret )); then
-   msg2 "$(gettext "Created signature file %s.")" "$filename.sig"
+   msg2 "$(gettext "Created signature file %s.")" 
"${filename##*/}.sig"
else
warning "$(gettext "Failed to sign package file.")"
fi
+
+   return $ret
+}
+
+create_package_signatures() {
+   local pkgarch pkg_file
+   local pkgname_backup=("${pkgname[@]}")
+   local fullver=$(get_full_version)
+
+   msg "$(gettext "Signing package(s)...")"
+
+   for pkgname in ${pkgname_backup[@]}; do
+   pkgarch=$(get_pkg_arch $pkgname)
+   pkg_file="$PKGDEST/${pkgname}-${fullver}-${pkgarch}${PKGEXT}"
+
+   create_signature "$pkg_file"
+   done
+
+   # check if debug package needs a signature
+   if ! check_option "debug" "y" || ! check_option "strip" "y"; then
+   return
+   fi
+
+   pkgname=$pkgbase-@DEBUGSUFFIX@
+   pkgarch=$(get_pkg_arch)
+   pkg_file="$PKGDEST/${pkgname}-${fullver}-${pkgarch}${PKGEXT}"
+   create_signature "$pkg_file"
+
+   pkgname=("${pkgname_backup[@]}")
 }
diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
index be0ea72e..ca9b6685 100644
--- a/scripts/makepkg.sh.in
+++ b/scripts/makepkg.sh.in
@@ -769,8 +769,6 @@ create_package() {
error "$(gettext "Failed to create package file.")"
exit 1 # TODO: error code
fi
-
-   create_signature "$pkg_file"
 }
 
 create_debug_package() {
@@ -868,8 +866,6 @@ create_srcpackage() {
exit 1 # TODO: error code
fi
 
-   create_signature "$pkg_file"
-
cd_safe "${startdir}"
rm -rf "${srclinks}"
 }
@@ -1625,6 +1621,9 @@ if (( SOURCEONLY )); then
 
enter_fakeroot
 
+   msg "$(gettext "Signing package...")"
+   create_signature "$SRCPKGDEST/${pkgbase}-${fullver}${SRCEXT}"
+
msg "$(gettext "Source package created: %s")" "$pkgbase ($(date))"
exit 0
 fi
@@ -1716,6 +1715,8 @@ else
fi
 
enter_fakeroot
+
+create_package_signatures
 fi
 
 # if inhibiting archive creation, go no further
-- 
2.12.0