[OE-core] [PATCH 1/1] opkg-build: Ignore tar error due to hardlinks issue when creating ipk files
If a the number of hard links decreases or increases while creating
the tar files used for an ipk package, tar fails with error code 1,
if this is the case we ignore the error and continue to create the ipk file
[YOCTO #7933]
Signed-off-by: Alejandro Hernandez alejandro.hernan...@linux.intel.com
---
.../opkg-utils/opkg-utils/tar_ignore_error.patch | 47 ++
meta/recipes-devtools/opkg-utils/opkg-utils_git.bb | 4 +-
2 files changed, 50 insertions(+), 1 deletion(-)
create mode 100644
meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch
diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch
b/meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch
new file mode 100644
index 000..4bcae4954
--- /dev/null
+++ b/meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch
@@ -0,0 +1,47 @@
+If a the number of hard links decreases or increases while creating
+the tar files used for an ipk package, tar fails with error code 1:
+
+| DEBUG: Executing python function do_package_ipk
+| tar: ./usr/src/debug/gperf/3.0.4-r0/gperf-3.0.4/src/main.cc: file changed as
we read it
+NOTE: recipe gperf-3.0.4-r0: task do_package_write_ipk: Failed
+ERROR: Task 6539 (recipes-extended/gperf/gperf_3.0.4.bb, do_package_write_ipk)
failed with exit code '1'
+
+
+We detect if the error code produced by tar is 1 and in this case ignore it.
+
+This a similar behavior to the one on dpkg:
+http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818
+
+Upsteam-Status: Inappropriate
+
+Signed-off-by: Alejandro Hernandez alejandro.hernan...@linux.intel.com
+
+
+Index: git/opkg-build
+===
+--- git.orig/opkg-build
git/opkg-build
+@@ -250,8 +250,21 @@ tmp_dir=$dest_dir/IPKG_BUILD.$$
+ mkdir $tmp_dir
+
+ echo $CONTROL $tmp_dir/tarX
+-( cd $pkg_dir tar $ogargs -X $tmp_dir/tarX -cz $tarformat -f
$tmp_dir/data.tar.gz . )
+-( cd $pkg_dir/$CONTROL tar $ogargs -cz $tarformat -f
$tmp_dir/control.tar.gz . )
++
++
++# Ignore error code 1, caused by modifying the number of hard links while
creating the tar file
++rc=0
++( cd $pkg_dir tar $ogargs -X $tmp_dir/tarX -cz $tarformat -f
$tmp_dir/data.tar.gz . ) || rc=$?
++if [[ $rc -ne 1 ]] [[ $rc -ne 0 ]] ;then
++exit $rc
++fi
++
++rc=0
++( cd $pkg_dir/$CONTROL tar $ogargs -cz $tarformat -f
$tmp_dir/control.tar.gz . ) || rc=$?
++if [[ $rc -ne 1 ]] [[ $rc -ne 0 ]] ;then
++exit $rc
++fi
++
+ rm $tmp_dir/tarX
+
+ echo 2.0 $tmp_dir/debian-binary
diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
b/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
index 2800a5d..74b974d 100644
--- a/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
+++ b/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
@@ -10,7 +10,9 @@ PROVIDES += virtual/update-alternatives
SRCREV = 53274f087565fd45d8452c5367997ba6a682a37a
PV = 0.1.8+git${SRCPV}
-SRC_URI = git://git.yoctoproject.org/opkg-utils
+SRC_URI = git://git.yoctoproject.org/opkg-utils
+
+SRC_URI_append_class-native = file://tar_ignore_error.patch
S = ${WORKDIR}/git
--
1.8.4.5
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] opkg-build: Ignore tar error due to hardlinks issue when creating ipk files
On Mon, Jul 06, 2015 at 10:17:19AM +, Alejandro Hernandez wrote:
If a the number of hard links decreases or increases while creating
the tar files used for an ipk package, tar fails with error code 1,
if this is the case we ignore the error and continue to create the ipk file
[YOCTO #7933]
Signed-off-by: Alejandro Hernandez alejandro.hernan...@linux.intel.com
---
.../opkg-utils/opkg-utils/tar_ignore_error.patch | 47
++
meta/recipes-devtools/opkg-utils/opkg-utils_git.bb | 4 +-
2 files changed, 50 insertions(+), 1 deletion(-)
create mode 100644
meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch
diff --git
a/meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch
b/meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch
new file mode 100644
index 000..4bcae4954
--- /dev/null
+++ b/meta/recipes-devtools/opkg-utils/opkg-utils/tar_ignore_error.patch
@@ -0,0 +1,47 @@
+If a the number of hard links decreases or increases while creating
+the tar files used for an ipk package, tar fails with error code 1:
+
+| DEBUG: Executing python function do_package_ipk
+| tar: ./usr/src/debug/gperf/3.0.4-r0/gperf-3.0.4/src/main.cc: file changed
as we read it
+NOTE: recipe gperf-3.0.4-r0: task do_package_write_ipk: Failed
+ERROR: Task 6539 (recipes-extended/gperf/gperf_3.0.4.bb,
do_package_write_ipk) failed with exit code '1'
+
+
+We detect if the error code produced by tar is 1 and in this case ignore it.
+
+This a similar behavior to the one on dpkg:
+http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818
+
+Upsteam-Status: Inappropriate
Typo and why is it inappropriate? Looks like bugfix to me, even for
issue which usually happens in OE way of using opkg-build.
+
+Signed-off-by: Alejandro Hernandez alejandro.hernan...@linux.intel.com
+
+
+Index: git/opkg-build
+===
+--- git.orig/opkg-build
git/opkg-build
+@@ -250,8 +250,21 @@ tmp_dir=$dest_dir/IPKG_BUILD.$$
+ mkdir $tmp_dir
+
+ echo $CONTROL $tmp_dir/tarX
+-( cd $pkg_dir tar $ogargs -X $tmp_dir/tarX -cz $tarformat -f
$tmp_dir/data.tar.gz . )
+-( cd $pkg_dir/$CONTROL tar $ogargs -cz $tarformat -f
$tmp_dir/control.tar.gz . )
++
++
++# Ignore error code 1, caused by modifying the number of hard links while
creating the tar file
++rc=0
++( cd $pkg_dir tar $ogargs -X $tmp_dir/tarX -cz $tarformat -f
$tmp_dir/data.tar.gz . ) || rc=$?
++if [[ $rc -ne 1 ]] [[ $rc -ne 0 ]] ;then
++exit $rc
++fi
++
++rc=0
++( cd $pkg_dir/$CONTROL tar $ogargs -cz $tarformat -f
$tmp_dir/control.tar.gz . ) || rc=$?
++if [[ $rc -ne 1 ]] [[ $rc -ne 0 ]] ;then
++exit $rc
++fi
++
+ rm $tmp_dir/tarX
+
+ echo 2.0 $tmp_dir/debian-binary
diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
b/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
index 2800a5d..74b974d 100644
--- a/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
+++ b/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb
@@ -10,7 +10,9 @@ PROVIDES += virtual/update-alternatives
SRCREV = 53274f087565fd45d8452c5367997ba6a682a37a
PV = 0.1.8+git${SRCPV}
-SRC_URI = git://git.yoctoproject.org/opkg-utils
+SRC_URI = git://git.yoctoproject.org/opkg-utils
+
+SRC_URI_append_class-native = file://tar_ignore_error.patch
Add leading space in append not in original SRC_URI variable.
Why is it applied only for native anyway?
S = ${WORKDIR}/git
--
1.8.4.5
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
Martin 'JaMa' Jansa jabber: martin.ja...@gmail.com
signature.asc
Description: Digital signature
--
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] opkg-build: Ignore tar error due to hardlinks issue when creating ipk files
On 06/07/15 15:10, Martin Jansa wrote: On Mon, Jul 06, 2015 at 10:17:19AM +, Alejandro Hernandez wrote: + +This a similar behavior to the one on dpkg: +http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818 + +Upsteam-Status: Inappropriate Typo and why is it inappropriate? Looks like bugfix to me, even for issue which usually happens in OE way of using opkg-build. It seemed inappropriate to me, since this behavior is not expected (number of hardlinks changing), we can't say for sure that this error will be caused by this reason on when using opkg on another environment -SRC_URI = git://git.yoctoproject.org/opkg-utils +SRC_URI = git://git.yoctoproject.org/opkg-utils + +SRC_URI_append_class-native = file://tar_ignore_error.patch Add leading space in append not in original SRC_URI variable. Why is it applied only for native anyway? Same argument as before, modifying the number of hardlinks while the tar is created happens when building using bitbake, which should not happen on target, if files changed when creating an ipk for any other reason it shouldn't be ignored; space was a typo -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] opkg-build: Ignore tar error due to hardlinks issue when creating ipk files
On 2015-07-06 15:10, Alejandro Hernandez wrote: On 06/07/15 15:10, Martin Jansa wrote: On Mon, Jul 06, 2015 at 10:17:19AM +, Alejandro Hernandez wrote: + +This a similar behavior to the one on dpkg: +http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818 + +Upsteam-Status: Inappropriate Typo and why is it inappropriate? Looks like bugfix to me, even for issue which usually happens in OE way of using opkg-build. It seemed inappropriate to me, since this behavior is not expected (number of hardlinks changing), we can't say for sure that this error will be caused by this reason on when using opkg on another environment -SRC_URI = git://git.yoctoproject.org/opkg-utils +SRC_URI = git://git.yoctoproject.org/opkg-utils + +SRC_URI_append_class-native = file://tar_ignore_error.patch Add leading space in append not in original SRC_URI variable. Why is it applied only for native anyway? Same argument as before, modifying the number of hardlinks while the tar is created happens when building using bitbake, which should not happen on target, if files changed when creating an ipk for any other reason it shouldn't be ignored; space was a typo Why is it happening when using bitbake? This seems like an error to me - one should not be generating the tarball (i.e. packing up the final results) if the set of files being packaged is changing in any way. Note: I'm pretty sure that error can also happen if the inode changes in many different ways, not just the number of links changing. -- Gary Thomas | Consulting for the MLB Associates |Embedded world -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] opkg-build: Ignore tar error due to hardlinks issue when creating ipk files
Hello, On Mon, Jul 6, 2015 at 6:16 PM, Gary Thomas g...@mlbassoc.com wrote: On 2015-07-06 15:10, Alejandro Hernandez wrote: On 06/07/15 15:10, Martin Jansa wrote: On Mon, Jul 06, 2015 at 10:17:19AM +, Alejandro Hernandez wrote: + +This a similar behavior to the one on dpkg: +http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818 + +Upsteam-Status: Inappropriate Typo and why is it inappropriate? Looks like bugfix to me, even for issue which usually happens in OE way of using opkg-build. It seemed inappropriate to me, since this behavior is not expected (number of hardlinks changing), we can't say for sure that this error will be caused by this reason on when using opkg on another environment -SRC_URI = git://git.yoctoproject.org/opkg-utils +SRC_URI = git://git.yoctoproject.org/opkg-utils + +SRC_URI_append_class-native = file://tar_ignore_error.patch Add leading space in append not in original SRC_URI variable. Why is it applied only for native anyway? Same argument as before, modifying the number of hardlinks while the tar is created happens when building using bitbake, which should not happen on target, if files changed when creating an ipk for any other reason it shouldn't be ignored; space was a typo Why is it happening when using bitbake? This seems like an error to me - one should not be generating the tarball (i.e. packing up the final results) if the set of files being packaged is changing in any way. Note: I'm pretty sure that error can also happen if the inode changes in many different ways, not just the number of links changing. I fully agree with Gary; this also makes unreproducible the number of shared hard links in a final rootfs which is also important. This (and dpkg related patch) seems to be covering the real bug. (Adding Richard so he can also comment on this and to bring his attention to the concerns exposed here) -- Otavio Salvador O.S. Systems http://www.ossystems.com.brhttp://code.ossystems.com.br Mobile: +55 (53) 9981-7854Mobile: +1 (347) 903-9750 -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [OE-core] [PATCH 1/1] opkg-build: Ignore tar error due to hardlinks issue when creating ipk files
On Mon, 2015-07-06 at 18:20 -0300, Otavio Salvador wrote: Hello, On Mon, Jul 6, 2015 at 6:16 PM, Gary Thomas g...@mlbassoc.com wrote: On 2015-07-06 15:10, Alejandro Hernandez wrote: On 06/07/15 15:10, Martin Jansa wrote: On Mon, Jul 06, 2015 at 10:17:19AM +, Alejandro Hernandez wrote: + +This a similar behavior to the one on dpkg: +http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=40731942515ec8d80c727ad561174986d4f05818 + +Upsteam-Status: Inappropriate Typo and why is it inappropriate? Looks like bugfix to me, even for issue which usually happens in OE way of using opkg-build. It seemed inappropriate to me, since this behavior is not expected (number of hardlinks changing), we can't say for sure that this error will be caused by this reason on when using opkg on another environment -SRC_URI = git://git.yoctoproject.org/opkg-utils +SRC_URI = git://git.yoctoproject.org/opkg-utils + +SRC_URI_append_class-native = file://tar_ignore_error.patch Add leading space in append not in original SRC_URI variable. Why is it applied only for native anyway? Same argument as before, modifying the number of hardlinks while the tar is created happens when building using bitbake, which should not happen on target, if files changed when creating an ipk for any other reason it shouldn't be ignored; space was a typo Why is it happening when using bitbake? This seems like an error to me - one should not be generating the tarball (i.e. packing up the final results) if the set of files being packaged is changing in any way. Note: I'm pretty sure that error can also happen if the inode changes in many different ways, not just the number of links changing. I fully agree with Gary; this also makes unreproducible the number of shared hard links in a final rootfs which is also important. This (and dpkg related patch) seems to be covering the real bug. (Adding Richard so he can also comment on this and to bring his attention to the concerns exposed here) The issue here is some of the attempts to decrease disk usage and increase the build speed. To do that we hardlink files in the package directories rather than create individual copies in packages/ and package-split/. There are cases where other parts of the build (e.g. populate-sysroot) can create or remote copies of the hardlinked file which changes the link count. The files themselves do not change, only the link does and then only rarely. We fixed dpkg a while ago for this issue, this is the same fix being applied to opkg-build. The resulting packages will be the same since the hardlink count is internal to the tarball and not related to the number of original file copies. Since we control the build env, it is safe to do what we're doing here, however I would agree with Alejandro that it isn't something upstream would want to do. Cheers, Richard -- ___ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
