With the following patch
From f29ec67239094a256fcfc119fb75be90923d3448 Mon Sep 17 00:00:00 2001
From: Devendra Tewari <[email protected]>
Date: Mon, 29 Mar 2021 21:11:56 -0300
Subject: [PATCH] Use shutil.move to rename sstate
Incremental build in Docker fails with
OSError: [Errno 18] Invalid cross-device link
When source and destination are on different overlay filesystems.
shutil.move uses os.rename when destination is on the current filesystem,
otherwise source is copied to destination and then removed.
---
meta/classes/sstate.bbclass | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index f579168162..2e87697e3e 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -384,6 +384,7 @@ def sstate_installpkg(ss, d):
def sstate_installpkgdir(ss, d):
import oe.path
import subprocess
+ import shutil
sstateinst = d.getVar("SSTATE_INSTDIR")
d.setVar('SSTATE_FIXMEDIR', ss['fixmedir'])
@@ -401,7 +402,7 @@ def sstate_installpkgdir(ss, d):
for state in ss['dirs']:
prepdir(state[1])
- os.rename(sstateinst + state[0], state[1])
+ shutil.move(sstateinst + state[0], state[1])
sstate_install(ss, d)
for plain in ss['plaindirs']:
@@ -413,7 +414,7 @@ def sstate_installpkgdir(ss, d):
dest = plain
bb.utils.mkdirhier(src)
prepdir(dest)
- os.rename(src, dest)
+ shutil.move(src, dest)
return True
@@ -638,6 +639,7 @@ python sstate_hardcode_path () {
def sstate_package(ss, d):
import oe.path
+ import shutil
tmpdir = d.getVar('TMPDIR')
@@ -664,7 +666,7 @@ def sstate_package(ss, d):
continue
bb.error("sstate found an absolute path symlink %s pointing at
%s. Please replace this with a relative link." % (srcpath, link))
bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1],
sstatebuild + state[0]))
- os.rename(state[1], sstatebuild + state[0])
+ shutil.move(state[1], sstatebuild + state[0])
workdir = d.getVar('WORKDIR')
sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
@@ -674,7 +676,7 @@ def sstate_package(ss, d):
pdir = plain.replace(sharedworkdir, sstatebuild)
bb.utils.mkdirhier(plain)
bb.utils.mkdirhier(pdir)
- os.rename(plain, pdir)
+ shutil.move(plain, pdir)
d.setVar('SSTATE_BUILDDIR', sstatebuild)
d.setVar('SSTATE_INSTDIR', sstatebuild)
--
2.29.2
The build fails with
#12 2734.2 2021-03-30 01:14:29 - INFO - NOTE: recipe
linux-libc-headers-5.10-r0: task do_package_write_rpm: Started
#12 2734.4 2021-03-30 01:14:29 - INFO - NOTE: recipe lzo-native-2.10-r0:
task do_prepare_recipe_sysroot: Started
#12 2734.8 2021-03-30 01:14:30 - INFO - NOTE: recipe lzo-native-2.10-r0:
task do_prepare_recipe_sysroot: Succeeded
#12 2734.9 2021-03-30 01:14:30 - INFO - NOTE: Running task 2441 of 4982
(virtual:native:/home/pi/docker-meta-raspberrypi/layers/poky/meta/recipes-support/lzo/lzo_2.10.bb:do_configure)
#12 2735.0 2021-03-30 01:14:30 - ERROR - ERROR: linux-libc-headers-5.10-r0
do_package_write_rpm: Error executing a python function in exec_python_func()
autogenerated:
#12 2735.0 2021-03-30 01:14:30 - ERROR -
#12 2735.0 2021-03-30 01:14:30 - ERROR - The stack trace of python calls
that resulted in this exception/failure was:
#12 2735.0 2021-03-30 01:14:30 - ERROR - File: 'exec_python_func()
autogenerated', lineno: 2, function: <module>
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0001:
#12 2735.0 2021-03-30 01:14:30 - ERROR - *** 0002:write_specfile(d)
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0003:
#12 2735.0 2021-03-30 01:14:30 - ERROR - File:
'/home/pi/docker-meta-raspberrypi/layers/poky/meta/classes/package_rpm.bbclass',
lineno: 342, function: write_specfile
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0338:
localdata.setVar('PKG', pkgname)
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0339:
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0340:
localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + pkg)
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0341:
#12 2735.0 2021-03-30 01:14:30 - ERROR - *** 0342: conffiles =
get_conffiles(pkg, d)
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0343: dirfiles =
localdata.getVar('DIRFILES')
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0344: if dirfiles is not
None:
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0345: dirfiles =
dirfiles.split()
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0346:
#12 2735.0 2021-03-30 01:14:30 - ERROR - File:
'/home/pi/docker-meta-raspberrypi/layers/poky/meta/classes/package.bbclass',
lineno: 304, function: get_conffiles
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0300:def get_conffiles(pkg, d):
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0301: pkgdest =
d.getVar('PKGDEST')
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0302: root =
os.path.join(pkgdest, pkg)
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0303: cwd = os.getcwd()
#12 2735.0 2021-03-30 01:14:30 - ERROR - *** 0304: os.chdir(root)
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0305:
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0306: conffiles =
d.getVar('CONFFILES_%s' % pkg);
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0307: if conffiles == None:
#12 2735.0 2021-03-30 01:14:30 - ERROR - 0308: conffiles =
d.getVar('CONFFILES')
#12 2735.0 2021-03-30 01:14:30 - ERROR - Exception: FileNotFoundError:
[Errno 2] No such file or directory:
'/home/pi/docker-meta-raspberrypi/build/tmp/work/arm1176jzfshf-vfp-poky-linux-gnueabi/linux-libc-headers/5.10-r0/packages-split/linux-libc-headers-src'
A fully reproducible build can be found at
https://github.com/tewarid/docker-meta-raspberrypi/tree/ee083820c565da646b745bc25e439c13f3205dbe.
Please see build instructions in the readme.
I can’t seem to find a root cause and don’t have the time to look into this any
further.
> On 29 Mar 2021, at 20:13, Devendra Tewari <[email protected]> wrote:
>
> Will do. Thanks.
>
>> On 29 Mar 2021, at 20:10, Andre McCurdy <[email protected]> wrote:
>>
>> On Mon, Mar 29, 2021 at 4:07 PM Devendra Tewari
>> <[email protected]> wrote:
>>>
>>> I did try that but got an error that does not happen when we try os.rename
>>> first. I'll try to reproduce it again.
>>>
>>> I suspect there may be subtle differences in os.rename vs shutil.move with
>>> respect to what happens when origin and/or destination do not exist or are
>>> invalid.
>>
>> If there are subtle issues which you don't yet understand then that's
>> all the more reason for not hiding the new shutil.move code behind a
>> test which will pass for 99.9% of users. Please figure that out before
>> sending another version of the patch.
>>
>>>> On 29 Mar 2021, at 20:00, Andre McCurdy <[email protected]> wrote:
>>>>
>>>> On Mon, Mar 29, 2021 at 3:45 PM Devendra Tewari
>>>> <[email protected]> wrote:
>>>>>
>>>>> Thanks! My bad. The example I looked up in Python docs had a break and I
>>>>> just realized it was a looping example.
>>>>>
>>>>> Here’s the updated patch (or should I submit it again via git send-email?)
>>>>
>>>> It would be better to use shutil.move unconditionally in all cases,
>>>> rather than have a separate shutil.move code path which only gets
>>>> tested by people doing incremental builds in docker.
>>>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#150072):
https://lists.openembedded.org/g/openembedded-core/message/150072
Mute This Topic: https://lists.openembedded.org/mt/81698791/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
