I suspect this is an issue in upstream createrepo_c tool, which has code that
does not work across overlay file systems
if (g_rename(out_repo, old_repodata_path) == -1) {
g_debug("Old repodata doesn't exists: Cannot rename %s -> %s: %s",
out_repo, old_repodata_path, g_strerror(errno));
} else {
g_debug("Renamed %s -> %s", out_repo, old_repodata_path);
old_repodata_renamed = TRUE;
}
I’ll try patching and report an issue to upstream project.
> On 27 Apr 2021, at 08:57, Devendra Tewari <[email protected]> wrote:
>
> These patches are working well at my end, but I’m still seeing the following
> error in do_rootfs, when building incrementally in a container. I’m not sure
> if this is due to timing issues alluded to earlier or something else
> entirely. I do know for sure that this is not caused by the patches because
> it happens even without them.
>
> ERROR: core-image-base-1.0-r0 do_rootfs: Error executing a python function in
> exec_python_func() autogenerated:
>
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
> 0001:
> *** 0002:do_rootfs(d)
> 0003:
> File:
> '/home/pi/docker-meta-raspberrypi/layers/poky/meta/classes/image.bbclass',
> lineno: 247, function: do_rootfs
> 0243: progress_reporter.next_stage()
> 0244:
> 0245: # generate rootfs
> 0246: d.setVarFlag('REPRODUCIBLE_TIMESTAMP_ROOTFS', 'export', '1')
> *** 0247: create_rootfs(d, progress_reporter=progress_reporter,
> logcatcher=logcatcher)
> 0248:
> 0249: progress_reporter.finish()
> 0250:}
> 0251:do_rootfs[dirs] = "${TOPDIR}"
> File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/rootfs.py',
> lineno: 375, function: create_rootfs
> 0371:
> 0372: img_type = d.getVar('IMAGE_PKGTYPE')
> 0373:
> 0374: cls = get_class_for_type(img_type)
> *** 0375: cls(d, manifest_dir, progress_reporter, logcatcher).create()
> 0376: os.environ.clear()
> 0377: os.environ.update(env_bkp)
> 0378:
> 0379:
> File: '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/rootfs.py',
> lineno: 203, function: create
> 0199: if self.progress_reporter:
> 0200: self.progress_reporter.next_stage()
> 0201:
> 0202: # call the package manager dependent create method
> *** 0203: self._create()
> 0204:
> 0205: sysconfdir = self.image_rootfs + self.d.getVar('sysconfdir')
> 0206: bb.utils.mkdirhier(sysconfdir)
> 0207: with open(sysconfdir + "/version", "w+") as ver:
> File:
> '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/package_manager/rpm/rootfs.py',
> lineno: 70, function: _create
> 0066: rpm_pre_process_cmds =
> self.d.getVar('RPM_PREPROCESS_COMMANDS')
> 0067: rpm_post_process_cmds =
> self.d.getVar('RPM_POSTPROCESS_COMMANDS')
> 0068:
> 0069: # update PM index files
> *** 0070: self.pm.write_index()
> 0071:
> 0072: execute_pre_post_process(self.d, rpm_pre_process_cmds)
> 0073:
> 0074: if self.progress_reporter:
> File:
> '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/package_manager/rpm/__init__.py',
> lineno: 141, function: write_index
> 0137:
> 0138: def write_index(self):
> 0139: lockfilename = self.d.getVar('DEPLOY_DIR_RPM') + "/rpm.lock"
> 0140: lf = bb.utils.lockfile(lockfilename, False)
> *** 0141: RpmIndexer(self.d, self.rpm_repo_dir).write_index()
> 0142: bb.utils.unlockfile(lf)
> 0143:
> 0144: def insert_feeds_uris(self, feed_uris, feed_base_paths,
> feed_archs):
> 0145: from urllib.parse import urlparse
> File:
> '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/package_manager/rpm/__init__.py',
> lineno: 11, function: write_index
> 0007:from oe.package_manager import *
> 0008:
> 0009:class RpmIndexer(Indexer):
> 0010: def write_index(self):
> *** 0011: self.do_write_index(self.deploy_dir)
> 0012:
> 0013: def do_write_index(self, deploy_dir):
> 0014: if self.d.getVar('PACKAGE_FEED_SIGN') == '1':
> 0015: signer = get_signer(self.d,
> self.d.getVar('PACKAGE_FEED_GPG_BACKEND'))
> File:
> '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/package_manager/rpm/__init__.py',
> lineno: 20, function: do_write_index
> 0016: else:
> 0017: signer = None
> 0018:
> 0019: createrepo_c = bb.utils.which(os.environ['PATH'],
> "createrepo_c")
> *** 0020: result = create_index("%s --update -q %s" % (createrepo_c,
> deploy_dir))
> 0021: if result:
> 0022: bb.fatal(result)
> 0023:
> 0024: # Sign repomd
> File:
> '/home/pi/docker-meta-raspberrypi/layers/poky/meta/lib/oe/package_manager/__init__.py',
> lineno: 26, function: create_index
> 0022:def create_index(arg):
> 0023: index_cmd = arg
> 0024:
> 0025: bb.note("Executing '%s' ..." % index_cmd)
> *** 0026: result = subprocess.check_output(index_cmd,
> stderr=subprocess.STDOUT, shell=True).decode("utf-8")
> 0027: if result:
> 0028: bb.note(result)
> 0029:
> 0030:def opkg_query(cmd_output):
> File: '/usr/lib/python3.8/subprocess.py', lineno: 411, function: check_output
> 0407: # Explicitly passing input=None was previously equivalent
> to passing an
> 0408: # empty string. That is maintained here for backwards
> compatibility.
> 0409: kwargs['input'] = '' if kwargs.get('universal_newlines',
> False) else b''
> 0410:
> *** 0411: return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
> 0412: **kwargs).stdout
> 0413:
> 0414:
> 0415:class CompletedProcess(object):
> File: '/usr/lib/python3.8/subprocess.py', lineno: 512, function: run
> 0508: # We don't call process.wait() as .__exit__ does that
> for us.
> 0509: raise
> 0510: retcode = process.poll()
> 0511: if check and retcode:
> *** 0512: raise CalledProcessError(retcode, process.args,
> 0513: output=stdout, stderr=stderr)
> 0514: return CompletedProcess(process.args, retcode, stdout, stderr)
> 0515:
> 0516:
> Exception: subprocess.CalledProcessError: Command
> '/home/pi/docker-meta-raspberrypi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/recipe-sysroot-native/usr/bin/createrepo_c
> --update -q
> /home/pi/docker-meta-raspberrypi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo'
> returned non-zero exit status 1.
>
> Subprocess output:
> Critical: Cannot rename
> /home/pi/docker-meta-raspberrypi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo/.repodata/
> ->
> /home/pi/docker-meta-raspberrypi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/oe-rootfs-repo/repodata/:
> Directory not empty
>
> ERROR: Logfile of failure stored in:
> /home/pi/docker-meta-raspberrypi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/temp/log.do_rootfs.165072
> ERROR: Task
> (/home/pi/docker-meta-raspberrypi/layers/poky/meta/recipes-core/images/core-image-base.bb:do_rootfs)
> failed with exit code '1'
>
>
>> On 21 Apr 2021, at 18:08, Devendra Tewari <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Separate patches for bitbake and oe-core, as requested
>>
>> <0001-use-bb.utils.rename-to-rename-file-or-directory.patch><0001-use-bb.utils.rename-to-rename-file-or-directory.patch>
>>
>> I've assumed each will be applied to the master-next branch on top of the
>> previous patch.
>>
>> Thanks,
>> Devendra
>>
>>> On 21 Apr 2021, at 16:21, Richard Purdie
>>> <[email protected]
>>> <mailto:[email protected]>> wrote:
>>>
>>> On Wed, 2021-04-21 at 16:15 -0300, Devendra Tewari wrote:
>>>> That's a neat idea - if the exception handler catches OSError with error
>>>> number
>>>> 18 (Invalid cross-device link) in os.rename, it should attempt
>>>> shutil.move,
>>>> otherwise it should re-throw the exception with raise. A wrapper function
>>>> will make it easier to handle other situations as they arise. I'll update
>>>> the patch.
>>>
>>> Sounds good. Could you also please split the patch into two, one for bitbake
>>> and one for OE-Core since they are two different repositories.
>>>
>>> Cheers,
>>>
>>> Richard
>>>
>>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#151035):
https://lists.openembedded.org/g/openembedded-core/message/151035
Mute This Topic: https://lists.openembedded.org/mt/81698791/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
