Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

I was asked about the createrepo_c patch in private. Replying here for closure. I had reported the issue upstream at https://github.com/rpm-software-management/createrepo_c/issues/266 and it has been fixed. Poky kirkstone and mainline should have the fix. Kind Regards, Devendra > On 28 Apr 202

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

Understood - carrying patches is a burden for sure. I’ll maintain patches for my own narrow use cases at my end, and perhaps submit them upstream. Thanks, Devendra > On 28 Apr 2021, at 18:42, Richard Purdie > wrote: > > On Wed, 2021-04-28 at 07:00 -0300, Devendra Tewari wrote: >> The gist of

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

On Wed, 2021-04-28 at 07:00 -0300, Devendra Tewari wrote: > The gist of it is that Docker uses the overlay2 storage driver by default,  > previously it used aufs. Both drivers have the same limitation in rename,  > where file rename across layers is handled by moving, but directory rename > across

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

The gist of it is that Docker uses the overlay2 storage driver by default, previously it used aufs. Both drivers have the same limitation in rename, where file rename across layers is handled by moving, but directory rename across layers results in error EXDEV. It is up to user space to handle t

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

Looks like an old issue with overlayfs - https://github.com/moby/moby/issues/25409. > On 27 Apr 2021, at 19:44, Devendra Tewari wrote: > > You’re absolutely right - I’ll investigate whether this is something the > container overlay filesystem should be handling. Thanks! > >> On 27 Apr 2021, a

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

You’re absolutely right - I’ll investigate whether this is something the container overlay filesystem should be handling. Thanks! > On 27 Apr 2021, at 19:39, Richard Purdie > wrote: > > On Tue, 2021-04-27 at 11:59 -0300, Devendra Tewari wrote: >> I suspect this is an issue in upstream createre

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

On Tue, 2021-04-27 at 11:59 -0300, Devendra Tewari wrote: > I suspect this is an issue in upstream createrepo_c tool, which has code that > does not work across overlay > file systems > >     if (g_rename(out_repo, old_repodata_path) == -1) { >         g_debug("Old repodata doesn't exists: Cannot

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

I suspect this is an issue in upstream createrepo_c tool, which has code that does not work across overlay file systems if (g_rename(out_repo, old_repodata_path) == -1) { g_debug("Old repodata doesn't exists: Cannot rename %s -> %s: %s", out_repo, old_repodata_path, g_

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

These patches are working well at my end, but I’m still seeing the following error in do_rootfs, when building incrementally in a container. I’m not sure if this is due to timing issues alluded to earlier or something else entirely. I do know for sure that this is not caused by the patches becau

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

Separate patches for bitbake and oe-core, as requested I've assumed each will be applied to the master-next branch on top of the previous patch. Thanks, Devendra > On 21 Apr 2021, at 16:21, Richard Purdie > wrote: > > On Wed, 2021-04-21 at 16:15 -0300, Devendra Tewari wrote: >> That's a ne

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

On Wed, 2021-04-21 at 16:15 -0300, Devendra Tewari wrote: > That's a neat idea - if the exception handler catches OSError with error > number  > 18 (Invalid cross-device link) in os.rename, it should attempt shutil.move,  > otherwise it should re-throw the exception with raise. A wrapper function 

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

That's a neat idea - if the exception handler catches OSError with error number 18 (Invalid cross-device link) in os.rename, it should attempt shutil.move, otherwise it should re-throw the exception with raise. A wrapper function will make it easier to handle other situations as they arise. I'll

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

On Wed, 2021-04-21 at 11:43 -0300, Devendra Tewari wrote: > I created an image for Raspberry Pi Zero W from code at  > https://github.com/tewarid/docker-meta-raspberrypi/tree/sstate, using Docker > for macOS, and on Ubuntu 20.04- > LTS, and wrote it out to an SDCard using bmaptool. Booted and che

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

On Wed, Apr 21, 2021 at 7:43 AM Devendra Tewari wrote: > > I created an image for Raspberry Pi Zero W from code at > https://github.com/tewarid/docker-meta-raspberrypi/tree/sstate, using Docker > for macOS, and on Ubuntu 20.04-LTS, and wrote it out to an SDCard using > bmaptool. Booted and chec

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

I created an image for Raspberry Pi Zero W from code at https://github.com/tewarid/docker-meta-raspberrypi/tree/sstate , using Docker for macOS, and on Ubuntu 20.04-LTS, and wrote it out to an SDCard using bmaptool. Booted and chec

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

None of my images are bootable when this patch is applied, because symlinks that busybox should provide arent created in the image. This patch should be looked into a bit further, how was it tested ? On Mon, Apr 19, 2021 at 11:43 AM Devendra Tewari wrote: > > > > > On 30 Mar 2021, at 08:55, Deve

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

> On 30 Mar 2021, at 08:55, Devendra Tewari wrote: > > I understand that parallel tasks can result in such issues, but would expect > some kind of dependency graph that would prevent them from happening. I wish > I had more time to understand the issue fully. > > Here’s the last version of m

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

>From 57b633a93bd91b7b1aa21cce5ac7997b958ca917 Mon Sep 17 00:00:00 2001 From: Devendra Tewari Date: Thu, 1 Apr 2021 16:07:25 -0300 Subject: [PATCH] Use shutil.move when os.rename fails Incremental build in Docker fails with OSError: [Errno 18] Invalid cross-device link When source and destinati

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

Here's the correct link https://bugzilla.yoctoproject.org/show_bug.cgi?id=14301. I'll resubmit the patch referencing the bug in the commit message. Thanks. > Em 30 de mar. de 2021, à(s) 18:59, Denys Dmytriyenko > escreveu: > > The link is for a 10-year old bug, probably not what you wanted.

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

The link is for a 10-year old bug, probably not what you wanted. Also, if a patch fixes existing bug in bugzilla, it needs to reference it in commit message as well - [YOCTO#1234567] On Mon, Mar 29, 2021 at 12:37:45PM -0300, Devendra Tewari wrote: > Also, this is due to https://bugzilla.yoctopro

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

I understand that parallel tasks can result in such issues, but would expect some kind of dependency graph that would prevent them from happening. I wish I had more time to understand the issue fully. Here’s the last version of my patch that still uses os.rename, and on error uses shutil.move,

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

On Mon, 2021-03-29 at 16:00 -0700, Andre McCurdy wrote: > On Mon, Mar 29, 2021 at 3:45 PM Devendra Tewari > wrote: > > > > Thanks! My bad. The example I looked up in Python docs had a break and I > > just realized it was a looping example. > > > > Here’s the updated patch (or should I submit it

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

With the following patch From f29ec67239094a256fcfc119fb75be90923d3448 Mon Sep 17 00:00:00 2001 From: Devendra Tewari Date: Mon, 29 Mar 2021 21:11:56 -0300 Subject: [PATCH] Use shutil.move to rename sstate Incremental build in Docker fails with OSError: [Errno 18] Invalid cross-device link Whe

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

Will do. Thanks. > On 29 Mar 2021, at 20:10, Andre McCurdy wrote: > > On Mon, Mar 29, 2021 at 4:07 PM Devendra Tewari > wrote: >> >> I did try that but got an error that does not happen when we try os.rename >> first. I'll try to reproduce it again. >> >> I suspect there may be subtle differ

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

On Mon, Mar 29, 2021 at 4:07 PM Devendra Tewari wrote: > > I did try that but got an error that does not happen when we try os.rename > first. I'll try to reproduce it again. > > I suspect there may be subtle differences in os.rename vs shutil.move with > respect to what happens when origin and/

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

I did try that but got an error that does not happen when we try os.rename first. I'll try to reproduce it again. I suspect there may be subtle differences in os.rename vs shutil.move with respect to what happens when origin and/or destination do not exist or are invalid. > On 29 Mar 2021, at

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

On Mon, Mar 29, 2021 at 3:45 PM Devendra Tewari wrote: > > Thanks! My bad. The example I looked up in Python docs had a break and I just > realized it was a looping example. > > Here’s the updated patch (or should I submit it again via git send-email?) It would be better to use shutil.move uncon

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

Thanks! My bad. The example I looked up in Python docs had a break and I just realized it was a looping example. Here’s the updated patch (or should I submit it again via git send-email?) >From 9deb390dcdcaef66cec2fae39454c7fb3c81c4e4 Mon Sep 17 00:00:00 2001 From: Devendra Tewari Date: Mon, 29

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

On Mon, 2021-03-29 at 12:14 -0300, Devendra Tewari wrote: > --- >  meta/classes/sstate.bbclass | 26 ++ >  1 file changed, 22 insertions(+), 4 deletions(-) > > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass > index f579168162..f94aa96d70 100644 > ---

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

Also, this is due to https://bugzilla.yoctoproject.org/show_bug.cgi?id=1430. > On 29 Mar 2021, at 12:35, Devendra Tewari wrote: > > Sure. > > Would the following commit message be sufficient? > >Use shutil.move when os.rename fails > >Incremental build in Docker fails with > >OSE

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

Sure. Would the following commit message be sufficient? Use shutil.move when os.rename fails Incremental build in Docker fails with OSError: [Errno 18] Invalid cross-device link When source and destination are on different overlay filesystems. This change handles the error

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

Yes please quote a bit from the python manpage [1] - I certainly see the difference (and the edge cases where os.rename might fail), but that should be documented as part of the commit message [1] https://docs.python.org/3/library/shutil.html#shutil.move On 29.03.21 17:21, Bruce Ashfield wrote

Re: [OE-core] [PATCH] Use shutil.move when os.rename fails

Can you document the cases that os.rename() is failing ? And also why would we expect the shutil.move() to work in those cases ? If a change like this cases issues in the future, we need that extra information in the commit head for proper triage. Bruce On Mon, Mar 29, 2021 at 11:16 AM Devendra

[OE-core] [PATCH] Use shutil.move when os.rename fails

--- meta/classes/sstate.bbclass | 26 ++ 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index f579168162..f94aa96d70 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -384,