Re: [OE-core] [PATCH] package_rpm: Fix backtrace on missing directory.
This was the commit that introduced the DIRFILES variable: https://git.yoctoproject.org/poky/commit/?id=2f42ef8d8fb1febf28252b98884cebabc931f720 Given that it's undocumented, untested, rpm-specific, and currently broken, and looks like a workaround for security label conflicts elsewhere I'm inclined to remove the code paths that use it altogether. Alex On Sun, 25 Feb 2024 at 19:34, Jeremy Puhlman via lists.openembedded.org wrote: > > > > On 2/25/2024 10:32 AM, Jeremy Puhlman wrote: > > > > > > On 2/25/2024 9:22 AM, Jeremy Puhlman wrote: > >> > >> > >> On 2/24/2024 2:13 PM, Richard Purdie wrote: > >>> On Fri, 2024-02-23 at 12:33 -0800, Jeremy Puhlman via > >>> lists.openembedded.org wrote: > It says usr//usr in the comment, but when I ran on a clean build it > started complaining about opt//opt as well. Its not really clear why > it is looking for those directories, and I am sure this is not likely > the correct solution, but I wanted to get it out here so someone who > has kicked this area of code might know what may be happening. > >>> Unfortunately there are layers of different pieces which could be the > >>> issue, such as pseudo in combination with the host distro libraries and > >>> uninative. > >>> > >>> Can you create a test case which reproduces the issue? > >> Yeah, I was going to try to get to it, but time got away from me. > >> Hopefully I should > >> be able to get some time later today. > > > > Trying to reduce the recipe to bare bones, I narrowed it down the the > > DIRFILES setting. > > Deep in the top level class configuration for the layer, it sets > > DIRFILES="1", which looks > > wrong given the code. OTOH, nothing else in any of the layers we are > > using sets the value > > to anything, so what ever recent changes that were made there are > > untested by anything. > > > > Turning that off should resolve my issue, but it looks like that code > > path may be vestigial at > > this point. > > > As an aside, adding: > DIRFILES = "${bindir}" > To something like the bash recipe, reproduces the behavior. > > >> > >>> Does it always > >>> happen with this recipe? > >> Yes. Originally it was happening in our distro setup, minimizing it > >> to just poky and the > >> meta-oe/meta-python, it happens with the recipe with the default > >> distro as well. > >> > >>> Does it happen on different host distros? > >> I have not tried that, so far it was just rocky 8. > >> > >>> Are > >>> you using uninative? > >> Yes. It happens with the default poky distro, too. > >> > >>> What about buildtools-tarball? > >> We always use buildtools-extended-tarball, to isolate as much as > >> possible from the host. > >> > >>> > >>> Cheers, > >>> > >>> Richard > >> > > > > -- > Jeremy Puhlman > jpuhl...@mvista.com > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#196200): https://lists.openembedded.org/g/openembedded-core/message/196200 Mute This Topic: https://lists.openembedded.org/mt/104536226/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] package_rpm: Fix backtrace on missing directory.
On 2/25/2024 10:32 AM, Jeremy Puhlman wrote: On 2/25/2024 9:22 AM, Jeremy Puhlman wrote: On 2/24/2024 2:13 PM, Richard Purdie wrote: On Fri, 2024-02-23 at 12:33 -0800, Jeremy Puhlman via lists.openembedded.org wrote: It says usr//usr in the comment, but when I ran on a clean build it started complaining about opt//opt as well. Its not really clear why it is looking for those directories, and I am sure this is not likely the correct solution, but I wanted to get it out here so someone who has kicked this area of code might know what may be happening. Unfortunately there are layers of different pieces which could be the issue, such as pseudo in combination with the host distro libraries and uninative. Can you create a test case which reproduces the issue? Yeah, I was going to try to get to it, but time got away from me. Hopefully I should be able to get some time later today. Trying to reduce the recipe to bare bones, I narrowed it down the the DIRFILES setting. Deep in the top level class configuration for the layer, it sets DIRFILES="1", which looks wrong given the code. OTOH, nothing else in any of the layers we are using sets the value to anything, so what ever recent changes that were made there are untested by anything. Turning that off should resolve my issue, but it looks like that code path may be vestigial at this point. As an aside, adding: DIRFILES = "${bindir}" To something like the bash recipe, reproduces the behavior. Does it always happen with this recipe? Yes. Originally it was happening in our distro setup, minimizing it to just poky and the meta-oe/meta-python, it happens with the recipe with the default distro as well. Does it happen on different host distros? I have not tried that, so far it was just rocky 8. Are you using uninative? Yes. It happens with the default poky distro, too. What about buildtools-tarball? We always use buildtools-extended-tarball, to isolate as much as possible from the host. Cheers, Richard -- Jeremy Puhlman jpuhl...@mvista.com -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#196145): https://lists.openembedded.org/g/openembedded-core/message/196145 Mute This Topic: https://lists.openembedded.org/mt/104536226/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] package_rpm: Fix backtrace on missing directory.
On 2/25/2024 9:22 AM, Jeremy Puhlman wrote: On 2/24/2024 2:13 PM, Richard Purdie wrote: On Fri, 2024-02-23 at 12:33 -0800, Jeremy Puhlman via lists.openembedded.org wrote: It says usr//usr in the comment, but when I ran on a clean build it started complaining about opt//opt as well. Its not really clear why it is looking for those directories, and I am sure this is not likely the correct solution, but I wanted to get it out here so someone who has kicked this area of code might know what may be happening. Unfortunately there are layers of different pieces which could be the issue, such as pseudo in combination with the host distro libraries and uninative. Can you create a test case which reproduces the issue? Yeah, I was going to try to get to it, but time got away from me. Hopefully I should be able to get some time later today. Trying to reduce the recipe to bare bones, I narrowed it down the the DIRFILES setting. Deep in the top level class configuration for the layer, it sets DIRFILES="1", which looks wrong given the code. OTOH, nothing else in any of the layers we are using sets the value to anything, so what ever recent changes that were made there are untested by anything. Turning that off should resolve my issue, but it looks like that code path may be vestigial at this point. Does it always happen with this recipe? Yes. Originally it was happening in our distro setup, minimizing it to just poky and the meta-oe/meta-python, it happens with the recipe with the default distro as well. Does it happen on different host distros? I have not tried that, so far it was just rocky 8. Are you using uninative? Yes. It happens with the default poky distro, too. What about buildtools-tarball? We always use buildtools-extended-tarball, to isolate as much as possible from the host. Cheers, Richard -- Jeremy Puhlman jpuhl...@mvista.com -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#196144): https://lists.openembedded.org/g/openembedded-core/message/196144 Mute This Topic: https://lists.openembedded.org/mt/104536226/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] package_rpm: Fix backtrace on missing directory.
On 2/24/2024 2:13 PM, Richard Purdie wrote: On Fri, 2024-02-23 at 12:33 -0800, Jeremy Puhlman via lists.openembedded.org wrote: It says usr//usr in the comment, but when I ran on a clean build it started complaining about opt//opt as well. Its not really clear why it is looking for those directories, and I am sure this is not likely the correct solution, but I wanted to get it out here so someone who has kicked this area of code might know what may be happening. Unfortunately there are layers of different pieces which could be the issue, such as pseudo in combination with the host distro libraries and uninative. Can you create a test case which reproduces the issue? Yeah, I was going to try to get to it, but time got away from me. Hopefully I should be able to get some time later today. Does it always happen with this recipe? Yes. Originally it was happening in our distro setup, minimizing it to just poky and the meta-oe/meta-python, it happens with the recipe with the default distro as well. Does it happen on different host distros? I have not tried that, so far it was just rocky 8. Are you using uninative? Yes. It happens with the default poky distro, too. What about buildtools-tarball? We always use buildtools-extended-tarball, to isolate as much as possible from the host. Cheers, Richard -- Jeremy Puhlman jpuhl...@mvista.com -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#196143): https://lists.openembedded.org/g/openembedded-core/message/196143 Mute This Topic: https://lists.openembedded.org/mt/104536226/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] package_rpm: Fix backtrace on missing directory.
On Fri, 2024-02-23 at 12:33 -0800, Jeremy Puhlman via lists.openembedded.org wrote: > It says usr//usr in the comment, but when I ran on a clean build it > started complaining about opt//opt as well. Its not really clear why > it is looking for those directories, and I am sure this is not likely > the correct solution, but I wanted to get it out here so someone who > has kicked this area of code might know what may be happening. Unfortunately there are layers of different pieces which could be the issue, such as pseudo in combination with the host distro libraries and uninative. Can you create a test case which reproduces the issue? Does it always happen with this recipe? Does it happen on different host distros? Are you using uninative? What about buildtools-tarball? Cheers, Richard -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#196135): https://lists.openembedded.org/g/openembedded-core/message/196135 Mute This Topic: https://lists.openembedded.org/mt/104536226/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] package_rpm: Fix backtrace on missing directory.
On Fri, 23 Feb 2024 at 23:58, Jeremy Puhlman wrote: > Is there a preferred way to do that? I mention in the commit message > that is not > the right way to handle it and followed up with a reply to the patch > mentioning > as such. You can simply use 'git send-email --rfc' to put [RFC PATCH] into the subject. And have the word 'workaround' in the header too. Alex -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#196124): https://lists.openembedded.org/g/openembedded-core/message/196124 Mute This Topic: https://lists.openembedded.org/mt/104536226/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH] package_rpm: Fix backtrace on missing directory.
On 2/23/2024 1:20 PM, Alexander Kanavin wrote: Somewhere in the overall function the paths aren't formed correctly, but this patch is very much not a correct fix either, and should be marked as such. Is there a preferred way to do that? I mention in the commit message that is not the right way to handle it and followed up with a reply to the patch mentioning as such. If you can provide a minimal reproducer recipe, that would help; I can't say what is wrong otherwise. I will try and reduce it down later this evening, but the recipe in question is here: https://github.com/MontaVista-OpenSourceTechnology/meta-qa/blob/master/meta-qa-framework/recipes-qatest/mvtest/mvtest_2.0.bb Using stock poky/meta-oe/meta-python and that repo causes the issue with mvtest. The recipe is fairly basic, nothing really crazy, which is what is baffling about the error. You might want to insert debugging statements to see where opt//opt formation is happening. Tried poking at that for a while and really didn't make much progress before I got pulled off to something else. Alex On Fri, 23 Feb 2024 at 21:31, Jeremy Puhlman via lists.openembedded.org wrote: From: "Jeremy A. Puhlman" This seems to work around the following issue. The recipe in quesiton places all of its content in /opt, and we turn on license collection globally. Systemd is turned on so usrmerge is also on. I doubt this is the correct way to deal with this as it is not clear why it is looking for /usr//usr in the first place. Not sure if its just a wierd edge case or what. File: '/build/../layers/poky/meta/classes-global/package_rpm.bbclass', lineno: 527, function: write_specfile 0523:spec_scriptlets_bottom.append('') 0524: 0525:# Now process files 0526:file_list = [] *** 0527:walk_files(root, file_list, conffiles, dirfiles) 0528:if not file_list and localdata.getVar('ALLOW_EMPTY', False) != "1": 0529:bb.note("Not creating empty RPM package for %s" % splitname) 0530:else: 0531:spec_files_bottom.append('%%files -n %s' % splitname) File: '/build/../layers/poky/meta/classes-global/package_rpm.bbclass', lineno: 249, function: walk_files 0245:target.append(get_attr(dir) + '%dir "' + escape_chars(p) + '"') 0246:else: 0247:# packages own only empty directories or explict directory. 0248:# This will prevent the overlapping of security permission. *** 0249:attr = get_attr(path) 0250:if path and not files and not dirs: 0251:target.append(attr + '%dir "' + escape_chars(path) + '"') 0252:elif path and path in dirfiles: 0253:target.append(attr + '%dir "' + escape_chars(path) + '"') File: '/build/../layers/poky/meta/classes-global/package_rpm.bbclass', lineno: 203, function: get_attr 0199:# of the walk, the isdir() test would then fail and the walk code would assume its a file 0200:# hence we check for the names in files too. 0201:for rootpath, dirs, files in os.walk(walkpath): 0202:def get_attr(path): *** 0203:stat_f = os.stat(rootpath + "/" + path, follow_symlinks=False) 0204:mode = stat.S_IMODE(stat_f.st_mode) 0205:try: 0206:owner = pwd.getpwuid(stat_f.st_uid).pw_name 0207:except Exception as e: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/build/tmp/work/corei7-64-montavista-linux/mvtest/2.0/packages-split/mvtest-lic/opt//opt' Signed-off-by: Jeremy A. Puhlman --- meta/classes-global/package_rpm.bbclass | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/meta/classes-global/package_rpm.bbclass b/meta/classes-global/package_rpm.bbclass index 2e3e4e8c79..a6885ac74e 100644 --- a/meta/classes-global/package_rpm.bbclass +++ b/meta/classes-global/package_rpm.bbclass @@ -200,6 +200,8 @@ python write_specfile () { # hence we check for the names in files too. for rootpath, dirs, files in os.walk(walkpath): def get_attr(path): +if not os.path.exists(rootpath + "/" + path): + return "" stat_f = os.stat(rootpath + "/" + path, follow_symlinks=False) mode = stat.S_IMODE(stat_f.st_mode) try: @@ -243,7 +245,7 @@ python write_specfile () { p = path + '/' + dir # All packages own the directories their files are in... target.append(get_attr(dir) + '%dir "' + escape_chars(p) + '"') -elif path: +elif path and os.path.exists(path): # packages own only empty directories or explict directory.
Re: [OE-core] [PATCH] package_rpm: Fix backtrace on missing directory.
Somewhere in the overall function the paths aren't formed correctly, but this patch is very much not a correct fix either, and should be marked as such. If you can provide a minimal reproducer recipe, that would help; I can't say what is wrong otherwise. You might want to insert debugging statements to see where opt//opt formation is happening. Alex On Fri, 23 Feb 2024 at 21:31, Jeremy Puhlman via lists.openembedded.org wrote: > > From: "Jeremy A. Puhlman" > > This seems to work around the following issue. The recipe in quesiton > places all of its content in /opt, and we turn on license collection > globally. Systemd is turned on so usrmerge is also on. I doubt this > is the correct way to deal with this as it is not clear why it is > looking for /usr//usr in the first place. Not sure if its just a > wierd edge case or what. > > File: > '/build/../layers/poky/meta/classes-global/package_rpm.bbclass', > lineno: 527, function: write_specfile > 0523:spec_scriptlets_bottom.append('') > 0524: > 0525:# Now process files > 0526:file_list = [] > *** 0527:walk_files(root, file_list, conffiles, dirfiles) > 0528:if not file_list and localdata.getVar('ALLOW_EMPTY', > False) != "1": > 0529:bb.note("Not creating empty RPM package for %s" % > splitname) > 0530:else: > 0531:spec_files_bottom.append('%%files -n %s' % > splitname) > File: > '/build/../layers/poky/meta/classes-global/package_rpm.bbclass', > lineno: 249, function: walk_files > 0245:target.append(get_attr(dir) + '%dir "' + > escape_chars(p) + '"') > 0246:else: > 0247:# packages own only empty directories or > explict directory. > 0248:# This will prevent the overlapping of > security permission. > *** 0249:attr = get_attr(path) > 0250:if path and not files and not dirs: > 0251:target.append(attr + '%dir "' + > escape_chars(path) + '"') > 0252:elif path and path in dirfiles: > 0253:target.append(attr + '%dir "' + > escape_chars(path) + '"') > File: > '/build/../layers/poky/meta/classes-global/package_rpm.bbclass', > lineno: 203, function: get_attr > 0199:# of the walk, the isdir() test would then fail and > the walk code would assume its a file > 0200:# hence we check for the names in files too. > 0201:for rootpath, dirs, files in os.walk(walkpath): > 0202:def get_attr(path): > *** 0203:stat_f = os.stat(rootpath + "/" + path, > follow_symlinks=False) > 0204:mode = stat.S_IMODE(stat_f.st_mode) > 0205:try: > 0206:owner = > pwd.getpwuid(stat_f.st_uid).pw_name > 0207:except Exception as e: > Exception: FileNotFoundError: [Errno 2] No such file or directory: > '/build/tmp/work/corei7-64-montavista-linux/mvtest/2.0/packages-split/mvtest-lic/opt//opt' > > Signed-off-by: Jeremy A. Puhlman > --- > meta/classes-global/package_rpm.bbclass | 6 -- > 1 file changed, 4 insertions(+), 2 deletions(-) > > diff --git a/meta/classes-global/package_rpm.bbclass > b/meta/classes-global/package_rpm.bbclass > index 2e3e4e8c79..a6885ac74e 100644 > --- a/meta/classes-global/package_rpm.bbclass > +++ b/meta/classes-global/package_rpm.bbclass > @@ -200,6 +200,8 @@ python write_specfile () { > # hence we check for the names in files too. > for rootpath, dirs, files in os.walk(walkpath): > def get_attr(path): > +if not os.path.exists(rootpath + "/" + path): > + return "" > stat_f = os.stat(rootpath + "/" + path, > follow_symlinks=False) > mode = stat.S_IMODE(stat_f.st_mode) > try: > @@ -243,7 +245,7 @@ python write_specfile () { > p = path + '/' + dir > # All packages own the directories their files are in... > target.append(get_attr(dir) + '%dir "' + escape_chars(p) > + '"') > -elif path: > +elif path and os.path.exists(path): > # packages own only empty directories or explict directory. > # This will prevent the overlapping of security permission. > attr = get_attr(path) > @@ -257,7 +259,7 @@ python write_specfile () { > p = path + '/' + file > if conffiles.count(p): > target.append(attr + '%config "' + escape_chars(p) + '"') > -else: > +elif attr: > target.append(attr + '"' + escape_chars(p) + '"') > > # Prevent the prerm/postrm scripts from being run during an upgrade > -- > 2.31.1 > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages
Re: [OE-core] [PATCH] package_rpm: Fix backtrace on missing directory.
It says usr//usr in the comment, but when I ran on a clean build it started complaining about opt//opt as well. Its not really clear why it is looking for those directories, and I am sure this is not likely the correct solution, but I wanted to get it out here so someone who has kicked this area of code might know what may be happening. On 2/23/2024 12:30 PM, Jeremy Puhlman via lists.openembedded.org wrote: From: "Jeremy A. Puhlman" This seems to work around the following issue. The recipe in quesiton places all of its content in /opt, and we turn on license collection globally. Systemd is turned on so usrmerge is also on. I doubt this is the correct way to deal with this as it is not clear why it is looking for /usr//usr in the first place. Not sure if its just a wierd edge case or what. File: '/build/../layers/poky/meta/classes-global/package_rpm.bbclass', lineno: 527, function: write_specfile 0523:spec_scriptlets_bottom.append('') 0524: 0525:# Now process files 0526:file_list = [] *** 0527:walk_files(root, file_list, conffiles, dirfiles) 0528:if not file_list and localdata.getVar('ALLOW_EMPTY', False) != "1": 0529:bb.note("Not creating empty RPM package for %s" % splitname) 0530:else: 0531:spec_files_bottom.append('%%files -n %s' % splitname) File: '/build/../layers/poky/meta/classes-global/package_rpm.bbclass', lineno: 249, function: walk_files 0245:target.append(get_attr(dir) + '%dir "' + escape_chars(p) + '"') 0246:else: 0247:# packages own only empty directories or explict directory. 0248:# This will prevent the overlapping of security permission. *** 0249:attr = get_attr(path) 0250:if path and not files and not dirs: 0251:target.append(attr + '%dir "' + escape_chars(path) + '"') 0252:elif path and path in dirfiles: 0253:target.append(attr + '%dir "' + escape_chars(path) + '"') File: '/build/../layers/poky/meta/classes-global/package_rpm.bbclass', lineno: 203, function: get_attr 0199:# of the walk, the isdir() test would then fail and the walk code would assume its a file 0200:# hence we check for the names in files too. 0201:for rootpath, dirs, files in os.walk(walkpath): 0202:def get_attr(path): *** 0203:stat_f = os.stat(rootpath + "/" + path, follow_symlinks=False) 0204:mode = stat.S_IMODE(stat_f.st_mode) 0205:try: 0206:owner = pwd.getpwuid(stat_f.st_uid).pw_name 0207:except Exception as e: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/build/tmp/work/corei7-64-montavista-linux/mvtest/2.0/packages-split/mvtest-lic/opt//opt' Signed-off-by: Jeremy A. Puhlman --- meta/classes-global/package_rpm.bbclass | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/meta/classes-global/package_rpm.bbclass b/meta/classes-global/package_rpm.bbclass index 2e3e4e8c79..a6885ac74e 100644 --- a/meta/classes-global/package_rpm.bbclass +++ b/meta/classes-global/package_rpm.bbclass @@ -200,6 +200,8 @@ python write_specfile () { # hence we check for the names in files too. for rootpath, dirs, files in os.walk(walkpath): def get_attr(path): +if not os.path.exists(rootpath + "/" + path): + return "" stat_f = os.stat(rootpath + "/" + path, follow_symlinks=False) mode = stat.S_IMODE(stat_f.st_mode) try: @@ -243,7 +245,7 @@ python write_specfile () { p = path + '/' + dir # All packages own the directories their files are in... target.append(get_attr(dir) + '%dir "' + escape_chars(p) + '"') -elif path: +elif path and os.path.exists(path): # packages own only empty directories or explict directory. # This will prevent the overlapping of security permission. attr = get_attr(path) @@ -257,7 +259,7 @@ python write_specfile () { p = path + '/' + file if conffiles.count(p): target.append(attr + '%config "' + escape_chars(p) + '"') -else: +elif attr: target.append(attr + '"' + escape_chars(p) + '"') # Prevent the prerm/postrm scripts from being run during an upgrade -- Jeremy Puhlman jpuhl...@mvista.com -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#196110): https://lists.openembedded.org/g/openembedded-core/message/196110 Mute This Topic: