Hi Alejandro, Am Mi., 12. Sep. 2018 um 23:03 Uhr schrieb Alejandro Hernandez <alejandro.enedino.hernandez-samani...@xilinx.com>: > > Hello Jens, > > > On 9/12/2018 4:53 AM, Jens Rehsack wrote: > > > > Am 11.09.2018 um 20:56 schrieb Alejandro Enedino Hernandez Samaniego > <alejandro.enedino.hernandez-samani...@xilinx.com>: > > Hey Jens, > > Hey Alejandro, > > On 09/10/2018 11:58 PM, Jens Rehsack wrote: > > > > Am 10.09.2018 um 23:33 schrieb Alejandro Enedino Hernandez Samaniego > <alejandro.enedino.hernandez-samani...@xilinx.com>: > > Hey Jens, > > > As I explained before, when you create a manifest for python (target), it > uses the native build as base (it literally runs the native python that was > just built), it is assumed its the same version as target and contains all > the modules provided by upstream, otherwise the missing modules cannot be > checked for dependencies, and the manifest becomes incoherent, so its not an > option to have an incomplete python native build. > > In that case, uuid for target never gets deployed, but it is. And I didn't > see any packaging issues for `python3` nor for `nativesdk-python3` > > > > I don't see what that has to do with anything, fixing the native build should > not affect what gets deployed on target, thats exactly why we have a > manifest, so they user can decide what to install and what not to install. > > > The manifest isn't used for python3-native. > You try to argue whether there is an error to be fixed - and I don't agree. > Each fix requires effort - and that's why some errors will never become > fixed. When the impact is reasonable or high enough, fixes are more likely. > > So: before you try to force me into "that's all ugly und must be seriously > beautified just because ..." I will not do anything. > When you give me a sane reason, I try to understand and make a decision. > > > > Ok first of all, I am not forcing you to do anything, and I've never said its > all ugly, so please don't reply aggressively since I am just trying to help > you get your patch merged, and it needs to work correctly for that to happen, > this is a community and if you don't want to do something just don't do it, > we're all trying to make this work in the best possible way, whether we get > paid for it or do it for fun.
Sounded a bit different, thanks for clarification. > Second of all, I know the manifest is not used for python3-native, I never > said python3-native used the manifest, but I am trying to explain to you why > the fix is necessary, and I am even telling you what the fix is, so you don't > have to waste your time and effort on that, here it goes once again: > > > There are python3-native and python3 packages. > > The python3 package uses a manifest file, to go through the installed files > by the whole python installation from upstream, and separates it in several > packages, providing more granularity and giving users the possibility of > installing a trimmed version of python, only with what they require on their > image. > > I think we're good until here. > > The problem here comes because we don't control upstream python and it > changes with every release, so new files show up, for example what I > mentioned before sha3.*.so. > > Since that file is not on the manifest, during packaging, it will end up on > python3-misc, which contains basically everything that doesn't belong > anywhere else, although clearly this file in specific should belong in > python3-crypt. > > What this means for the user, is that he/she will install python3-crypt and > expect it to work at runtime, but this wont happen, and when sha3 gets > imported (or something that depends on it to run), it will fail, this will > cause a bug on our system, which we would need to fix (yes, fixing bugs takes > effort and time). > > At this point an usual fix would be to just install python3-misc as well, > because it contains the required library, but python3-misc also contains > other unnecessary stuff that the user doesn't need, but anyways an RDEPENDS > to python3-misc would somewhat solve it. The correct fix however, is to add > the sha3.*.so file to the manifest, on the python3-crypt package, that way, > it will get installed and the user would be able to use it correctly. This > would solve one problem, but its still not enough, because, I can see (from > the log), that there are some dependencies from other packages to sha3, which > means that yes, it will work correctly if we decide to install python3-crypt, > but if we just install one of the other packages that need it, it won't, > again, it will fail to import it and its the same story, its introducing a > bug into the system. Everything clear until here. > OK, up to this point I've explained how this will cause bugs in our system, > now, what does this fix that I mentioned has to do with everything?. > > There is a special task for the python3 package (not native), that creates a > manifest for us, this task tries to import every module from the manifest and > get its dependencies, for example, if the python3-numbers package needed the > sha3 library (this is just an example I really don't know if numbers needs > this), this will be picked up, and the python3-numbers package will RDEPEND > on python3-crypt, hence solving both the bugs I mentioned before, even before > they exist. I get lost at this point and probably getting that resolved - which task is it, how is the python3-manifest.json involved or better: how can this become automatically updated? > Now why is a fix to python3-native important for python3 target? > > Because of the way the previously mentioned script works, it calls > python3-native (it has to, because it can't use python from host for example, > because its not guaranteed its the same version of python3 target, and we > need it to contain exactly the same files) to import all modules in an > automated manner and get their dependencies, that's how we know if there are > new files and/or rdependencies between python3 packages. > > So what will happen if the python3-native build isn't complete?, whatever > file its missing, we wont be able to catch dependencies to it, causing them > to end up in python3-misc, causing the issue #1, and also, in any case, there > wont be an RDEPENDS created to python3-misc (which would solve issue #1), > sometimes files do belong in python3-misc, but we need to know that so we can > add an RDEPENDS to it, but again if we just add an RDEPENDS to python3-misc > we'll be causing issue/bug #2, which we won't catch, until someone actually > tries to run something on a running image and gets an import failure on their > application. > > > I'm not even asking you to fix the manifest, I am simply asking you to > provide a functional native package (in this case, functional means with all > modules built), so it doesn't break the current functionality that helps us > catch these bugs. I hope its clear now that I am not asking you to just > change something because I want "beautiful" code everywhere and I never was, > I am asking you to do it because your patch will introduce bugs into the > system, and I am trying to help you fix them. That's all an implication of one big unknown - that python3-manifest.json has to be upgraded by invoking a special task. Once I understand what's to be done, I'll add an appropriate note for future updaters. > Alejandro > > > > I know you didn't see any packaging issues, but that doesn't mean they don't > exist, just from the log I showed you, I can tell you that the python3-crypt > package is not created correctly, for example, if you do: > > IMAGE_INSTALL_append = " python3-crypt" > > Boot the image, run python3 and you try to import sha3, it will fail, because > the sha3*.so library won't be on the filesystem. > > > And thats because the sha3.*.so library was just introduced in this upgrade, > and our manifest isn't aware it exists, hence it'll end up on python3-misc > and you have just created an unnecessary dependency from python3-crypt to > python3-misc (and worse, a dependency which were not even aware of, until we > test it manually), which beats the whole purpose of the manifest. > > The do_package function is not gonna fail just because, so you won't see > errors, but the files will be packaged incorrectly, causing runtime errors as > a consequence, the create_manifest task tries to solve these runtime issues > before they happen. > > > That's something completely different. I'm not a python guy and I don't get > from the ChangeLog what I have to change in the manifest. > I can and will do such stuff when doing the perl update. For python, I ask > for review because of things I don't know. > > You can give me a reasonable list of such changes and I update the manifest > accordingly when I rebase the patch after Ross PGO patches are committed. > > Cheers, > > Alejandro > > > Yes you probably need a patch to look at the correct directories for the h > files, as well as a dependency to make the h files available on > recipe-sysroot-native. > > Please check the submission. > > > I did, its not checking inside recipe-sysroot-native. > > > I know. You could check the submission anyway and find the right patch and > then argue differently. > > Cheers > -- > Jens Rehsack - rehs...@gmail.com > > -- > _______________________________________________ > Openembedded-core mailing list > Openembedded-core@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/openembedded-core -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core
