Hi Alejandro,
Am Mi., 12. Sep. 2018 um 23:03 Uhr schrieb Alejandro Hernandez
<[email protected]>:
Hello Jens,
On 9/12/2018 4:53 AM, Jens Rehsack wrote:
Am 11.09.2018 um 20:56 schrieb Alejandro Enedino Hernandez Samaniego
<[email protected]>:
Hey Jens,
Hey Alejandro,
On 09/10/2018 11:58 PM, Jens Rehsack wrote:
Am 10.09.2018 um 23:33 schrieb Alejandro Enedino Hernandez Samaniego
<[email protected]>:
Hey Jens,
As I explained before, when you create a manifest for python (target), it uses
the native build as base (it literally runs the native python that was just
built), it is assumed its the same version as target and contains all the
modules provided by upstream, otherwise the missing modules cannot be checked
for dependencies, and the manifest becomes incoherent, so its not an option to
have an incomplete python native build.
In that case, uuid for target never gets deployed, but it is. And I didn't see
any packaging issues for `python3` nor for `nativesdk-python3`
I don't see what that has to do with anything, fixing the native build should
not affect what gets deployed on target, thats exactly why we have a manifest,
so they user can decide what to install and what not to install.
The manifest isn't used for python3-native.
You try to argue whether there is an error to be fixed - and I don't agree.
Each fix requires effort - and that's why some errors will never become fixed.
When the impact is reasonable or high enough, fixes are more likely.
So: before you try to force me into "that's all ugly und must be seriously
beautified just because ..." I will not do anything.
When you give me a sane reason, I try to understand and make a decision.
Ok first of all, I am not forcing you to do anything, and I've never said its
all ugly, so please don't reply aggressively since I am just trying to help you
get your patch merged, and it needs to work correctly for that to happen, this
is a community and if you don't want to do something just don't do it, we're
all trying to make this work in the best possible way, whether we get paid for
it or do it for fun.
Sounded a bit different, thanks for clarification.
Second of all, I know the manifest is not used for python3-native, I never said
python3-native used the manifest, but I am trying to explain to you why the fix
is necessary, and I am even telling you what the fix is, so you don't have to
waste your time and effort on that, here it goes once again:
There are python3-native and python3 packages.
The python3 package uses a manifest file, to go through the installed files by
the whole python installation from upstream, and separates it in several
packages, providing more granularity and giving users the possibility of
installing a trimmed version of python, only with what they require on their
image.
I think we're good until here.
The problem here comes because we don't control upstream python and it changes
with every release, so new files show up, for example what I mentioned before
sha3.*.so.
Since that file is not on the manifest, during packaging, it will end up on
python3-misc, which contains basically everything that doesn't belong anywhere
else, although clearly this file in specific should belong in python3-crypt.
What this means for the user, is that he/she will install python3-crypt and
expect it to work at runtime, but this wont happen, and when sha3 gets imported
(or something that depends on it to run), it will fail, this will cause a bug
on our system, which we would need to fix (yes, fixing bugs takes effort and
time).
At this point an usual fix would be to just install python3-misc as well,
because it contains the required library, but python3-misc also contains other
unnecessary stuff that the user doesn't need, but anyways an RDEPENDS to
python3-misc would somewhat solve it. The correct fix however, is to add the
sha3.*.so file to the manifest, on the python3-crypt package, that way, it will
get installed and the user would be able to use it correctly. This would solve
one problem, but its still not enough, because, I can see (from the log), that
there are some dependencies from other packages to sha3, which means that yes,
it will work correctly if we decide to install python3-crypt, but if we just
install one of the other packages that need it, it won't, again, it will fail
to import it and its the same story, its introducing a bug into the system.
Everything clear until here.
OK, up to this point I've explained how this will cause bugs in our system,
now, what does this fix that I mentioned has to do with everything?.
There is a special task for the python3 package (not native), that creates a
manifest for us, this task tries to import every module from the manifest and
get its dependencies, for example, if the python3-numbers package needed the
sha3 library (this is just an example I really don't know if numbers needs
this), this will be picked up, and the python3-numbers package will RDEPEND on
python3-crypt, hence solving both the bugs I mentioned before, even before they
exist.
I get lost at this point and probably getting that resolved - which
task is it, how is the python3-manifest.json involved or better: how
can this become automatically updated?
Now why is a fix to python3-native important for python3 target?
Because of the way the previously mentioned script works, it calls
python3-native (it has to, because it can't use python from host for example,
because its not guaranteed its the same version of python3 target, and we need
it to contain exactly the same files) to import all modules in an automated
manner and get their dependencies, that's how we know if there are new files
and/or rdependencies between python3 packages.
So what will happen if the python3-native build isn't complete?, whatever file
its missing, we wont be able to catch dependencies to it, causing them to end
up in python3-misc, causing the issue #1, and also, in any case, there wont be
an RDEPENDS created to python3-misc (which would solve issue #1), sometimes
files do belong in python3-misc, but we need to know that so we can add an
RDEPENDS to it, but again if we just add an RDEPENDS to python3-misc we'll be
causing issue/bug #2, which we won't catch, until someone actually tries to run
something on a running image and gets an import failure on their application.
I'm not even asking you to fix the manifest, I am simply asking you to provide a
functional native package (in this case, functional means with all modules built), so it
doesn't break the current functionality that helps us catch these bugs. I hope its clear
now that I am not asking you to just change something because I want
"beautiful" code everywhere and I never was, I am asking you to do it because
your patch will introduce bugs into the system, and I am trying to help you fix them.
That's all an implication of one big unknown - that
python3-manifest.json has to be upgraded by invoking a special task.
Once I understand what's to be done, I'll add an appropriate note for
future updaters.
