Re: [gentoo-portage-dev] DB and binary dependency
On Friday 24 March 2006 15:24, tvali wrote: Unfortunately, your wrong. This only makes sure that you have the right slots to put your squares, triangles and circles in. It does not say that b(int,int) from the first lib actually does the same thing as b(int,int) from the second library. Sorry i thought about it for a moment, then left out later as broken design cases didnt seems so important. Versioning of distinct functions would be good, anyway not automatic. In many cases these things are not broken design. My example is, to make it clear. Unfortunately at the moment you do automatic dependencies, you MUST be correct between 99.9% and 100%. These cases are too common to disregard. Changing semantics are common enough. Sometimes the old semantics are just broken. While the above example is clearly broken design, this does happen enough in actual libraries in way more subtle ways. And that is disregarding the fact that the linux/elf ABI does not include argument lists in symbol linking. As such b(int,char) is indistinguishable from b(int,int). To overcome this C++ uses name mangling which creates names based on the signature. Ok, assuming broken design it's harder. Anyway, how would binary deps solve this? This seems to be humans work in all cases. We don't look at what packages do. We take a default heuristic, and let humans give the correct answer when the default is wrong. Anyway, as .h files have become as well as standard, you may show in code as well as in bin, which functions are needed. Except that the way header files are used is not standard. You know there is a reason for the way gcc handles precompiled header files. That is because of the way they work. Header files can have unpredictable interactions. Including A then B can be really really different from including B then A. While in most cases it isn't, it is in too many cases. Is there any advantage of bin dep over code dep is a question for me ..some here suggest that there is, i think that everything can be done as well with code deps. bin dep is not about how to get the dependencies. Binary dependencies specify what the dependencies have become after compilation. Binary dependencies basically strengthen the runtime dependencies, by taking the concrete dependencies used into account. With binary dependencies comes the need to verify them. This is because too many ebuilds do not use the version of a multislotted library that is expected. As such verifying the dependencies of a binary package gives the opportunity for developers to signal and fix this. I'm actually sure that this all can be calculated up from sourcecode and bindep would be after that a check if cpu didnt calculate something wrong :P Another question, how difficult it is and is it worth the time. Perhaps you should read up your knowledge of the C language. After you found that the C language is a mess, try C++, it makes things worse. After that's finished take a look at solving this problem for ALL languages. I actually know c++, but dont know c. Anyway, what i mean is that if you have .h files, you may make up automatic check if interfaces fit each other. I have to think about if c++ is a mess :) C++ is every bit as much a mess as C as it is a superset. The problem is actually partly in the C language and partly in the preprocessor. First the preprocessor is fragile, and can depend on arbitrary command line options (Defines). The language problem is that C/C++ has no notion of modules. file a.c can define a function as int a(char) and b.c can have its prototype as int a(wchar). The compiler will not complain about this, AND with the right defines it IS actually the same. The fundamental problem is that the compiler only knows about symbols that may be imported from other object files. It does not look at the actual object to look at the prototypes of the symbols defined there (nor do .o files outputted by gcc contain this information). Instead header files can be used to share this information. Often they do this in a broken way and errors similar to the above exist. Automatic source analysis for dependency calculation is a dead end. Even if you manage to find the proper interfaces (oops, the package had it's own gl.h instead of using the system one), you don't know anything about the semantics of those interfaces. Two things with the same name may very well have very different behaviours. I think it should be solved with: // system's header #include ... // own header #include ... It's not your code, you can't change it. And including two headers which define the same symbols can be disastrous. Ok -- i agree that there are several problems, which have to be solved before taking this topic up again and that they are possibly unsolvable in many cases and possibly too much work in many others. The thing is, with reasonable effort you are probably
Re: [gentoo-portage-dev] DB and binary dependency
On 24/03/06, Paul de Vrieze [EMAIL PROTECTED] wrote:
On Thursday 23 March 2006 21:35, tvali wrote:
BINSLOT is a new word for me.
Ok BINSLOT is normally slot. However in some cases packages are in the
same slot, but not binary compatible (like their libraries having a
different SONAME e.g. openssl-0.9.6 and 0.9.7 (iirc)).
Calculating after build is only for validating configurations. It allows
one to see whether one can merge binary package A on the current system
Ok, i'm starting to fully understand it :)
So it is something what could be helpful even on other linux systems
than gentoo?
Header files come from packages that should be mentioned in the
dependencies.
Yes i'm here thinking about my code-deps :)
Calculating dependencies automatically is diabolically hard. The fact that
we have a tree that largely works shows that predicting them works well.
The reason is that the dependencies are statically defined by the source
of the package. At those parts where people make mistakes, or the build
systems are broken, occur the dependencies could be verified by automatic
tools that check most dependencies. Doing that for native code is easiest
done by disecting the libraries and executables. For header files,
inclusion parsing is probably a good approach. But then look at what is
installed, not what was started with. The reason being that at what is
installed is no longer dynamic based on a configure script, but totally
static. The only disadvantage being that this only checks the current
combination of useflags+context. Many checks by many users would however
give a good test coverage.
If i can understand you now, then i would think that you didnt
understand me fully. I will write my script, then..
Not really, with a static library A and a static library B that uses the
static A, this dependency is only runtime when the interface of B exposes
the headers of A. If not, one could unmerge both the static library and
the headers and one could still compile against the library.
I meant headers as .h or .hpp files, which contain function headers.
They're needed, but they are in many cases, included in pack which
uses them, even if that means putting some .h's into all libs.
Ok, let me clarify. With a binary dependency I mean the actual
requirements of a package once it has been compiled. In general one could
say that given a depend atom sys-libs/db-4.* could be statisfied by
ebuilds in 5 slots: db-4.{0,1,2,3,4}. The sourcecode does not care which
one is used. However, code compiled agaist db-4.0.14 will not run against
any libdb-4.{1,2,3,4}.*. This is because these should libraries live in
different BINSLOT's (in other words, they are incompatible). As these
libraries have different SLOT's they can coexist though. For a binary
package therefore it is necessary to record the version of db it was
actually compiled against. As this might be very strict I also want to
say that any package within the same BINSLOT with a higher version than
the one compiled against also works (as it does).
Cant understand it. Do the *headers* change so much through versions?
Is there no backward-compability in so many cases?
Paul
--
Paul de Vrieze
Gentoo Developer
Mail: [EMAIL PROTECTED]
Homepage: http://www.devrieze.net
--
tvali
Measuring programming progress by lines of code is like measuring
aircraft building progress by weight. -Bill Gates
For every complex problem there is an answer that is clear, simple,
and wrong. -H L Mencken
Theory is when you know something, but it doesn't work. Practice is
when something works, but you don't know why. Programmers combine
theory and practice: Nothing works and they don't know why.
http://www.eskimo.com/~hottub/software/programming_quotes.html
http://www.softwarequotes.com/
--
gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] DB and binary dependency
On Fri, Mar 24, 2006 at 01:40:01PM +0200, tvali wrote: On 24/03/06, Paul de Vrieze [EMAIL PROTECTED] wrote: On Thursday 23 March 2006 21:38, Gustavo Sverzut Barbieri wrote: Cons: - it's not the final solution to the problem, as said, interfaces would be better... but interfaces would demand much more effort and not being automatically generated, would be async and probably incorrect at some point Interfaces, while nice, would overcomplicate things. They are also not needed as we have depend atoms. Paul Interface can be made somewhat automatically checkable. Checking the interfaces/symbols sucks however, because you can only do it _after_ you've built whatever you're building (packages do adjust the defines/typedefs/structs dependant on configure/build options). As I stated earlier, bincompat (not binslot paul :P) is the route to go- it gives you up front information so a resolver can plan out what has to be rebuilt automatically. ~harring pgpVQWlcq5D0D.pgp Description: PGP signature
Re: [gentoo-portage-dev] DB and binary dependency
On 24/03/06, Brian Harring [EMAIL PROTECTED] wrote: Checking the interfaces/symbols sucks however, because you can only do it _after_ you've built whatever you're building (packages do adjust the defines/typedefs/structs dependant on configure/build options). As I stated earlier, bincompat (not binslot paul :P) is the route to go- it gives you up front information so a resolver can plan out what has to be rebuilt automatically. no-no-no i'm not talking about binary and i will make another thread when i go further with my own topic :) to use those interfaces, u dont need to compile anything. ~harring -- tvali Measuring programming progress by lines of code is like measuring aircraft building progress by weight. -Bill Gates For every complex problem there is an answer that is clear, simple, and wrong. -H L Mencken Theory is when you know something, but it doesn't work. Practice is when something works, but you don't know why. Programmers combine theory and practice: Nothing works and they don't know why. http://www.eskimo.com/~hottub/software/programming_quotes.html http://www.softwarequotes.com/ -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] DB and binary dependency
On Friday 24 March 2006 12:40, tvali wrote:
Interface can be made somewhat automatically checkable.
For example:
void a(int);
void b(int, int);
void b(int, char);
Is compatible with:
void a(int);
void b(int, int);
Unfortunately, your wrong. This only makes sure that you have the right
slots to put your squares, triangles and circles in. It does not say that
b(int,int) from the first lib actually does the same thing as b(int,int)
from the second library.
To make it worse, suppose that header file 1 defines the following macro:
#define c(x,y) { int A = x; int B = y; if (AB) { b(A,B); } else
{ b(B,A); }
And header file 2 defines the following macro
#define c(x,y) { int A = x; int B = y; if (AB) { b(A,B); } else
{ b(B,A); }
And the semantics of B are such that one is supposed to use c(x,y) to call
b, and b1 accordingly expects the first argument to be smaller, and b2
expects the second one to be smaller. On the source level, this macro
makes it transparent. The libraries, although they may seem different,
are incompatible. While they implement the same source interface, they do
NOT implement the same binary interface. On a binary (read compiled)
level, they can not be exchanged for eachother.
While the above example is clearly broken design, this does happen enough
in actual libraries in way more subtle ways. And that is disregarding the
fact that the linux/elf ABI does not include argument lists in symbol
linking. As such b(int,char) is indistinguishable from b(int,int). To
overcome this C++ uses name mangling which creates names based on the
signature.
And this can be checked automatically - real header must have all
headings that interface describes. Microsoft has actually done good
work in C# on that topic -- first place where i saw something like
that after thinking about it myself :) This MS version is feature of a
language, of course, not a feature of compiler. Anyway, that might
give an idea.
There is a reason that MS implements it on the language level, in C#. It
is impossible to do while keeping the (admittedly extremely poor)
semantics of C and C++. Unfortunately the linux ABI (Application Binary
Interface) is based on C. As such one has to take C into account.
I'm actually sure that this all can be calculated up from sourcecode
and bindep would be after that a check if cpu didnt calculate
something wrong :P Another question, how difficult it is and is it
worth the time.
Perhaps you should read up your knowledge of the C language. After you
found that the C language is a mess, try C++, it makes things worse.
After that's finished take a look at solving this problem for ALL
languages.
Automatic source analysis for dependency calculation is a dead end. Even
if you manage to find the proper interfaces (oops, the package had it's
own gl.h instead of using the system one), you don't know anything about
the semantics of those interfaces. Two things with the same name may very
well have very different behaviours.
Paul
--
Paul de Vrieze
Gentoo Developer
Mail: [EMAIL PROTECTED]
Homepage: http://www.devrieze.net
pgp9iE5mTKGuU.pgp
Description: PGP signature
Re: [gentoo-portage-dev] DB and binary dependency
On Friday 24 March 2006 13:10, Brian Harring wrote: As I stated earlier, bincompat (not binslot paul :P) is the route to If you want to call it bincompat, I'd have to insist to make it BINCOMPAT ;-). go- it gives you up front information so a resolver can plan out what has to be rebuilt automatically. An argument I forgot to mention indeed. Paul -- Paul de Vrieze Gentoo Developer Mail: [EMAIL PROTECTED] Homepage: http://www.devrieze.net pgpeje4vSUL4a.pgp Description: PGP signature
Re: [gentoo-portage-dev] DB and binary dependency
On 24/03/06, Paul de Vrieze [EMAIL PROTECTED] wrote:
On Friday 24 March 2006 12:40, tvali wrote:
Interface can be made somewhat automatically checkable.
For example:
void a(int);
void b(int, int);
void b(int, char);
Is compatible with:
void a(int);
void b(int, int);
Unfortunately, your wrong. This only makes sure that you have the right
slots to put your squares, triangles and circles in. It does not say that
b(int,int) from the first lib actually does the same thing as b(int,int)
from the second library.
Sorry i thought about it for a moment, then left out later as broken
design cases didnt seems so important. Versioning of distinct
functions would be good, anyway not automatic.
To make it worse, suppose that header file 1 defines the following macro:
#define c(x,y) { int A = x; int B = y; if (AB) { b(A,B); } else
{ b(B,A); }
And header file 2 defines the following macro
#define c(x,y) { int A = x; int B = y; if (AB) { b(A,B); } else
{ b(B,A); }
And the semantics of B are such that one is supposed to use c(x,y) to call
b, and b1 accordingly expects the first argument to be smaller, and b2
expects the second one to be smaller. On the source level, this macro
makes it transparent. The libraries, although they may seem different,
are incompatible. While they implement the same source interface, they do
NOT implement the same binary interface. On a binary (read compiled)
level, they can not be exchanged for eachother.
While the above example is clearly broken design, this does happen enough
in actual libraries in way more subtle ways. And that is disregarding the
fact that the linux/elf ABI does not include argument lists in symbol
linking. As such b(int,char) is indistinguishable from b(int,int). To
overcome this C++ uses name mangling which creates names based on the
signature.
Ok, assuming broken design it's harder. Anyway, how would binary deps
solve this? This seems to be humans work in all cases.
And this can be checked automatically - real header must have all
headings that interface describes. Microsoft has actually done good
work in C# on that topic -- first place where i saw something like
that after thinking about it myself :) This MS version is feature of a
language, of course, not a feature of compiler. Anyway, that might
give an idea.
There is a reason that MS implements it on the language level, in C#. It
is impossible to do while keeping the (admittedly extremely poor)
semantics of C and C++. Unfortunately the linux ABI (Application Binary
Interface) is based on C. As such one has to take C into account.
Thats true that C# has much more clearness in it's language about what is what.
Anyway, as .h files have become as well as standard, you may show in
code as well as in bin, which functions are needed.
Is there any advantage of bin dep over code dep is a question for me
..some here suggest that there is, i think that everything can be done
as well with code deps.
I'm actually sure that this all can be calculated up from sourcecode
and bindep would be after that a check if cpu didnt calculate
something wrong :P Another question, how difficult it is and is it
worth the time.
Perhaps you should read up your knowledge of the C language. After you
found that the C language is a mess, try C++, it makes things worse.
After that's finished take a look at solving this problem for ALL
languages.
I actually know c++, but dont know c. Anyway, what i mean is that if
you have .h files, you may make up automatic check if interfaces fit
each other. I have to think about if c++ is a mess :)
Automatic source analysis for dependency calculation is a dead end. Even
if you manage to find the proper interfaces (oops, the package had it's
own gl.h instead of using the system one), you don't know anything about
the semantics of those interfaces. Two things with the same name may very
well have very different behaviours.
I think it should be solved with:
// system's header
#include ...
// own header
#include ...
But, yes, not in all cases.
Ok -- i agree that there are several problems, which have to be solved
before taking this topic up again and that they are possibly
unsolvable in many cases and possibly too much work in many others.
Paul
--
Paul de Vrieze
Gentoo Developer
Mail: [EMAIL PROTECTED]
Homepage: http://www.devrieze.net
--
tvali
If I had eight hours to chop down a tree, I'd spend six sharpening my
axe. -Abraham Lincoln
http://www.kuro5hin.org/story/2005/1/28/32622/4244
--
gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] DB and binary dependency
Can someone tell me, which portage python commands should be used or which kind of file created, if i'm going to test this idea? -- in beginning, i would like to just add simple deps - are ebuilds the only place to change and is there any clear doc of them [as i wouldnt like to go through them all to be sure that there is no rarely used things i dont know]? Also, where i can find relations between useflags and c/c++ #defines?-- tvaliMeasuring programming progress by lines of code is like measuring aircraft building progress by weight. -Bill Gates For every complex problem there is an answer that is clear, simple, and wrong. -H L MenckenTheory is when you know something, but it doesn't work. Practice is when something works, but you don't know why. Programmers combine theory and practice: Nothing works and they don't know why.http://www.eskimo.com/~hottub/software/programming_quotes.html http://www.softwarequotes.com/
Re: [gentoo-portage-dev] DB and binary dependency
tvali wrote: Can someone tell me, which portage python commands should be used or which kind of file created, if i'm going to test this idea? -- in beginning, i would like to just add simple deps - are ebuilds the only place to change and is there any clear doc of them [as i wouldnt like to go through them all to be sure that there is no rarely used things i dont know]? Also, where i can find relations between useflags and c/c++ #defines? USE flags technically have anything to do with C++ defines; they can technically enable/disable any optional support ( docs, plugins, etc.. ) Usually USE flags interact with a packages build system (configure or whatnot) Dependencies are only in DEPEND, RDEPEND,PDEPEND. -- tvali Measuring programming progress by lines of code is like measuring aircraft building progress by weight. -Bill Gates For every complex problem there is an answer that is clear, simple, and wrong. -H L Mencken Theory is when you know something, but it doesn't work. Practice is when something works, but you don't know why. Programmers combine theory and practice: Nothing works and they don't know why. http://www.eskimo.com/~hottub/software/programming_quotes.html http://www.softwarequotes.com/ -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] DB and binary dependency
On 23/03/06, Alec Warner [EMAIL PROTECTED] wrote: tvali wrote: Can someone tell me, which portage python commands should be used or which kind of file created, if i'm going to test this idea? -- in beginning, i would like to just add simple deps - are ebuilds the only place to change and is there any clear doc of them [as i wouldnt like to go through them all to be sure that there is no rarely used things i dont know]? Also, where i can find relations between useflags and c/c++ #defines? USE flags technically have anything to do with C++ defines; they can technically enable/disable any optional support ( docs, plugins, etc.. ) Usually USE flags interact with a packages build system (configure or whatnot) In c++ code, use flags must definitely have something to do with #defines. Otherwise they couldnt do what they do -- so uflags=buildsys=defines. Configure, for example, will define set of #defines. Ok, your answer at least shows that this relation is not as simple and clear as i would like it :) Where i could see some useflag actually interacting with build? Where i see that line which makes it so that if i define gtk, gtk support will be added to makefile? -- tvali Measuring programming progress by lines of code is like measuring aircraft building progress by weight. -Bill Gates For every complex problem there is an answer that is clear, simple, and wrong. -H L Mencken Theory is when you know something, but it doesn't work. Practice is when something works, but you don't know why. Programmers combine theory and practice: Nothing works and they don't know why. http://www.eskimo.com/~hottub/software/programming_quotes.html http://www.softwarequotes.com/ -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] DB and binary dependency
>From Paul de Vrieze:The semantics that make up the relationships between useflags and the actual source as goes out of the preprocessor is very complicated. Probably theeasiest way to find it out is to try each permutation and somehow hook intogcc/g++ to get the result of that choice.And that's only for C and C++ sources. Ok, thanks :) So this interaction is one more thing to get simpler? I'm starting to think that i should seek for some very-very small part of portage to develop, because size of things [amount of work], which i already think i should improve in some way, is getting to somewhere near infinity :) ...ok, near what infinity is for me, not theoretical infinity ;) What is the simplest way to get one portage running in some virtual space where i could install and uninstall without actually changing my system? Using chroot in the same way as when installing gentoo is good? -- tvaliMeasuring programming progress by lines of code is like measuring aircraft building progress by weight. -Bill GatesFor every complex problem there is an answer that is clear, simple, and wrong. -H L Mencken Theory is when you know something, but it doesn't work. Practice is when something works, but you don't know why. Programmers combine theory and practice: Nothing works and they don't know why.http://www.eskimo.com/~hottub/software/programming_quotes.html http://www.softwarequotes.com/
Re: [gentoo-portage-dev] DB and binary dependency
On Thursday 23 March 2006 20:53, tvali wrote: So this interaction is one more thing to get simpler? I'm starting to think that i should seek for some very-very small part of portage to develop, because size of things [amount of work], which i already think i should improve in some way, is getting to somewhere near infinity :) ...ok, near what infinity is for me, not theoretical infinity ;) What is the simplest way to get one portage running in some virtual space where i could install and uninstall without actually changing my system? Using chroot in the same way as when installing gentoo is good? Certainly, chroot combined with lvm snapshots would be the easiest way. If you want to focus on binary packages, you might want to start with not doing it automatically, but using some crude heuristics. You can make it configurable for when the heuristics don't work. Basically what is the difference between a binary package and the source package is the dependencies. You can regard a dependency as a restriction on the configurations in which the package can be installed. For binary packages only runtime dependencies are relevant (well in gentoo anyway, we don't have initial configuration deps). Binary packages do however further restrict the configurations from the runtime dependencies. These restrictions originate from the configuration when compiling the package. Then I have some good and bad news. The good news: - Binary dependencies are as necessary for binary packages, as for validating a current configuration (set of installed packages). The bad: - For proper binary dependencies the depend expression possibilities must be extended. Useflag dependencies would be at least usefull. BINSLOT would be almost required. We already saw that SLOT can't always be used for BINSLOT. Paul -- Paul de Vrieze Gentoo Developer Mail: [EMAIL PROTECTED] Homepage: http://www.devrieze.net pgpC1Bf8otT4G.pgp Description: PGP signature
Re: [gentoo-portage-dev] DB and binary dependency
On 23/03/06, Paul de Vrieze [EMAIL PROTECTED] wrote: Certainly,chroot combined with lvm snapshots would be the easiest way.If you want to focus on binary packages, you might want to start with notdoing it automatically, but using some crude heuristics. You can make it configurable for when the heuristics don't work.Basically what is the difference between a binary package and the sourcepackage is the dependencies.You can regard a dependency as a restriction on the configurations in which the package can be installed. For binary packages only runtime dependenciesare relevant (well in gentoo anyway, we don't have initial configurationdeps). Binary packages do however further restrict the configurations from the runtime dependencies. These restrictions originate from the configurationwhen compiling the package.Then I have some good and bad news. The good news:- Binary dependencies are as necessary for binary packages, as for validating a current configuration (set of installed packages).The bad:- For proper binary dependencies the depend _expression_ possibilities must beextended. Useflag dependencies would be at least usefull. BINSLOT would be almost required. We already saw that SLOT can't always be used for BINSLOT. BINSLOT is a new word for me. I dont see any meaning in calculating dependencies after build. There would be too much problems: * If any of packages you depend on, are masked, you get an error after build, but should get it before * In many cases, not only libs, but also headers are included from another pack [kernel headers, for example, as i have seen from error messages] * You cant pre-calculate dependancies My interest, for sure, is code dependancies. At first i see a clear way to get it work for now, secondly i think it can be used for bin deps, also. In cases, where both headers and libs are included from other pack, which is same on both cases, this is totally irrelevant for deps if this other pack is statically or dynamically used. In cases, where headers are in pack to be compiled and .so is in another package, i havent yet checked, how to understand if dependancy is binary -- i cant say for sure that this does not include some work from user. Ok, i better write few lines of code now as i know where i should start (or at least i have one part of code, which could be discussed here, in front of my minds-eye). Thanks. You will understand me better when something works :) Paul--Paul de VriezeGentoo DeveloperMail: [EMAIL PROTECTED]Homepage: http://www.devrieze.net-- tvaliMeasuring programming progress by lines of code is like measuring aircraft building progress by weight. -Bill Gates For every complex problem there is an answer that is clear, simple, and wrong. -H L MenckenTheory is when you know something, but it doesn't work. Practice is when something works, but you don't know why. Programmers combine theory and practice: Nothing works and they don't know why.http://www.eskimo.com/~hottub/software/programming_quotes.html http://www.softwarequotes.com/
Re: [gentoo-portage-dev] DB and binary dependency
Ok... this discussion is missing my initial point that is how to provide binary dependency and avoid many crashes we have now with almost no effort. My initial proposal was to, after compile and before install is done, we should parse linker information and check for each library it depends, which package-version provides it and then mark the just compiled package runtime-dependent on them. If the dependency was not installed by portage, then issue a warning/error and take some action (ask user, abort or ignore) Pros: - it would avoid many crashes we have now because we remove packages that others depend on. - it would avoid many crashes we have now because we update packages that others depend on, but are not listed correctly in ebuilds. - it would avoid dependency errors caused by missing --disable-feature with autoconf-like packages Cons: - it's not the final solution to the problem, as said, interfaces would be better... but interfaces would demand much more effort and not being automatically generated, would be async and probably incorrect at some point -- Gustavo Sverzut Barbieri -- Jabber: [EMAIL PROTECTED] MSN: [EMAIL PROTECTED] ICQ#: 17249123 Skype: gsbarbieri Mobile: +55 (81) 9927 0010 Phone: +1 (347) 624 6296; [EMAIL PROTECTED] GPG: 0xB640E1A2 @ wwwkeys.pgp.net -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] DB and binary dependency
On 23/03/06, Gustavo Sverzut Barbieri [EMAIL PROTECTED] wrote: Ok... this discussion is missing my initial point that is how toprovide binary dependency and avoid many crashes we have now withalmost no effort. Paul was not missing it ;) Part of his message was for me, part was for you. I have got my job now so i leave for a while :) --Gustavo Sverzut Barbieri--Jabber: [EMAIL PROTECTED] MSN: [EMAIL PROTECTED]ICQ#: 17249123 Skype: gsbarbieriMobile: +55 (81) 9927 0010 Phone:+1 (347) 624 6296; [EMAIL PROTECTED] GPG: 0xB640E1A2 @ wwwkeys.pgp.net--gentoo-portage-dev@gentoo.org mailing list-- tvaliMeasuring programming progress by lines of code is like measuring aircraft building progress by weight. -Bill GatesFor every complex problem there is an answer that is clear, simple, and wrong. -H L Mencken Theory is when you know something, but it doesn't work. Practice is when something works, but you don't know why. Programmers combine theory and practice: Nothing works and they don't know why.http://www.eskimo.com/~hottub/software/programming_quotes.html http://www.softwarequotes.com/
Re: [gentoo-portage-dev] DB and binary dependency
This thread keeps going and going and it's a subject thats already been covered... So I'll just Re you here. Search the archives here for RRDEPEND, LDEPEND As soon as I can figure out a way in python to do fast lookups of libs it will be integrated. I can do it really really fast in c but I don't think everybody here is keen on installing the portage-utils. -peace On Wed, 2006-03-15 at 12:13 -0300, Gustavo Sverzut Barbieri wrote: Hello, There is any provision for binary dependency on Gentoo/Portage? The way it works now is quite messy with things like revdep-rebuild. I have an idea to solve this problem: after software is build, you check which files it links (ldd binaries libraries) and check the used against installed packages. If a library is not provided by an installed package we could have a policy to inform user or just abort installation. Also, if we implement these dependencies in rpm-generator, we could just generate RPM packages and install it in the RPM-DB and let it handle these kind of things. With these in place emerge would handle the build stage (where it excels), but rpm would handle the binary installation and dependencies (where it excels). -- Gustavo Sverzut Barbieri -- Mobile: +55 (81) 9927 0010 Phone: +1 (347) 624 6296; [EMAIL PROTECTED] Jabber: [EMAIL PROTECTED] ICQ#: 17249123 MSN: [EMAIL PROTECTED] GPG: 0xB640E1A2 @ wwwkeys.pgp.net -- solar [EMAIL PROTECTED] Gentoo Linux -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] DB and binary dependency
On 3/20/06, tvali [EMAIL PROTECTED] wrote: 2006/3/20, Gustavo Sverzut Barbieri [EMAIL PROTECTED]: I do think you're overcomplicating things where you shouldn't. Declaring stuff manually will always break, and to ensure a safe system, it's better to use compiler information. In all cases, dependancy should be based on interfaces, not code. All packages may: * Provide an interface * Use an interface Depending on useflags, OS and other compile options, it differs, which interfaces are provided and used. This is, abstractly, what portage does with interfaces. If portage uses some interface, it may need it's header files when building. It may also need another lib for static build. This means that binary check is not possible in all cases. Now, the problem is: * How to get an information about a package, which specifies exactly, which interface is needed. How to get it before building in case when this interface is needed to be emerged before compilation [before linking everything together, at least]. Which is a form of this information and what could be read out from that? * How to get information about which interfaces are provided by which packages *not yet emerged* -- by their current use flags(?). This means that it must be possible to know, which interfaces are provided by packages, without first building it -- and the form given by binary check must be the same as the form of descriptor used by this package check. So, how to get correct provider together with correct client? Ok, I agree with you that this would be the perfect solution, but it would demand too much effort to have this right. I'm not proposing the final-perfect solution, just something better than we have now. It would not cover every case, but would cover most cases in a satisfactory way. -- Gustavo Sverzut Barbieri -- Jabber: [EMAIL PROTECTED] MSN: [EMAIL PROTECTED] ICQ#: 17249123 Skype: gsbarbieri Mobile: +55 (81) 9927 0010 Phone: +1 (347) 624 6296; [EMAIL PROTECTED] GPG: 0xB640E1A2 @ wwwkeys.pgp.net -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] DB and binary dependency
I was thinking about it, too, and found something i do like maybe more. It would be not binary, but code dependancy. This is limited to specific languages, then, but after all, there may also be different binary dependencies, too [for example, you may depend on fonts images from another package]. Ok, my thought is like that. Lets imagine a simple c file: #ifdef usepackagex #include packagex.h #endif As it is simple to see, this file could be used to calculate non-binary dependency, inluding useflags. And there is a plus -- tool, which just reads those #ifdef's and #includes, could easily understand, which flags make up which dependancy, without building package again and again with different useflags. PS. i dont mean X by packagex, but rather a random pack. I write here as if #define always includes useflag name - i may be wrong, but it doesnt make a huge difference. Sure, there are #ifdef's in code, which should not read, so we must have an array for legal useflags - but this is also not important in my explanation. Lets add some custom comment syntax (line numbers are given for explanation): 1. #ifdef usepackagex 2. /* Portage: packagex.h = abc-def/packagex = 1.0.2 */ 3. /* Portage: use fonts/customfontpackage */ 4. #include packagex.h 5. #endif Now, what a parser does: Start parse: setup empty array UseFlags = [] Line 1: #ifdef usepackagex Add useflag name into useflags array: UseFlags = ['usepackagex'] Line 2: /* Portage: packagex.h = abc-def/packagex = 1.0.2 */ This line maybe outside of any #ifdef's as well, but in our small application there is no external conf.c file ;) Dictionary 'Links' will contain a notice that whenever packagex is included, it must be at least version 1.0.2 of package abc-def/packagex. Line 3: /* Portage: use fonts/customfontpackage */ Dependancy on fonts/customfontpackage, therefore add to dependancies: [['usepackagex'], 'fonts/customfontpackage'] Line 4: #include packagex.h Dependancy on fonts/customfontpackage, therefore add to dependancies: [['usepackagex'], 'abc-def/packagex = 1.0.2'] Line 5: #endif Remove 'usepackagex' from useflags: UseFlags = [] 2006/3/22, Gustavo Sverzut Barbieri [EMAIL PROTECTED]: On 3/20/06, tvali [EMAIL PROTECTED] wrote: 2006/3/20, Gustavo Sverzut Barbieri [EMAIL PROTECTED]: I do think you're overcomplicating things where you shouldn't. Declaring stuff manually will always break, and to ensure a safe system, it's better to use compiler information. In all cases, dependancy should be based on interfaces, not code. All packages may: * Provide an interface * Use an interface Depending on useflags, OS and other compile options, it differs, which interfaces are provided and used. This is, abstractly, what portage does with interfaces. If portage uses some interface, it may need it's header files when building. It may also need another lib for static build. This means that binary check is not possible in all cases. Now, the problem is: * How to get an information about a package, which specifies exactly, which interface is needed. How to get it before building in case when this interface is needed to be emerged before compilation [before linking everything together, at least]. Which is a form of this information and what could be read out from that? * How to get information about which interfaces are provided by which packages *not yet emerged* -- by their current use flags(?). This means that it must be possible to know, which interfaces are provided by packages, without first building it -- and the form given by binary check must be the same as the form of descriptor used by this package check. So, how to get correct provider together with correct client?Ok, I agree with you that this would be the perfect solution, but it would demand too much effort to have this right.I'm not proposing the final-perfect solution, just something betterthan we have now. It would not cover every case, but would cover mostcases in a satisfactory way. --Gustavo Sverzut Barbieri--Jabber: [EMAIL PROTECTED] MSN: [EMAIL PROTECTED] ICQ#: 17249123 Skype: gsbarbieriMobile: +55 (81) 9927 0010 Phone:+1 (347) 624 6296; [EMAIL PROTECTED] GPG: 0xB640E1A2 @ wwwkeys.pgp.net--gentoo-portage-dev@gentoo.org mailing list-- tvaliFrom a programmer's point of view, the user is a peripheral that types when you issue a read request.-P. Williams If you think your management doesn't know what it's doing or that your organisation turns out low-quality software crap that embarrasses you, then leave.-Ed YourdonWe all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. -Larry Wall[ http://www.softwarequotes.com/ ] - http://www.softwarequotes.com/ShowQuotes.asp?ID=544Name=Borenstein,_Nathaniel_S. - http://www.softwarequotes.com/ShowQuotes.asp?ID=571Name=Boehm,_Barry
Re: [gentoo-portage-dev] DB and binary dependency
I hope you did read my previous mail of code dependancy -- if not, it's at the end of message. This here is not so much overthought thing, but a good starting point, maybe, if code deps are used. Anyway, i give here the same idea in more complete form and good syntax (good, because it could be added to compilers, for example): /* As compiler may give warnings or errors otherwise, portage code parses defines portage-providers. */ #ifdef runasdaemon #ifdef portage-providers #provide interface customdaemon /* if provided *as*, then does not actually provide anything before include clause */ #use interface daemonheaders 1.0.2 as dhead.h /* when test.h is included, depends on dev-apps/test at least version 1.0.2 */ #provider-atom test set package=dev-apps/test #provider-atom test set version=1.0.2 /* require testflag to be set */ #provider-atom test need useflag testflag Hello world! /* require testflag2 to be unset */ #provider-atom test hate useflag testflag2 Testflag2 bad :( /* have some good taste in warnings */ #provider-atom test warn useflag testflag3 Test warning! #provider-atom test ask useflag testflag4 Testflag4 good :) #use atom test as test.h #provider-atom otheratom set package=dev-apps/otherpackage /* if runasdaemon, otheratom will be required */ #use atom otheratom #endif #ifdef something /* if runasdaemon and something, daemonheaders 1.0.2 is needed */ #include dhead.h /* if runasdaemon and something, atom test is needed */ #include test.h /* atoms could be provided as well; interface is a simple atom with few params; just test.h could be provided, also: */ #ifdef portage-providers #provide header welcome.h #use header someheader.h #endif #endif #endif2006/3/22, tvali [EMAIL PROTECTED]: I was thinking about it, too, and found something i do like maybe more. It would be not binary, but code dependancy. This is limited to specific languages, then, but after all, there may also be different binary dependencies, too [for example, you may depend on fonts images from another package]. Ok, my thought is like that. Lets imagine a simple c file: #ifdef usepackagex #include packagex.h #endif As it is simple to see, this file could be used to calculate non-binary dependency, inluding useflags. And there is a plus -- tool, which just reads those #ifdef's and #includes, could easily understand, which flags make up which dependancy, without building package again and again with different useflags. PS. i dont mean X by packagex, but rather a random pack. I write here as if #define always includes useflag name - i may be wrong, but it doesnt make a huge difference. Sure, there are #ifdef's in code, which should not read, so we must have an array for legal useflags - but this is also not important in my explanation. Lets add some custom comment syntax (line numbers are given for explanation): 1. #ifdef usepackagex 2. /* Portage: packagex.h = abc-def/packagex = 1.0.2 */ 3. /* Portage: use fonts/customfontpackage */ 4. #include packagex.h 5. #endif Now, what a parser does: Start parse: setup empty array UseFlags = [] Line 1: #ifdef usepackagex Add useflag name into useflags array: UseFlags = ['usepackagex'] Line 2: /* Portage: packagex.h = abc-def/packagex = 1.0.2 */ This line maybe outside of any #ifdef's as well, but in our small application there is no external conf.c file ;) Dictionary 'Links' will contain a notice that whenever packagex is included, it must be at least version 1.0.2 of package abc-def/packagex. Line 3: /* Portage: use fonts/customfontpackage */ Dependancy on fonts/customfontpackage, therefore add to dependancies: [['usepackagex'], 'fonts/customfontpackage'] Line 4: #include packagex.h Dependancy on fonts/customfontpackage, therefore add to dependancies: [['usepackagex'], 'abc-def/packagex = 1.0.2'] Line 5: #endif Remove 'usepackagex' from useflags: UseFlags = [] -- tvaliFrom a programmer's point of view, the user is a peripheral that types when you issue a read request.-P. WilliamsIf you think your management doesn't know what it's doing or that your organisation turns out low-quality software crap that embarrasses you, then leave.-Ed YourdonWe all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. -Larry Wall[ http://www.softwarequotes.com/ ] - http://www.softwarequotes.com/ShowQuotes.asp?ID=544Name=Borenstein,_Nathaniel_S. - http://www.softwarequotes.com/ShowQuotes.asp?ID=571Name=Boehm,_Barry
Re: [gentoo-portage-dev] DB and binary dependency
On Wed, 2006-03-22 at 19:40 +0200, tvali wrote: I was thinking about it, too, and found something i do like maybe more. It would be not binary, but code dependancy. This is limited to specific languages, then, but after all, there may also be different binary dependencies, too [for example, you may depend on fonts images from another package]. Ok, my thought is like that. Lets imagine a simple c file: #ifdef usepackagex #include packagex.h #endif As it is simple to see, this file could be used to calculate non-binary dependency, inluding useflags. And there is a plus -- tool, which just reads those #ifdef's and #includes, could easily understand, which flags make up which dependancy, without building package again and again with different useflags. There are a few reasons why this won't work :-) First: What if I have assembler? python? perl? Your example is limited to the C preprocessor. Second: You'll have to get this depend information applied by upstream unless you want to patch every file ... and as upstream I'd laugh at anyone proposing that Third: Since there is no easy way of generating this automatically you'll have to replicate every bit of dependency information that is in the ebuilds. That will be much work, you won't keep ebuilds synchronized to the included dep info ... So in short, interesting concept, but I don't see how it can work and how it is an improvement over what we have now. Please don't reinvent the wheel ;-) Patrick -- Stand still, and let the rest of the universe move signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] DB and binary dependency
2006/3/22, Patrick Lauer [EMAIL PROTECTED]: There are a few reasons why this won't work :-)First: What if I have assembler? python? perl?Your example is limited to the C preprocessor. Yes that i did tell there that it's limited to c already :) but bin dep, after all, is limited to lib dependencies and probably just .so files -- which means probably not python (as it's too dynamic to programmatically understand, where and which libs are used), not perl (as it doesnt have binaries). So i dont see my version being much worse than bin. If you have an assembler, then everything depends on which one you have. If that's internal assembler of c, everything works fine ;) If that is some macro-assembler, then there is a bit different syntax from c, but other code is basically the same (parsing needed information out from assembler code is not too hard). If that's python, then you cant anymore make it so simple -- there is nothing like #define in python, as it's dynamic. In python, you have several ways -- adding dependencies in classical way; adding a *.c file, which talks about dependancies or making some python script, which is able to tell on what it depends -- the last way has the same weakness with bin dep that it should be downloaded before. About perl, i dont know enough to tell, how it could be used there, but anyway, as it is scripting language, i think that smth similar to py applies. I think that there could be simple dependency builder for packages, which will be compiled, but not for dynamic languages. There is no simple way in dynamic languages for knowing, which libs will be used, even if you know the useflags or have them built into pyc files. As code deps are only my thought about repairing the biggest bottleneck in bin-dep - that you cant say anything about deps of a random combination of flags without rebuilding with excactly that combination -, i am not trying to make it usable in dynamic/scripting languages. Second: You'll have to get this depend information applied by upstreamunless you want to patch every file ... and as upstream I'd laugh at anyone proposing that This is actually simple build. I just told about method, which would allow you to include dependancy metadata directly in c code. It could be compiled into current portage dependancy data as well. Third: Since there is no easy way of generating this automaticallyyou'll have to replicate every bit of dependency information that is in the ebuilds. That will be much work, you won't keep ebuilds synchronizedto the included dep info ... Data in ebuilds should be generated from that data. So in short, interesting concept, but I don't see how it can work andhow it is an improvement over what we have now. Please don't reinvent the wheel ;-) Yes, if it comes to wheels, i better try to invent a better wheel, not reinvent an old wheel ;) PS. in my last example, i dont show how useflags are related to #defines. That's because i guess that it must be already described somewhere and have not to be in c code :) I hope it is so ;) Patrick--Stand still, and let the rest of the universe move-BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2-ecc0.1.6 (GNU/Linux)iD8DBQBEIZTiqER3hOUoZM4RAkEtAJ913WnthMPjRZrUGQCgYnOw9Xcu7ACeO1DhVKWKYb6XcDuugBJSSBFMgc0==xD2e-END PGP SIGNATURE- -- tvaliFrom a programmer's point of view, the user is a peripheral that types when you issue a read request.-P. WilliamsIf you think your management doesn't know what it's doing or that your organisation turns out low-quality software crap that embarrasses you, then leave.-Ed YourdonWe all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. -Larry Wall[ http://www.softwarequotes.com/ ] - http://www.softwarequotes.com/ShowQuotes.asp?ID=544Name=Borenstein,_Nathaniel_S. - http://www.softwarequotes.com/ShowQuotes.asp?ID=571Name=Boehm,_Barry
Re: [gentoo-portage-dev] DB and binary dependency
As an addition to code deps discussion. I didnt understand exactly, why bin deps were supposed to be better than what we have now, as i am not yet exactly sure what we have :) Anyway, i see one basic plus of code deps. It's that you may have huge number of codelines, all containing #defines and #ifdefs. You may know that whenever it uses ?.h header, it needs library ? -- but when you need ?.h may be somewhat complex case. There may be, for example, 40 different .h files of your own, which include ?.h file -- and each of them may be included only if corresponding useflag is set. In such case, it's easier to describe, which package you need when ?.h is used than to write all those if's twise in code and some other place, which make sure if you need that ? pack or not. I havent done such thing in reality, so i dont know, how big problem it is for a programmer (how much you have that situation i described in real life), but i guess that this is the problem what binary deps were supposed to solve. -- tvaliFrom a programmer's point of view, the user is a peripheral that types when you issue a read request.-P. WilliamsIf you think your management doesn't know what it's doing or that your organisation turns out low-quality software crap that embarrasses you, then leave.-Ed YourdonWe all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. -Larry Wall[ http://www.softwarequotes.com/ ] - http://www.softwarequotes.com/ShowQuotes.asp?ID=544Name=Borenstein,_Nathaniel_S. - http://www.softwarequotes.com/ShowQuotes.asp?ID=571Name=Boehm,_Barry
Re: [gentoo-portage-dev] DB and binary dependency
I'm replying to this one, but I've read the whole thread... On 3/16/06, Paul de Vrieze [EMAIL PROTECTED] wrote: On Wednesday 15 March 2006 16:13, Gustavo Sverzut Barbieri wrote: Hello, There is any provision for binary dependency on Gentoo/Portage? The way it works now is quite messy with things like revdep-rebuild. I have an idea to solve this problem: after software is build, you check which files it links (ldd binaries libraries) and check the used against installed packages. If a library is not provided by an installed package we could have a policy to inform user or just abort installation. Also, if we implement these dependencies in rpm-generator, we could just generate RPM packages and install it in the RPM-DB and let it handle these kind of things. With these in place emerge would handle the build stage (where it excels), but rpm would handle the binary installation and dependencies (where it excels). Solving this is not trivial. Basically suppose application X depends on sys-libs/db-4.* This can be resolved by differently slotted libraries. We need to record which one was actually used (not easy by itself, but more an issue of the ebuild itself, if more than 1 candidate is available). But suppose that we know that the application was compiled to use db-4.3.29. We must then know that it is ok to replace the db-4.3.29 package with 4.3.30, but that it isn't ok to replace it by 4.3.28 or 4.4.20. To make this things worse, the above example assumes that within a slot, the libraries are binary compatible. There are examples of libraries that are not. And what about a library whose interface is dependent on a third library: B uses A, C uses B, but B exports A. So B is dependent on A, and the binary package of C must record that B was compiled with A. In short, welcome to binary package hell. This is the reason that binary distributions must use versions. Even debian. It is just very very hard to fix these kinds of indirect dependencies. I do think you're overcomplicating things where you shouldn't. Declaring stuff manually will always break, and to ensure a safe system, it's better to use compiler information. So I wouldn't mind fixing it to one package instead of a slot. I mean, if user compiled software X-1.0 and it depends on library Y-2.0, provided at the moment of X-1.0 compilation by package Z-3.0, then make X-1.0 depend on Z-3.0. If you were to remove Z-3.0 or upgrade it or even add other option by means of USE, have X-1.0 to be recompiled too... you could be even more correct to make it check CFLAGS too. Of course this correctness could piss users, then you could have a --nodeps or something like that to avoid this and use the old behaviour. -- Gustavo Sverzut Barbieri -- Jabber: [EMAIL PROTECTED] MSN: [EMAIL PROTECTED] ICQ#: 17249123 Skype: gsbarbieri Mobile: +55 (81) 9927 0010 Phone: +1 (347) 624 6296; [EMAIL PROTECTED] GPG: 0xB640E1A2 @ wwwkeys.pgp.net -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] DB and binary dependency
On Wednesday 15 March 2006 16:13, Gustavo Sverzut Barbieri wrote: Hello, There is any provision for binary dependency on Gentoo/Portage? The way it works now is quite messy with things like revdep-rebuild. I have an idea to solve this problem: after software is build, you check which files it links (ldd binaries libraries) and check the used against installed packages. If a library is not provided by an installed package we could have a policy to inform user or just abort installation. Also, if we implement these dependencies in rpm-generator, we could just generate RPM packages and install it in the RPM-DB and let it handle these kind of things. With these in place emerge would handle the build stage (where it excels), but rpm would handle the binary installation and dependencies (where it excels). Solving this is not trivial. Basically suppose application X depends on sys-libs/db-4.* This can be resolved by differently slotted libraries. We need to record which one was actually used (not easy by itself, but more an issue of the ebuild itself, if more than 1 candidate is available). But suppose that we know that the application was compiled to use db-4.3.29. We must then know that it is ok to replace the db-4.3.29 package with 4.3.30, but that it isn't ok to replace it by 4.3.28 or 4.4.20. To make this things worse, the above example assumes that within a slot, the libraries are binary compatible. There are examples of libraries that are not. And what about a library whose interface is dependent on a third library: B uses A, C uses B, but B exports A. So B is dependent on A, and the binary package of C must record that B was compiled with A. In short, welcome to binary package hell. This is the reason that binary distributions must use versions. Even debian. It is just very very hard to fix these kinds of indirect dependencies. Paul -- Paul de Vrieze Gentoo Developer Mail: [EMAIL PROTECTED] Homepage: http://www.devrieze.net pgpqRcIBYHhbT.pgp Description: PGP signature
Re: [gentoo-portage-dev] DB and binary dependency
To make this things worse, the above example assumes that within a slot, the libraries are binary compatible. There are examples of libraries that are not. And what about a library whose interface is dependent on a third library: B uses A, C uses B, but B exports A. So B is dependent on A, and the binary package of C must record that B was compiled with A. In short, welcome to binary package hell. This is the reason that binary distributions must use versions. Even debian. It is just very very hard to fix these kinds of indirect dependencies. I think that there are many reasons, why such kind of important thing should not be automatically generated. Starting from the fact that one thing is dependency in binary, another thing is knowledge if this combination actually works, has no bugs, should not be masked. Anyway, if such binary dependancyes should ever be used, then it would be best to: 1. register version number of interfaces 2. register interface groups I mean: If some lib supports commands turnon() and turnoff(), but one version supports additionally command int checkifturnedon() and another supports char checkifturnedon(), then there should be: * Interface group, which states only that there are turnon and turnoff functions -- as many apps use only most standard and minimalistic set of commands from one lib, there is no need to know if some advanced features are 0.9.80 or 0.9.64 compatible. * Two interface versions, 0.9.80a and 0.9.80b, for example All those interfaces and groups could be file links -- lib_interfacegroup_interface1.a would link to libinterface1.a. If there is 1 library with 2 sets of minimal functionality, then it's questionable, how to implement both (there would be linker error in some cases) -- which makes this conception weaker. Anyway, when standard interfaces are used and linked, that would help a bit. Anyway -- why should i check dependencies *after* building of a big pack, maybe hours of building, when i can check it without even downloading it, when there is nice portage tree? And, i think that making it binary would allow too much bad style -- there are imho things, which should not be automated without very-very careful thinking even if one can only win with automating them in theory and portage tree is most definitely one of them. Paul -- Paul de Vrieze Gentoo Developer Mail: [EMAIL PROTECTED] Homepage: http://www.devrieze.net -- tvali (e-mail: [EMAIL PROTECTED]; msn: [EMAIL PROTECTED]; icq: 317-492-912) I like net more than life, cause if i do something wrong, then people in net will tell me that i do, so that i can fix it -- people in life will tell others that i do. -- gentoo-portage-dev@gentoo.org mailing list
Re: [gentoo-portage-dev] DB and binary dependency
On Thu, Mar 16, 2006 at 02:58:00PM +0100, Paul de Vrieze wrote: On Wednesday 15 March 2006 16:13, Gustavo Sverzut Barbieri wrote: Hello, There is any provision for binary dependency on Gentoo/Portage? The way it works now is quite messy with things like revdep-rebuild. Solving this is not trivial. Basically suppose application X depends on sys-libs/db-4.* This can be resolved by differently slotted libraries. We need to record which one was actually used (not easy by itself, but more an issue of the ebuild itself, if more than 1 candidate is available). But suppose that we know that the application was compiled to use db-4.3.29. We must then know that it is ok to replace the db-4.3.29 package with 4.3.30, but that it isn't ok to replace it by 4.3.28 or 4.4.20. To make this things worse, the above example assumes that within a slot, the libraries are binary compatible. There are examples of libraries that are not. And what about a library whose interface is dependent on a third library: B uses A, C uses B, but B exports A. So B is dependent on A, and the binary package of C must record that B was compiled with A. In short, welcome to binary package hell. This is the reason that binary distributions must use versions. Even debian. It is just very very hard to fix these kinds of indirect dependencies. You're forgetting we're equally screwed when upstream (*cough* openssl) goes and changes the soname while preserving abi. Old solution to it was to add another metadata key to ebuilds, bincompat that is compared within slotting to see if a rebuild of rdepend consumers needs to occur. fex... alsa-lib-0.90 slot=0 bincompat=0 alsa-lib-1.0 slot=0 bincompat=1 The pkg manager does it's little dance to figure out what should be installed where (and what versions can coexist), while doing it if it's going to replace a pkg and bincompat differs, pull in the subset of the state graph at that point (including vdb) that rdeps on that package/slot and force a rebuild of them. As per the norm, requires a smart resolver; for c++ would expect cycles to occur where the only solution is to pull in libstdc++ (fex) to sidestep horkage while doing the rebuilds... Also note that (like all general solutions) it's not going to be able to cover all insane stuff; if the api breaks in transitioning bincompat is going to cause a minor explossion that is only resolvable by packages specifying exact dependencies (read: version ranges). ~harring pgpFDZuONXCnW.pgp Description: PGP signature
Re: [gentoo-portage-dev] DB and binary dependency
On Thursday 16 March 2006 15:18, tvali wrote: To make this things worse, the above example assumes that within a slot, the libraries are binary compatible. There are examples of libraries that are not. And what about a library whose interface is dependent on a third library: B uses A, C uses B, but B exports A. So B is dependent on A, and the binary package of C must record that B was compiled with A. In short, welcome to binary package hell. This is the reason that binary distributions must use versions. Even debian. It is just very very hard to fix these kinds of indirect dependencies. I think that there are many reasons, why such kind of important thing should not be automatically generated. Starting from the fact that one thing is dependency in binary, another thing is knowledge if this combination actually works, has no bugs, should not be masked. Anyway, if such binary dependancyes should ever be used, then it would be best to: 1. register version number of interfaces 2. register interface groups I mean: If some lib supports commands turnon() and turnoff(), but one version supports additionally command int checkifturnedon() and another supports char checkifturnedon(), then there should be: * Interface group, which states only that there are turnon and turnoff functions -- as many apps use only most standard and minimalistic set of commands from one lib, there is no need to know if some advanced features are 0.9.80 or 0.9.64 compatible. * Two interface versions, 0.9.80a and 0.9.80b, for example While theoretically one can't even know that the semantics are correct, we know from experience that hell will not break through. Downgrading libraries is always problematic though. It's not desirable enough to try to support it. All those interfaces and groups could be file links -- lib_interfacegroup_interface1.a would link to libinterface1.a. Looking at a sub-package level is going to make things way too complex. I don't think that doing so is going to help gentoo/portage in any way. If there is 1 library with 2 sets of minimal functionality, then it's questionable, how to implement both (there would be linker error in some cases) -- which makes this conception weaker. Anyway, when standard interfaces are used and linked, that would help a bit. IMHO we need to know from every binary or installed package which package version and slot was used to satisfy it. We also need to know which of those dependencies must be exported to packages which use this one. All this must also take useflags into account ;-) of course. Anyway -- why should i check dependencies *after* building of a big pack, maybe hours of building, when i can check it without even downloading it, when there is nice portage tree? And, i think that making it binary would allow too much bad style -- there are imho things, which should not be automated without very-very careful thinking even if one can only win with automating them in theory and portage tree is most definitely one of them. The difference between binary packages and tracking what is installed with which relations are very small. Basically once compiled, the candidates that may resolve a dependency are fewer. While a BINSLOT variable could be an option (Although I'm not sure whether this would be needed), we still need to know the minimal version of the package within the slot. It's not about checking dependencies after building. It's about seeing whether one should rebuild package B because it depends on package A that has been rebuilt. Perhaps package A is slotted, so I don't need to rebuilt. But then when I want to remove the old version, portage should realise that while the source dependencies are still met, the binary dependencies have not been met, so a rebuild is needed. Revdep-rebuild is not going to solve this. It is a hack because it does not work on portage/package level. It tries to understand the elf (executable) format and as a result it tries to find libraries and executables (can't find all though, if they are installed at the wrong place) that depend on a particular library. Then it tries to find the package that the dependant library/executable comes from. As however libraries may have unresolved symbols, some libraries (e.g. libapr-util) do not actually state their dependencies. Further revdep-rebuild does not work with downgrades, or with scripts or other things that are not in the elf format. Finally revdep-rebuild needs to scour the filesystem too much. Recording binary dependencies can be used to resolve this. It is however not easy to do. Paul -- Paul de Vrieze Gentoo Developer Mail: [EMAIL PROTECTED] Homepage: http://www.devrieze.net pgp9RhC4YUuuN.pgp Description: PGP signature
Re: [gentoo-portage-dev] DB and binary dependency
On Thursday 16 March 2006 15:24, Brian Harring wrote: I would have called bincompat BINSLOT, but the idea is the same. As per the norm, requires a smart resolver; for c++ would expect cycles to occur where the only solution is to pull in libstdc++ (fex) to sidestep horkage while doing the rebuilds... Don't start about c++. Suppose you have a library A that is build against libstdc++ 5, It contains a class B with a toString method that returns an std::string instance. Next we have application C that extends class B in class D. It (also) redefines the toString method. This application is binary and we don't have (or don't want to use) the source. It was however linked against libstdc++ 6. Question: Which std::string should class D return in it's toString method? The one from libstdc++5 or from libstdc++6 ? (And for a moment disregard the fact that actually the string classes are binary compatible so it doesn't matter). -- answer follows after some lines --- Answer: There is no right answer. We have hit a situation whose solution is undefined. In reality this is solved by the fact that the dynamic linker does not look at which library a particular symbol (read name) is from. With linker symbol versions, what happens is that the types are silently cast. This however breaks runtime type information. We can namely now get the situation where two std::string instances do not actually have the same runtime type. This breaks casting. Fun isn't it. Paul -- Paul de Vrieze Gentoo Developer Mail: [EMAIL PROTECTED] Homepage: http://www.devrieze.net pgpTksQAd6Qop.pgp Description: PGP signature
