Re: [all] When to update dependencies?
> On Jul 25, 2020, at 5:28 PM, Matt Sicker wrote: > > Exactly. It’s not a general problem, but one specific to various legacy > APIs still commonly used. +1 > >> On Sat, Jul 25, 2020 at 14:56 Bernd Eckenfels >> wrote: >> >> It will only cause problems if this is registered as a service/spi/driver >> like XML, Crypto, JDBC, Authentication and so on. For internal libraries >> functions which don't have such global state it's fine. >> >> Gruss >> Bernd >> >> >> -- >> http://bernd.eckenfels.net >> >> Von: Torsten Curdt >> Gesendet: Saturday, July 25, 2020 8:29:21 PM >> An: Commons Developers List >> Betreff: Re: [all] When to update dependencies? >> >>> >>> They’re not the same class and that’s the problem. It’s similar to >> loading >>> two versions of the same library in OSGi >> >> >> Let's say you are dealing with >> >> org.vafer.Foo >> >> the library relocates this as >> >> org.apache.commons.compress.internal.org.vafer.Foo >> >> The library only deals with >> >> org.apache.commons.compress.internal.org.vafer.Foo >> >> They cannot get mixed up. >> Please explain how this will cause problems. >> > -- > Matt Sicker - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [all] When to update dependencies?
Exactly. It’s not a general problem, but one specific to various legacy APIs still commonly used. On Sat, Jul 25, 2020 at 14:56 Bernd Eckenfels wrote: > It will only cause problems if this is registered as a service/spi/driver > like XML, Crypto, JDBC, Authentication and so on. For internal libraries > functions which don't have such global state it's fine. > > Gruss > Bernd > > > -- > http://bernd.eckenfels.net > > Von: Torsten Curdt > Gesendet: Saturday, July 25, 2020 8:29:21 PM > An: Commons Developers List > Betreff: Re: [all] When to update dependencies? > > > > > They’re not the same class and that’s the problem. It’s similar to > loading > > two versions of the same library in OSGi > > > Let's say you are dealing with > > org.vafer.Foo > > the library relocates this as > > org.apache.commons.compress.internal.org.vafer.Foo > > The library only deals with > > org.apache.commons.compress.internal.org.vafer.Foo > > They cannot get mixed up. > Please explain how this will cause problems. > -- Matt Sicker
Re: [all] When to update dependencies?
When you copy/shade classes then there is no sealing metadata copied, so unless the code verifies how it is packaged, it will not know that it isn't sealed anymore (or to a different package). But I guess it doesn't matter, there are many libraries which can be shaded and many which can't, for various reasons. That does not mean it's generally not possible to reduce some dependencies. Gruss -- http://bernd.eckenfels.net Von: Gary Gregory Gesendet: Saturday, July 25, 2020 10:22:58 PM An: Commons Developers List Betreff: Re: [all] When to update dependencies? How does this work with sealed jars? Gary On Sat, Jul 25, 2020, 15:57 Bernd Eckenfels wrote: > It will only cause problems if this is registered as a service/spi/driver > like XML, Crypto, JDBC, Authentication and so on. For internal libraries > functions which don't have such global state it's fine. > > Gruss > Bernd > > > -- > http://bernd.eckenfels.net > > Von: Torsten Curdt > Gesendet: Saturday, July 25, 2020 8:29:21 PM > An: Commons Developers List > Betreff: Re: [all] When to update dependencies? > > > > > They’re not the same class and that’s the problem. It’s similar to > loading > > two versions of the same library in OSGi > > > Let's say you are dealing with > > org.vafer.Foo > > the library relocates this as > > org.apache.commons.compress.internal.org.vafer.Foo > > The library only deals with > > org.apache.commons.compress.internal.org.vafer.Foo > > They cannot get mixed up. > Please explain how this will cause problems. >
Re: [all] When to update dependencies?
How does this work with sealed jars? Gary On Sat, Jul 25, 2020, 15:57 Bernd Eckenfels wrote: > It will only cause problems if this is registered as a service/spi/driver > like XML, Crypto, JDBC, Authentication and so on. For internal libraries > functions which don't have such global state it's fine. > > Gruss > Bernd > > > -- > http://bernd.eckenfels.net > > Von: Torsten Curdt > Gesendet: Saturday, July 25, 2020 8:29:21 PM > An: Commons Developers List > Betreff: Re: [all] When to update dependencies? > > > > > They’re not the same class and that’s the problem. It’s similar to > loading > > two versions of the same library in OSGi > > > Let's say you are dealing with > > org.vafer.Foo > > the library relocates this as > > org.apache.commons.compress.internal.org.vafer.Foo > > The library only deals with > > org.apache.commons.compress.internal.org.vafer.Foo > > They cannot get mixed up. > Please explain how this will cause problems. >
Re: [all] When to update dependencies?
It will only cause problems if this is registered as a service/spi/driver like XML, Crypto, JDBC, Authentication and so on. For internal libraries functions which don't have such global state it's fine. Gruss Bernd -- http://bernd.eckenfels.net Von: Torsten Curdt Gesendet: Saturday, July 25, 2020 8:29:21 PM An: Commons Developers List Betreff: Re: [all] When to update dependencies? > > They’re not the same class and that’s the problem. It’s similar to loading > two versions of the same library in OSGi Let's say you are dealing with org.vafer.Foo the library relocates this as org.apache.commons.compress.internal.org.vafer.Foo The library only deals with org.apache.commons.compress.internal.org.vafer.Foo They cannot get mixed up. Please explain how this will cause problems.
Re: [all] When to update dependencies?
> > They’re not the same class and that’s the problem. It’s similar to loading > two versions of the same library in OSGi Let's say you are dealing with org.vafer.Foo the library relocates this as org.apache.commons.compress.internal.org.vafer.Foo The library only deals with org.apache.commons.compress.internal.org.vafer.Foo They cannot get mixed up. Please explain how this will cause problems.
Re: [all] When to update dependencies?
They’re not the same class and that’s the problem. It’s similar to loading two versions of the same library in OSGi and allowing the two classes to get mixed up somehow (fun errors like Foo is not type Foo), typically a bug. On Fri, Jul 24, 2020 at 11:30 Gilles Sadowski wrote: > Le ven. 24 juil. 2020 à 17:46, Matt Sicker a écrit : > > > > Shading also violates a lot of common ClassLoader assumptions like not > > having multiple copies of the same class in the same visible context > (even > > if different packages) > > How classes in different packages could be the same class? > > Gilles > > > [...] > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > > -- Matt Sicker
Re: [all] When to update dependencies?
On 7/24/20 1:04 AM, Stefan Bodewig wrote: Hi all here I'd like to explain why I prefer not to update dependencies just because we can. Maybe you can convince me that I'm wrong. I've tried to make this point in different threads but either it has been lost or it just wasn't worth discussing. First of all let me get a few things out of the way * I'm not talking about emails, I can deal with them * I don't care whether a bot or a human asks for a version update * I'm only talking about dependencies that are visible to our users. Test time dependencies or versions of Maven plugins are probably not an issue. Although Compress has mananaged to break its OSGi bundle just by upgrading the parent POM in the past. https://issues.apache.org/jira/browse/COMPRESS-498 All our components have downstream users. I.e. our dependencies become somebody else's dependencies as well. Let's say commons-foo 1.1.0 depends on A 1.12.4 and bumps the dependency to A 1.12.18 for commons-foo 1.2.0. When a user of commons-foo upgrades to 1.2.0 and hasn't defined their dependency on A explicitly they will also upgrade A to 1.12.18. This may be fine or it may cause problems. The new version of A may have made incompatible changes that break the user's code or it may just have bugs that were not present in A 1.12.4 and now raise their head. Of course the users can explicitly state a dependency on A 1.12.4 themselves. But there is no guarantee commons-foo compiled against A 1.12.18 will still work with A 1.12.4. About fifteen years ago Ant was bitten by StringBuffer adding a new method append(StringBuffer) in Java 1.4 (if memory serves me right). Code that called someStringBuffer.append(anotherStringBuffer) compiled on Java 1.3 would call append(Object), but compiled on 1.4 it would call the new version and thus could not run on Java 1.3. This is the kind of change animal sniffer was invented to detect and the --release option of javac deals with. There is no such tool helping us with APIs that are not part of the Java classlib. Therefore I believe updating a dependency is a risk and we should leave it to the users to decide which version they want to use. Unless we've got real reasons to update. Real reasons IMHO are security issues, bugs in dependencies causing bugs in our code or when we really want to use new features introduced in a new version. +1 Above seems a good list of reasons to me. I would also add that we should try as much as possible to avoid dependencies in Commons components. There is no need to update dependencies that don't exist :) I agree with Gary though that shading / pasting should not be used to avoid dependencies in Commons components. Some can't avoid having dependencies. When they do, I think your criteria for when to update is good. I agree strongly with the point you make below about the need to care more about our downstream users than ourselves or upstream. If we pass too many regression or compatibility bugs forward, we will eventually lose the former, which makes our components kind of pointless. I also agree with your view on automatic transitive dependency resolution. Violates what I call the law of conservation of real work. But that ship has sailed... Phil Outside of these good reasons I wouldn't want to ever update a dependency. Stefan - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [all] When to update dependencies?
> > > - library L1 shades library ShadedA-1.0 and ShadedB-1.1. > >> - library L2 shades library ShadedA-1.1 and ShadedB-1.0. > >> - An app wants to use L1, L2, ShadedA-1.1, and ShadedB-1.1 but it can't > no > >> matter what classpath ordering it uses. > >> - An app wants to use L1, L2, ShadedA-1.0, and ShadedB-1.0 but it can't > no > >> matter what classpath ordering it uses. > > I agree here. Shading is quite subtle indeed and can cause Jar hell. I see > what you’re saying. Hm…this does indeed become interesting. > I just cannot cause jar hell. Stop with the FUD or explain how it is a problem. The above is an example of what it prevents.
Re: [all] When to update dependencies?
> > The key phrase is "which would be stupid" which you as a consumer have zero > control over. People do stupid things all the time. > If people use a package that is called "internal" or something - there is nothing anyone can do. And really beyond the point. > > To be more explicit. I am maintaining org.vafer.jdependency which uses > > org.objectweb.asm. > > If you look at the final jar you find > > > > org/vafer/jdeb/shaded/objectweb/asm/*.class > > > > The shaded classes are relocated and become part of the context of the > > library that is shading. > > A shading hell is just not possible as long as there are no classpath > > problems on the library itself. > > > > And this contributes to the problem: This library has now made it > impossible to pick up a fix in ASM or just plain support for a new Java > version. > It does not - at all ...and is a completely different topic.
Re: [all] When to update dependencies?
Le ven. 24 juil. 2020 à 17:39, Gary Gregory a écrit : > > On Fri, Jul 24, 2020 at 11:30 AM Rob Tompkins wrote: > > > > > > > > On Jul 24, 2020, at 9:41 AM, Torsten Curdt wrote: > > > > > >> > > >> You can imagine all manner of jar-hell created by shading. For > > instance: > > >> > > > - library L1 shades library ShadedA-1.0 and ShadedB-1.1. > > >> - library L2 shades library ShadedA-1.1 and ShadedB-1.0. > > >> - An app wants to use L1, L2, ShadedA-1.1, and ShadedB-1.1 but it can't > > no > > >> matter what classpath ordering it uses. > > >> - An app wants to use L1, L2, ShadedA-1.0, and ShadedB-1.0 but it can't > > no > > >> matter what classpath ordering it uses. > > > > I agree here. Shading is quite subtle indeed and can cause Jar hell. I see > > what you’re saying. Hm…this does indeed become interesting. > > > > I suppose we need to have a standard pattern for shading when to, when not > > to, etc. so that we can more effectively have dependency upgrade > > automation?? > > > > That said, the point that you make is indeed subtle enough that unless we > > are very simplistic with our shading, dependency upversioning and > > compatibility can > > become an NP-complete problem. Thus we need standards here. > > > > My standard is simple: Don't shade if you offer a jar for reuse. A shaded class is not part of the public API; how does it affect reuse? I also don't understand your expectation in the Jacoco example: Why should it support more recent class files than those for which that particular version was released? Assuming that Jacoco did not shade that library, perhaps some things would still work with another version of it, while other things would break (I think this was the scenario pointed out by Stefan). > Anything offered as a FOSS jar as the potential for reuse and therefore > becomes a potential contributor to shade hell. I'd also favour explicit dependency over shading. The possibility was presented as an alternative to the "coding horrors" that arise in Commons because of the "no dependency" argument. Gilles > > Gary > > >> [...] - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [all] When to update dependencies?
Le ven. 24 juil. 2020 à 17:46, Matt Sicker a écrit : > > Shading also violates a lot of common ClassLoader assumptions like not > having multiple copies of the same class in the same visible context (even > if different packages) How classes in different packages could be the same class? Gilles > [...] - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [all] When to update dependencies?
On Fri, Jul 24, 2020 at 12:00 PM Gary Gregory wrote: > On Fri, Jul 24, 2020 at 11:43 AM Rob Tompkins wrote: > >> >> >> > On Jul 24, 2020, at 11:36 AM, Gary Gregory >> wrote: >> > >> > On Fri, Jul 24, 2020 at 11:30 AM Rob Tompkins >> wrote: >> > >> >> >> >> >> >>> On Jul 24, 2020, at 9:41 AM, Torsten Curdt wrote: >> >>> >> >> You can imagine all manner of jar-hell created by shading. For >> >> instance: >> >> >>> - library L1 shades library ShadedA-1.0 and ShadedB-1.1. >> - library L2 shades library ShadedA-1.1 and ShadedB-1.0. >> - An app wants to use L1, L2, ShadedA-1.1, and ShadedB-1.1 but it >> can't >> >> no >> matter what classpath ordering it uses. >> - An app wants to use L1, L2, ShadedA-1.0, and ShadedB-1.0 but it >> can't >> >> no >> matter what classpath ordering it uses. >> >> >> >> I agree here. Shading is quite subtle indeed and can cause Jar hell. I >> see >> >> what you’re saying. Hm…this does indeed become interesting. >> >> >> >> I suppose we need to have a standard pattern for shading when to, when >> not >> >> to, etc. so that we can more effectively have dependency upgrade >> >> automation?? >> >> >> >> That said, the point that you make is indeed subtle enough that unless >> we >> >> are very simplistic with our shading, dependency upversioning and >> >> compatibility can >> >> become an NP-complete problem. Thus we need standards here. >> >> >> > >> > My standard is simple: Don't shade if you offer a jar for reuse. >> > >> > Anything offered as a FOSS jar as the potential for reuse and therefore >> > becomes a potential contributor to shade hell. >> >> Sounds like a good place to start. Don’t we shade lang into text though >> (could totally be wrong here)? >> > > Text does not shade lang. Commons, as far as I know, does not shade > anything, and IMO should not. > It is critical to understand the difference between shading vs. depending. In "depending", Text expects Lang to be on the classpath, and this is enforced at build time by Maven and your POM such that tests can run. In shading (with the Maven shade plugin for example), the build _copies_ the class files from a third party jar into _your_ jar file. Whether you keep the original package names or repackage the third party code is your choice. Gary > > Gary > > >> >> -Rob >> >> > >> > Gary >> > >> > >> >> -Rob >> >> >> >> >>> >> >>> Sorry, but I cannot follow the problems you are trying to show. >> >>> >> >>> L1 has ShadeA and ShadeB >> >>> L2 has ShadeA and ShadeB >> >>> but both have their own version that doesn't clash and that doesn't >> know >> >>> anything about the other one. >> >>> The app does not see ShadeA or ShadeB from L1 or L2 (unless it uses >> the >> >>> hidden package from L1 which would be stupid) >> >>> There are no clashes and every library uses the version that it needs. >> >>> >> >>> To be more explicit. I am maintaining org.vafer.jdependency which uses >> >>> org.objectweb.asm. >> >>> If you look at the final jar you find >> >>> >> >>> org/vafer/jdeb/shaded/objectweb/asm/*.class >> >>> >> >>> The shaded classes are relocated and become part of the context of the >> >>> library that is shading. >> >>> A shading hell is just not possible as long as there are no classpath >> >>> problems on the library itself. >> >>> >> >>> It's like you are copy the code into your package. >> >>> Bytecode manipulation is really not that bad as you make it to be. >> >>> >> >>> Just creating uberjars (without relocation) - that's a whole >> different >> >>> story. >> >>> That should only ever been done on the final application artifact. >> >> >> >> >> >> - >> >> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >> >> For additional commands, e-mail: dev-h...@commons.apache.org >> >> >> >> >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >> For additional commands, e-mail: dev-h...@commons.apache.org >> >>
Re: [all] When to update dependencies?
> On Jul 24, 2020, at 12:00 PM, Gary Gregory wrote: > > On Fri, Jul 24, 2020 at 11:43 AM Rob Tompkins wrote: > >> >> >>> On Jul 24, 2020, at 11:36 AM, Gary Gregory >> wrote: >>> >>> On Fri, Jul 24, 2020 at 11:30 AM Rob Tompkins >> wrote: >>> > On Jul 24, 2020, at 9:41 AM, Torsten Curdt wrote: > >> >> You can imagine all manner of jar-hell created by shading. For instance: >> > - library L1 shades library ShadedA-1.0 and ShadedB-1.1. >> - library L2 shades library ShadedA-1.1 and ShadedB-1.0. >> - An app wants to use L1, L2, ShadedA-1.1, and ShadedB-1.1 but it >> can't no >> matter what classpath ordering it uses. >> - An app wants to use L1, L2, ShadedA-1.0, and ShadedB-1.0 but it >> can't no >> matter what classpath ordering it uses. I agree here. Shading is quite subtle indeed and can cause Jar hell. I >> see what you’re saying. Hm…this does indeed become interesting. I suppose we need to have a standard pattern for shading when to, when >> not to, etc. so that we can more effectively have dependency upgrade automation?? That said, the point that you make is indeed subtle enough that unless >> we are very simplistic with our shading, dependency upversioning and compatibility can become an NP-complete problem. Thus we need standards here. >>> >>> My standard is simple: Don't shade if you offer a jar for reuse. >>> >>> Anything offered as a FOSS jar as the potential for reuse and therefore >>> becomes a potential contributor to shade hell. >> >> Sounds like a good place to start. Don’t we shade lang into text though >> (could totally be wrong here)? >> > > Text does not shade lang. Commons, as far as I know, does not shade > anything, and IMO should not. > Good! I wholeheartedly agree, and this get’s us out of this discussion I believe. Our consumers may deal with shading as they see fit. But, staying up to date on dependencies in an automated fashion is invaluable. I think this is the way we prevent security vulnerabilities cropping up. -Rob > Gary > > >> >> -Rob >> >>> >>> Gary >>> >>> -Rob >> > > Sorry, but I cannot follow the problems you are trying to show. > > L1 has ShadeA and ShadeB > L2 has ShadeA and ShadeB > but both have their own version that doesn't clash and that doesn't >> know > anything about the other one. > The app does not see ShadeA or ShadeB from L1 or L2 (unless it uses the > hidden package from L1 which would be stupid) > There are no clashes and every library uses the version that it needs. > > To be more explicit. I am maintaining org.vafer.jdependency which uses > org.objectweb.asm. > If you look at the final jar you find > > org/vafer/jdeb/shaded/objectweb/asm/*.class > > The shaded classes are relocated and become part of the context of the > library that is shading. > A shading hell is just not possible as long as there are no classpath > problems on the library itself. > > It's like you are copy the code into your package. > Bytecode manipulation is really not that bad as you make it to be. > > Just creating uberjars (without relocation) - that's a whole different > story. > That should only ever been done on the final application artifact. - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >> For additional commands, e-mail: dev-h...@commons.apache.org >> >> - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [all] When to update dependencies?
On Fri, Jul 24, 2020 at 11:43 AM Rob Tompkins wrote: > > > > On Jul 24, 2020, at 11:36 AM, Gary Gregory > wrote: > > > > On Fri, Jul 24, 2020 at 11:30 AM Rob Tompkins > wrote: > > > >> > >> > >>> On Jul 24, 2020, at 9:41 AM, Torsten Curdt wrote: > >>> > > You can imagine all manner of jar-hell created by shading. For > >> instance: > > >>> - library L1 shades library ShadedA-1.0 and ShadedB-1.1. > - library L2 shades library ShadedA-1.1 and ShadedB-1.0. > - An app wants to use L1, L2, ShadedA-1.1, and ShadedB-1.1 but it > can't > >> no > matter what classpath ordering it uses. > - An app wants to use L1, L2, ShadedA-1.0, and ShadedB-1.0 but it > can't > >> no > matter what classpath ordering it uses. > >> > >> I agree here. Shading is quite subtle indeed and can cause Jar hell. I > see > >> what you’re saying. Hm…this does indeed become interesting. > >> > >> I suppose we need to have a standard pattern for shading when to, when > not > >> to, etc. so that we can more effectively have dependency upgrade > >> automation?? > >> > >> That said, the point that you make is indeed subtle enough that unless > we > >> are very simplistic with our shading, dependency upversioning and > >> compatibility can > >> become an NP-complete problem. Thus we need standards here. > >> > > > > My standard is simple: Don't shade if you offer a jar for reuse. > > > > Anything offered as a FOSS jar as the potential for reuse and therefore > > becomes a potential contributor to shade hell. > > Sounds like a good place to start. Don’t we shade lang into text though > (could totally be wrong here)? > Text does not shade lang. Commons, as far as I know, does not shade anything, and IMO should not. Gary > > -Rob > > > > > Gary > > > > > >> -Rob > >> > > >>> > >>> Sorry, but I cannot follow the problems you are trying to show. > >>> > >>> L1 has ShadeA and ShadeB > >>> L2 has ShadeA and ShadeB > >>> but both have their own version that doesn't clash and that doesn't > know > >>> anything about the other one. > >>> The app does not see ShadeA or ShadeB from L1 or L2 (unless it uses the > >>> hidden package from L1 which would be stupid) > >>> There are no clashes and every library uses the version that it needs. > >>> > >>> To be more explicit. I am maintaining org.vafer.jdependency which uses > >>> org.objectweb.asm. > >>> If you look at the final jar you find > >>> > >>> org/vafer/jdeb/shaded/objectweb/asm/*.class > >>> > >>> The shaded classes are relocated and become part of the context of the > >>> library that is shading. > >>> A shading hell is just not possible as long as there are no classpath > >>> problems on the library itself. > >>> > >>> It's like you are copy the code into your package. > >>> Bytecode manipulation is really not that bad as you make it to be. > >>> > >>> Just creating uberjars (without relocation) - that's a whole different > >>> story. > >>> That should only ever been done on the final application artifact. > >> > >> > >> - > >> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > >> For additional commands, e-mail: dev-h...@commons.apache.org > >> > >> > > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > >
Re: [all] When to update dependencies?
Shading also violates a lot of common ClassLoader assumptions like not having multiple copies of the same class in the same visible context (even if different packages) and gets more complex as your types do. I’ve seen this problem when projects tried to shade Avro or Protobuf. As a downstream user of the library, it becomes impossible to use the upstream code without using its shaded code. Of course, this is not always the case, but it’s usually sloppy. That’s not even counting reflection APIs. On Fri, Jul 24, 2020 at 10:34 Gary Gregory wrote: > On Fri, Jul 24, 2020 at 10:50 AM Gilles Sadowski > wrote: > > > Hello. > > > > Le ven. 24 juil. 2020 à 14:48, Gary Gregory a > > écrit : > > > > > > On Fri, Jul 24, 2020 at 6:03 AM Gilles Sadowski > > > wrote: > > > > > > > 2020-07-24 11:25 UTC+02:00, Torsten Curdt : > > > > > It still needs a person to decide to merge a PR for a new version. > > > > > So this indeed is just about the dependency upgrade policies. > > > > > > > > > > But isn't that what the version definition is for? > > > > > > > > Ideally. > > > > In practice, I think that all we can assume is that the version > > > > string is a quick-glance summary of changes, for human > > > > consumption. > > > > > > > > > I'd argue that 1.12.4 <-> 1.12.6 should be a compatible upgrade AND > > > > > downgrade, > > > > > 1.12.4 -> 1.20.0 not so much. > > > > > > > > > > But to avoid all this is why I usually try to inline dependencies > for > > > > > libraries as much as possible. Basically pretending to not have > any. > > > > > This of course depends on whether the dependency can be isolated > that > > > > way. > > > > > > > > > > Also a point I made many times. > > > > > Just wanted to mention it - again :) > > > > > > > > I think that it would be great that "Commons" has a common > > > > policy for dealing with this (so that we don't have to repeat > > > > ourselves every now and then). > > > > A long time ago, the "shade" feature seemed the perfect > > > > answer to that problem. Yet, to avoid dependencies even > > > > on another Commons component, several of us continue to > > > > support the copy/paste option. > > > > > > > > > > There are a few items I'd like to address in this thread so I'll start > > with > > > the above. > > > > > > I would like to offer my view on shading and copy/paste as - to borrow > > Jeff > > > Atwood's expression - a "coding horror". While you may think that there > > are > > > still edge cases where this may seem to some as necessary or > expedient, I > > > hope to dissuade you from these practices. > > > > > > The first and most obvious horror to me is the copying and pasting of > > bugs > > > and hard to maintain code. You may also end up taking along supporting > > > methods and classes to support the code you copy. While this may seem > to > > > follow the YAGNI principle, my claim is that you can end up with an > > iceberg > > > of new code, where the new API surface above the water does not end up > > > exercising all of the supporting code you had to drag along that lies > > > below. > > > > > > The question where I am most curious to hear back from the copy clan > is: > > Do > > > you also copy the unit tests that cover the code you copied? No? Coding > > > horror #2. > > > > > > The third horror does not affect us within Apache but its general > > practice > > > begs its adherents to answer: Did you check the license of the code you > > > copied? No? Coding horror #3. Again, this is not an issue within Apache > > > projects, but the question pops up time and time again at Apache (and > at > > > work): Can I copy this code I found outside of Apache? > > > > > > The fourth and final copy horror falls in the same category as the > first > > > and deserves its own mention: security; and can be best understood with > > > this question: How often do you check for CVEs for the library > containing > > > the code you copied? Never? Coding horror #4. > > > > Agreeing on all the above. > > > > Yet, every time there is an opportunity to engage in this wise > > path, the copying/pasting continues and slowly grows the iceberg. > > A recent example is the issue with pseudo-random numbers > > generation where maintaining duplicate functionality in [Lang] > > was preferred even though [RNG] has 100% coverage (and a > > explicit scope that entails it will never have any dependency of > > its own). > > > > Would you be willing to provide a PR that deprecates the relevant APIs and > points to their equivalent in RNG? It will be more cruft we can trim for > 4.0, whenever that happens. > > > > > Shading has its limited use in my view, and I've used it to provide > > > monolithic versions of applications, but beyond that, I've no use for > it; > > > but the pain it can cause is very real indeed. > > > > > > You can imagine all manner of jar-hell created by shading. For > instance: > > > - library L1 shades library ShadedA-1.0 and ShadedB-1.1. > > > - library L2 shades library
Re: [all] When to update dependencies?
> On Jul 24, 2020, at 11:36 AM, Gary Gregory wrote: > > On Fri, Jul 24, 2020 at 11:30 AM Rob Tompkins wrote: > >> >> >>> On Jul 24, 2020, at 9:41 AM, Torsten Curdt wrote: >>> You can imagine all manner of jar-hell created by shading. For >> instance: >>> - library L1 shades library ShadedA-1.0 and ShadedB-1.1. - library L2 shades library ShadedA-1.1 and ShadedB-1.0. - An app wants to use L1, L2, ShadedA-1.1, and ShadedB-1.1 but it can't >> no matter what classpath ordering it uses. - An app wants to use L1, L2, ShadedA-1.0, and ShadedB-1.0 but it can't >> no matter what classpath ordering it uses. >> >> I agree here. Shading is quite subtle indeed and can cause Jar hell. I see >> what you’re saying. Hm…this does indeed become interesting. >> >> I suppose we need to have a standard pattern for shading when to, when not >> to, etc. so that we can more effectively have dependency upgrade >> automation?? >> >> That said, the point that you make is indeed subtle enough that unless we >> are very simplistic with our shading, dependency upversioning and >> compatibility can >> become an NP-complete problem. Thus we need standards here. >> > > My standard is simple: Don't shade if you offer a jar for reuse. > > Anything offered as a FOSS jar as the potential for reuse and therefore > becomes a potential contributor to shade hell. Sounds like a good place to start. Don’t we shade lang into text though (could totally be wrong here)? -Rob > > Gary > > >> -Rob >> >>> >>> Sorry, but I cannot follow the problems you are trying to show. >>> >>> L1 has ShadeA and ShadeB >>> L2 has ShadeA and ShadeB >>> but both have their own version that doesn't clash and that doesn't know >>> anything about the other one. >>> The app does not see ShadeA or ShadeB from L1 or L2 (unless it uses the >>> hidden package from L1 which would be stupid) >>> There are no clashes and every library uses the version that it needs. >>> >>> To be more explicit. I am maintaining org.vafer.jdependency which uses >>> org.objectweb.asm. >>> If you look at the final jar you find >>> >>> org/vafer/jdeb/shaded/objectweb/asm/*.class >>> >>> The shaded classes are relocated and become part of the context of the >>> library that is shading. >>> A shading hell is just not possible as long as there are no classpath >>> problems on the library itself. >>> >>> It's like you are copy the code into your package. >>> Bytecode manipulation is really not that bad as you make it to be. >>> >>> Just creating uberjars (without relocation) - that's a whole different >>> story. >>> That should only ever been done on the final application artifact. >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >> For additional commands, e-mail: dev-h...@commons.apache.org >> >> - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [all] When to update dependencies?
On Fri, Jul 24, 2020 at 11:30 AM Rob Tompkins wrote: > > > > On Jul 24, 2020, at 9:41 AM, Torsten Curdt wrote: > > > >> > >> You can imagine all manner of jar-hell created by shading. For > instance: > >> > > - library L1 shades library ShadedA-1.0 and ShadedB-1.1. > >> - library L2 shades library ShadedA-1.1 and ShadedB-1.0. > >> - An app wants to use L1, L2, ShadedA-1.1, and ShadedB-1.1 but it can't > no > >> matter what classpath ordering it uses. > >> - An app wants to use L1, L2, ShadedA-1.0, and ShadedB-1.0 but it can't > no > >> matter what classpath ordering it uses. > > I agree here. Shading is quite subtle indeed and can cause Jar hell. I see > what you’re saying. Hm…this does indeed become interesting. > > I suppose we need to have a standard pattern for shading when to, when not > to, etc. so that we can more effectively have dependency upgrade > automation?? > > That said, the point that you make is indeed subtle enough that unless we > are very simplistic with our shading, dependency upversioning and > compatibility can > become an NP-complete problem. Thus we need standards here. > My standard is simple: Don't shade if you offer a jar for reuse. Anything offered as a FOSS jar as the potential for reuse and therefore becomes a potential contributor to shade hell. Gary > -Rob > > >> > > > > Sorry, but I cannot follow the problems you are trying to show. > > > > L1 has ShadeA and ShadeB > > L2 has ShadeA and ShadeB > > but both have their own version that doesn't clash and that doesn't know > > anything about the other one. > > The app does not see ShadeA or ShadeB from L1 or L2 (unless it uses the > > hidden package from L1 which would be stupid) > > There are no clashes and every library uses the version that it needs. > > > > To be more explicit. I am maintaining org.vafer.jdependency which uses > > org.objectweb.asm. > > If you look at the final jar you find > > > > org/vafer/jdeb/shaded/objectweb/asm/*.class > > > > The shaded classes are relocated and become part of the context of the > > library that is shading. > > A shading hell is just not possible as long as there are no classpath > > problems on the library itself. > > > > It's like you are copy the code into your package. > > Bytecode manipulation is really not that bad as you make it to be. > > > > Just creating uberjars (without relocation) - that's a whole different > > story. > > That should only ever been done on the final application artifact. > > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > >
Re: [all] When to update dependencies?
On Fri, Jul 24, 2020 at 10:50 AM Gilles Sadowski wrote: > Hello. > > Le ven. 24 juil. 2020 à 14:48, Gary Gregory a > écrit : > > > > On Fri, Jul 24, 2020 at 6:03 AM Gilles Sadowski > > wrote: > > > > > 2020-07-24 11:25 UTC+02:00, Torsten Curdt : > > > > It still needs a person to decide to merge a PR for a new version. > > > > So this indeed is just about the dependency upgrade policies. > > > > > > > > But isn't that what the version definition is for? > > > > > > Ideally. > > > In practice, I think that all we can assume is that the version > > > string is a quick-glance summary of changes, for human > > > consumption. > > > > > > > I'd argue that 1.12.4 <-> 1.12.6 should be a compatible upgrade AND > > > > downgrade, > > > > 1.12.4 -> 1.20.0 not so much. > > > > > > > > But to avoid all this is why I usually try to inline dependencies for > > > > libraries as much as possible. Basically pretending to not have any. > > > > This of course depends on whether the dependency can be isolated that > > > way. > > > > > > > > Also a point I made many times. > > > > Just wanted to mention it - again :) > > > > > > I think that it would be great that "Commons" has a common > > > policy for dealing with this (so that we don't have to repeat > > > ourselves every now and then). > > > A long time ago, the "shade" feature seemed the perfect > > > answer to that problem. Yet, to avoid dependencies even > > > on another Commons component, several of us continue to > > > support the copy/paste option. > > > > > > > There are a few items I'd like to address in this thread so I'll start > with > > the above. > > > > I would like to offer my view on shading and copy/paste as - to borrow > Jeff > > Atwood's expression - a "coding horror". While you may think that there > are > > still edge cases where this may seem to some as necessary or expedient, I > > hope to dissuade you from these practices. > > > > The first and most obvious horror to me is the copying and pasting of > bugs > > and hard to maintain code. You may also end up taking along supporting > > methods and classes to support the code you copy. While this may seem to > > follow the YAGNI principle, my claim is that you can end up with an > iceberg > > of new code, where the new API surface above the water does not end up > > exercising all of the supporting code you had to drag along that lies > > below. > > > > The question where I am most curious to hear back from the copy clan is: > Do > > you also copy the unit tests that cover the code you copied? No? Coding > > horror #2. > > > > The third horror does not affect us within Apache but its general > practice > > begs its adherents to answer: Did you check the license of the code you > > copied? No? Coding horror #3. Again, this is not an issue within Apache > > projects, but the question pops up time and time again at Apache (and at > > work): Can I copy this code I found outside of Apache? > > > > The fourth and final copy horror falls in the same category as the first > > and deserves its own mention: security; and can be best understood with > > this question: How often do you check for CVEs for the library containing > > the code you copied? Never? Coding horror #4. > > Agreeing on all the above. > > Yet, every time there is an opportunity to engage in this wise > path, the copying/pasting continues and slowly grows the iceberg. > A recent example is the issue with pseudo-random numbers > generation where maintaining duplicate functionality in [Lang] > was preferred even though [RNG] has 100% coverage (and a > explicit scope that entails it will never have any dependency of > its own). > Would you be willing to provide a PR that deprecates the relevant APIs and points to their equivalent in RNG? It will be more cruft we can trim for 4.0, whenever that happens. > > Shading has its limited use in my view, and I've used it to provide > > monolithic versions of applications, but beyond that, I've no use for it; > > but the pain it can cause is very real indeed. > > > > You can imagine all manner of jar-hell created by shading. For instance: > > - library L1 shades library ShadedA-1.0 and ShadedB-1.1. > > - library L2 shades library ShadedA-1.1 and ShadedB-1.0. > > - An app wants to use L1, L2, ShadedA-1.1, and ShadedB-1.1 but it can't > no > > matter what classpath ordering it uses. > > - An app wants to use L1, L2, ShadedA-1.0, and ShadedB-1.0 but it can't > no > > matter what classpath ordering it uses. > > I didn't understand how the problem is supposed to arise. > Fortunately, Torsten already replied to this point. :-) > > Unless I'm mistaken, there is one drawback to shading: Since > it creates new classes, there will be less shared code, hence > one could imagine a potential hit on performance (?). > > > Lastly, there is a special kind of shading jar hell that deserves its own > > section: the copied and re-packaged source/jar. In this lovely planned > > development managed by Hades, an entire
Re: [all] When to update dependencies?
> On Jul 24, 2020, at 9:41 AM, Torsten Curdt wrote: > >> >> You can imagine all manner of jar-hell created by shading. For instance: >> > - library L1 shades library ShadedA-1.0 and ShadedB-1.1. >> - library L2 shades library ShadedA-1.1 and ShadedB-1.0. >> - An app wants to use L1, L2, ShadedA-1.1, and ShadedB-1.1 but it can't no >> matter what classpath ordering it uses. >> - An app wants to use L1, L2, ShadedA-1.0, and ShadedB-1.0 but it can't no >> matter what classpath ordering it uses. I agree here. Shading is quite subtle indeed and can cause Jar hell. I see what you’re saying. Hm…this does indeed become interesting. I suppose we need to have a standard pattern for shading when to, when not to, etc. so that we can more effectively have dependency upgrade automation?? That said, the point that you make is indeed subtle enough that unless we are very simplistic with our shading, dependency upversioning and compatibility can become an NP-complete problem. Thus we need standards here. -Rob >> > > Sorry, but I cannot follow the problems you are trying to show. > > L1 has ShadeA and ShadeB > L2 has ShadeA and ShadeB > but both have their own version that doesn't clash and that doesn't know > anything about the other one. > The app does not see ShadeA or ShadeB from L1 or L2 (unless it uses the > hidden package from L1 which would be stupid) > There are no clashes and every library uses the version that it needs. > > To be more explicit. I am maintaining org.vafer.jdependency which uses > org.objectweb.asm. > If you look at the final jar you find > > org/vafer/jdeb/shaded/objectweb/asm/*.class > > The shaded classes are relocated and become part of the context of the > library that is shading. > A shading hell is just not possible as long as there are no classpath > problems on the library itself. > > It's like you are copy the code into your package. > Bytecode manipulation is really not that bad as you make it to be. > > Just creating uberjars (without relocation) - that's a whole different > story. > That should only ever been done on the final application artifact. - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [all] When to update dependencies?
On Fri, Jul 24, 2020 at 10:02 AM Torsten Curdt wrote: > > > > You can imagine all manner of jar-hell created by shading. For instance: > > > - library L1 shades library ShadedA-1.0 and ShadedB-1.1. > > - library L2 shades library ShadedA-1.1 and ShadedB-1.0. > > - An app wants to use L1, L2, ShadedA-1.1, and ShadedB-1.1 but it can't > no > > matter what classpath ordering it uses. > > - An app wants to use L1, L2, ShadedA-1.0, and ShadedB-1.0 but it can't > no > > matter what classpath ordering it uses. > > > > Sorry, but I cannot follow the problems you are trying to show. > > L1 has ShadeA and ShadeB > L2 has ShadeA and ShadeB > but both have their own version that doesn't clash and that doesn't know > anything about the other one. > The app does not see ShadeA or ShadeB from L1 or L2 (unless it uses the > hidden package from L1 which would be stupid) > There are no clashes and every library uses the version that it needs. > The key phrase is "which would be stupid" which you as a consumer have zero control over. People do stupid things all the time. > To be more explicit. I am maintaining org.vafer.jdependency which uses > org.objectweb.asm. > If you look at the final jar you find > > org/vafer/jdeb/shaded/objectweb/asm/*.class > > The shaded classes are relocated and become part of the context of the > library that is shading. > A shading hell is just not possible as long as there are no classpath > problems on the library itself. > And this contributes to the problem: This library has now made it impossible to pick up a fix in ASM or just plain support for a new Java version. Gary > It's like you are copy the code into your package. > Bytecode manipulation is really not that bad as you make it to be. > > Just creating uberjars (without relocation) - that's a whole different > story. > That should only ever been done on the final application artifact. >
Re: [all] When to update dependencies?
> > Unless I'm mistaken, there is one drawback to shading: Since > it creates new classes, there will be less shared code, hence > one could imagine a potential hit on performance (?). > Of course it means some code duplication in the final artifact. This means slightly bigger artifacts - but disk space is cheap. In the worst case scenario it means some more memory gets used. For the libraries I am looking at this is insignificant. Same should be the case for performance. But you never know for your use case until you measure it. I see merely a single drawback - and that's when reflection is in the picture. IIRC my original minijar code did work around this for static references, but for dynamic references you were on your own. I think the current incarnation of the shade plugin does not support reflection magic. And maybe it's OK to keep it simpler. I must be missing something: Since shaded classes are supposed > to be "internal", if there is an impact on the public API, the only sane > solution is a bug-fix release that shades the fixed version of the internal > library. No? > That's exactly how it works.
Re: [all] When to update dependencies?
On 2020-07-24, Bernd Eckenfels wrote: > When it comes to dependencies wie have both problems: if we upgrade > dependencies to aggressively (or if we don't test with older dependencies) > then users have the problem that they might not easily be able to upgrade to > a new commons version since the required new dependency version might > conflict with other (thirdparty) users of that dependency. > On the other hand if we not continuously update external dependencies we > might miss out on their new features, performance and fixes. In addition we > might fall behind and have the to do painful Big Bang Upgrades. Also when > our transitive dependencies are outdated and contain bugs (or compliance > violations due to old code) some customers might not be happy. I hear you. I'm not opposed to updating a dependency if we want to use new features. What I'd like to avoid is updating without a reason other than "there is a new version and it seems to work according to our tests". > So there is a middle ground to be found, which unfortunately collides with > the current limited effort maintenance of some of the components: > - we should define a minimum baseline version of dependencies and runtimes > and on each release we check if we still meet them. When we raise the > baseline we should ship a new minor (or even major) version. Also we might > want to ship security fixes only as a micro update (I.e. not requiring major > updates besides the affected code) One problem with patch updates for security releases is the release vote will reveal a security update is in the making and the diff will be small enough to give away the details before the vote has passed. I may be paranoid. > - we should regularly test against latest dependency versions (at least > within the same minor branch). Apache Gump? What you describe sounds good, but "unfortunately collides with the current limited effort maintenance of some of the components". Stefan - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [all] When to update dependencies?
Hello. Le ven. 24 juil. 2020 à 14:48, Gary Gregory a écrit : > > On Fri, Jul 24, 2020 at 6:03 AM Gilles Sadowski > wrote: > > > 2020-07-24 11:25 UTC+02:00, Torsten Curdt : > > > It still needs a person to decide to merge a PR for a new version. > > > So this indeed is just about the dependency upgrade policies. > > > > > > But isn't that what the version definition is for? > > > > Ideally. > > In practice, I think that all we can assume is that the version > > string is a quick-glance summary of changes, for human > > consumption. > > > > > I'd argue that 1.12.4 <-> 1.12.6 should be a compatible upgrade AND > > > downgrade, > > > 1.12.4 -> 1.20.0 not so much. > > > > > > But to avoid all this is why I usually try to inline dependencies for > > > libraries as much as possible. Basically pretending to not have any. > > > This of course depends on whether the dependency can be isolated that > > way. > > > > > > Also a point I made many times. > > > Just wanted to mention it - again :) > > > > I think that it would be great that "Commons" has a common > > policy for dealing with this (so that we don't have to repeat > > ourselves every now and then). > > A long time ago, the "shade" feature seemed the perfect > > answer to that problem. Yet, to avoid dependencies even > > on another Commons component, several of us continue to > > support the copy/paste option. > > > > There are a few items I'd like to address in this thread so I'll start with > the above. > > I would like to offer my view on shading and copy/paste as - to borrow Jeff > Atwood's expression - a "coding horror". While you may think that there are > still edge cases where this may seem to some as necessary or expedient, I > hope to dissuade you from these practices. > > The first and most obvious horror to me is the copying and pasting of bugs > and hard to maintain code. You may also end up taking along supporting > methods and classes to support the code you copy. While this may seem to > follow the YAGNI principle, my claim is that you can end up with an iceberg > of new code, where the new API surface above the water does not end up > exercising all of the supporting code you had to drag along that lies > below. > > The question where I am most curious to hear back from the copy clan is: Do > you also copy the unit tests that cover the code you copied? No? Coding > horror #2. > > The third horror does not affect us within Apache but its general practice > begs its adherents to answer: Did you check the license of the code you > copied? No? Coding horror #3. Again, this is not an issue within Apache > projects, but the question pops up time and time again at Apache (and at > work): Can I copy this code I found outside of Apache? > > The fourth and final copy horror falls in the same category as the first > and deserves its own mention: security; and can be best understood with > this question: How often do you check for CVEs for the library containing > the code you copied? Never? Coding horror #4. Agreeing on all the above. Yet, every time there is an opportunity to engage in this wise path, the copying/pasting continues and slowly grows the iceberg. A recent example is the issue with pseudo-random numbers generation where maintaining duplicate functionality in [Lang] was preferred even though [RNG] has 100% coverage (and a explicit scope that entails it will never have any dependency of its own). > Shading has its limited use in my view, and I've used it to provide > monolithic versions of applications, but beyond that, I've no use for it; > but the pain it can cause is very real indeed. > > You can imagine all manner of jar-hell created by shading. For instance: > - library L1 shades library ShadedA-1.0 and ShadedB-1.1. > - library L2 shades library ShadedA-1.1 and ShadedB-1.0. > - An app wants to use L1, L2, ShadedA-1.1, and ShadedB-1.1 but it can't no > matter what classpath ordering it uses. > - An app wants to use L1, L2, ShadedA-1.0, and ShadedB-1.0 but it can't no > matter what classpath ordering it uses. I didn't understand how the problem is supposed to arise. Fortunately, Torsten already replied to this point. :-) Unless I'm mistaken, there is one drawback to shading: Since it creates new classes, there will be less shared code, hence one could imagine a potential hit on performance (?). > Lastly, there is a special kind of shading jar hell that deserves its own > section: the copied and re-packaged source/jar. In this lovely planned > development managed by Hades, an entire jar has been copied then repackaged > such that instead of being in its original Java package org.foo.library, it > is now under org.other.bar.org.foo.library. Residents of this charming > cottage have now the proud owners of every bug and CVE for that specific > version of the source jar, without, and this is the kicker, the option of > overriding the jar with a new version (I'll leave aside your option to take > a new version of the jar and
Re: [all] When to update dependencies?
On 2020-07-24, Xeno Amess wrote: > how about: > 1. we force versions of dependencies in commons-parent > 2. we make every commons repo use commons-parent as parent. > 3. we make sure no other repos forces versions of dependencies; all of the > versions number shall be inherited from commons-parent > 4. we upgrade versions in commons-parent every several months. As you may guess I'd be strongly opposed to this as it would mean we updated everey few months just because we can, noot because there was a good reason. On top no component could upgrade more agressively if it really needed a feature of a newer version. Apart from that I'd expect some component developers to be rather unhappy if *anything* was forced on them. :-) Honestly I doubt we could even find a common ground on which version of Java to support if we tried to have one version for all components. Stefan - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [all] When to update dependencies?
Hello, We can certainly put some dependencies in commons parent dependency management (and the plugins like we used to), but for a lot of stuff I don't think we will be able to find a common ground, and it will be quite painful if projects no longer can update commons parent without major testing and integration work (plus things like jackrabbit which is only(?) used in vfs might really be overhead to maintain that in a parent) -- http://bernd.eckenfels.net Von: Xeno Amess Gesendet: Friday, July 24, 2020 3:53:18 PM An: Commons Developers List Betreff: Re: [all] When to update dependencies? how about: 1. we force versions of dependencies in commons-parent 2. we make every commons repo use commons-parent as parent. 3. we make sure no other repos forces versions of dependencies; all of the versions number shall be inherited from commons-parent 4. we upgrade versions in commons-parent every several months. Bernd Eckenfels 于2020年7月24日周五 下午9:49写道： > Hello, > > When it comes to dependencies wie have both problems: if we upgrade > dependencies to aggressively (or if we don't test with older dependencies) > then users have the problem that they might not easily be able to upgrade > to a new commons version since the required new dependency version might > conflict with other (thirdparty) users of that dependency. > > On the other hand if we not continuously update external dependencies we > might miss out on their new features, performance and fixes. In addition we > might fall behind and have the to do painful Big Bang Upgrades. Also when > our transitive dependencies are outdated and contain bugs (or compliance > violations due to old code) some customers might not be happy. > > So there is a middle ground to be found, which unfortunately collides with > the current limited effort maintenance of some of the components: > > - we should define a minimum baseline version of dependencies and runtimes > and on each release we check if we still meet them. When we raise the > baseline we should ship a new minor (or even major) version. Also we might > want to ship security fixes only as a micro update (I.e. not requiring > major updates besides the affected code) > - we should regularly test against latest dependency versions (at least > within the same minor branch). Dependbot could be a help here, however I do > find it a bit annoying and not needed as I would run the dependency report > if I have the need to find me something to work on. > > Gruss > Bend > > > -- > http://bernd.eckenfels.net > > Von: Stefan Bodewig > Gesendet: Friday, July 24, 2020 3:36:36 PM > An: Commons Developers List > Betreff: Re: [all] When to update dependencies? > > On 2020-07-24, Gary Gregory wrote: > > > Now back to our code depending on other dependencies. My view is that > there > > are bugs, you just have not hit them. If I find one in a dependency and > it > > gets fixed, it's going to mean a new version of that dependency. > > This either is "we hit a bug that affects us" - where I already say this > is a reason to upgrade for me - or "the user hits a bug" - in which case > I prefer to let the user update the buggy dependency. I don't want to > force the user to update because they might be affected by bugs. > > And of course we both know that newer versions not only fix bugs, they > introduce new ones as well. > > > Furthermore, when I look online for Javadoc and examples, if I use a > > current jar version, then my odds are better that I can implement what I > > find online. > > https://javadoc.io/ or similar services? > > > I view it as simpler and safer to update from a "near" dependency than > > letting a dependency acquire "bit rot" and upgrade later, especially > > if an update means making adjustments. I want to make smaller > > increments of adjustment rather than a larger set. Just like I prefer > > to RERO instead of big bangs for releasing Commons components. > > I fully agree with you if nobody else depends on my stuff. I.e. I'm > working on an application rather than a library. When working on a > library I really want the user to be and stay in control. > > > Another way to look at this is to look at a large software stack: If > every > > library developer never updates dependencies, then your application, > > through transitive dependencies will end up depending (virtually) on many > > versions of the each library, which is much more likely to create jar > hell > > and other problems than the other edge case where everyone uses the > latest > > version (or a fixed version.) > > You are just hitting home on
Re: [all] When to update dependencies?
how about: 1. we force versions of dependencies in commons-parent 2. we make every commons repo use commons-parent as parent. 3. we make sure no other repos forces versions of dependencies; all of the versions number shall be inherited from commons-parent 4. we upgrade versions in commons-parent every several months. Bernd Eckenfels 于2020年7月24日周五 下午9:49写道： > Hello, > > When it comes to dependencies wie have both problems: if we upgrade > dependencies to aggressively (or if we don't test with older dependencies) > then users have the problem that they might not easily be able to upgrade > to a new commons version since the required new dependency version might > conflict with other (thirdparty) users of that dependency. > > On the other hand if we not continuously update external dependencies we > might miss out on their new features, performance and fixes. In addition we > might fall behind and have the to do painful Big Bang Upgrades. Also when > our transitive dependencies are outdated and contain bugs (or compliance > violations due to old code) some customers might not be happy. > > So there is a middle ground to be found, which unfortunately collides with > the current limited effort maintenance of some of the components: > > - we should define a minimum baseline version of dependencies and runtimes > and on each release we check if we still meet them. When we raise the > baseline we should ship a new minor (or even major) version. Also we might > want to ship security fixes only as a micro update (I.e. not requiring > major updates besides the affected code) > - we should regularly test against latest dependency versions (at least > within the same minor branch). Dependbot could be a help here, however I do > find it a bit annoying and not needed as I would run the dependency report > if I have the need to find me something to work on. > > Gruss > Bend > > > -- > http://bernd.eckenfels.net > > Von: Stefan Bodewig > Gesendet: Friday, July 24, 2020 3:36:36 PM > An: Commons Developers List > Betreff: Re: [all] When to update dependencies? > > On 2020-07-24, Gary Gregory wrote: > > > Now back to our code depending on other dependencies. My view is that > there > > are bugs, you just have not hit them. If I find one in a dependency and > it > > gets fixed, it's going to mean a new version of that dependency. > > This either is "we hit a bug that affects us" - where I already say this > is a reason to upgrade for me - or "the user hits a bug" - in which case > I prefer to let the user update the buggy dependency. I don't want to > force the user to update because they might be affected by bugs. > > And of course we both know that newer versions not only fix bugs, they > introduce new ones as well. > > > Furthermore, when I look online for Javadoc and examples, if I use a > > current jar version, then my odds are better that I can implement what I > > find online. > > https://javadoc.io/ or similar services? > > > I view it as simpler and safer to update from a "near" dependency than > > letting a dependency acquire "bit rot" and upgrade later, especially > > if an update means making adjustments. I want to make smaller > > increments of adjustment rather than a larger set. Just like I prefer > > to RERO instead of big bangs for releasing Commons components. > > I fully agree with you if nobody else depends on my stuff. I.e. I'm > working on an application rather than a library. When working on a > library I really want the user to be and stay in control. > > > Another way to look at this is to look at a large software stack: If > every > > library developer never updates dependencies, then your application, > > through transitive dependencies will end up depending (virtually) on many > > versions of the each library, which is much more likely to create jar > hell > > and other problems than the other edge case where everyone uses the > latest > > version (or a fixed version.) > > You are just hitting home on Stefan's almost 20 year old impression that > automatic transitive dependency resolution is a BAD IDEA. ;-) > > I fully expect you (all of you) to ignore that last remark. > > > From a hand-waving-talking-over-beers-more-FOSS-y philosophical-POV, I'd > > like to think that by eating our own current Apache dog food as well as > > other FOSS dog food, we all help each other make our software better. > > We might help the developers of our dependencies but I'm afraid we are > hurting our dependees - and I tend to care more for the latter. > > Stefan > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > >
Re: [all] When to update dependencies?
Hello, When it comes to dependencies wie have both problems: if we upgrade dependencies to aggressively (or if we don't test with older dependencies) then users have the problem that they might not easily be able to upgrade to a new commons version since the required new dependency version might conflict with other (thirdparty) users of that dependency. On the other hand if we not continuously update external dependencies we might miss out on their new features, performance and fixes. In addition we might fall behind and have the to do painful Big Bang Upgrades. Also when our transitive dependencies are outdated and contain bugs (or compliance violations due to old code) some customers might not be happy. So there is a middle ground to be found, which unfortunately collides with the current limited effort maintenance of some of the components: - we should define a minimum baseline version of dependencies and runtimes and on each release we check if we still meet them. When we raise the baseline we should ship a new minor (or even major) version. Also we might want to ship security fixes only as a micro update (I.e. not requiring major updates besides the affected code) - we should regularly test against latest dependency versions (at least within the same minor branch). Dependbot could be a help here, however I do find it a bit annoying and not needed as I would run the dependency report if I have the need to find me something to work on. Gruss Bend -- http://bernd.eckenfels.net Von: Stefan Bodewig Gesendet: Friday, July 24, 2020 3:36:36 PM An: Commons Developers List Betreff: Re: [all] When to update dependencies? On 2020-07-24, Gary Gregory wrote: > Now back to our code depending on other dependencies. My view is that there > are bugs, you just have not hit them. If I find one in a dependency and it > gets fixed, it's going to mean a new version of that dependency. This either is "we hit a bug that affects us" - where I already say this is a reason to upgrade for me - or "the user hits a bug" - in which case I prefer to let the user update the buggy dependency. I don't want to force the user to update because they might be affected by bugs. And of course we both know that newer versions not only fix bugs, they introduce new ones as well. > Furthermore, when I look online for Javadoc and examples, if I use a > current jar version, then my odds are better that I can implement what I > find online. https://javadoc.io/ or similar services? > I view it as simpler and safer to update from a "near" dependency than > letting a dependency acquire "bit rot" and upgrade later, especially > if an update means making adjustments. I want to make smaller > increments of adjustment rather than a larger set. Just like I prefer > to RERO instead of big bangs for releasing Commons components. I fully agree with you if nobody else depends on my stuff. I.e. I'm working on an application rather than a library. When working on a library I really want the user to be and stay in control. > Another way to look at this is to look at a large software stack: If every > library developer never updates dependencies, then your application, > through transitive dependencies will end up depending (virtually) on many > versions of the each library, which is much more likely to create jar hell > and other problems than the other edge case where everyone uses the latest > version (or a fixed version.) You are just hitting home on Stefan's almost 20 year old impression that automatic transitive dependency resolution is a BAD IDEA. ;-) I fully expect you (all of you) to ignore that last remark. > From a hand-waving-talking-over-beers-more-FOSS-y philosophical-POV, I'd > like to think that by eating our own current Apache dog food as well as > other FOSS dog food, we all help each other make our software better. We might help the developers of our dependencies but I'm afraid we are hurting our dependees - and I tend to care more for the latter. Stefan - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [all] When to update dependencies?
> > You can imagine all manner of jar-hell created by shading. For instance: > - library L1 shades library ShadedA-1.0 and ShadedB-1.1. > - library L2 shades library ShadedA-1.1 and ShadedB-1.0. > - An app wants to use L1, L2, ShadedA-1.1, and ShadedB-1.1 but it can't no > matter what classpath ordering it uses. > - An app wants to use L1, L2, ShadedA-1.0, and ShadedB-1.0 but it can't no > matter what classpath ordering it uses. > Sorry, but I cannot follow the problems you are trying to show. L1 has ShadeA and ShadeB L2 has ShadeA and ShadeB but both have their own version that doesn't clash and that doesn't know anything about the other one. The app does not see ShadeA or ShadeB from L1 or L2 (unless it uses the hidden package from L1 which would be stupid) There are no clashes and every library uses the version that it needs. To be more explicit. I am maintaining org.vafer.jdependency which uses org.objectweb.asm. If you look at the final jar you find org/vafer/jdeb/shaded/objectweb/asm/*.class The shaded classes are relocated and become part of the context of the library that is shading. A shading hell is just not possible as long as there are no classpath problems on the library itself. It's like you are copy the code into your package. Bytecode manipulation is really not that bad as you make it to be. Just creating uberjars (without relocation) - that's a whole different story. That should only ever been done on the final application artifact.
Re: [all] When to update dependencies?
On 2020-07-24, Gary Gregory wrote: > Now back to our code depending on other dependencies. My view is that there > are bugs, you just have not hit them. If I find one in a dependency and it > gets fixed, it's going to mean a new version of that dependency. This either is "we hit a bug that affects us" - where I already say this is a reason to upgrade for me - or "the user hits a bug" - in which case I prefer to let the user update the buggy dependency. I don't want to force the user to update because they might be affected by bugs. And of course we both know that newer versions not only fix bugs, they introduce new ones as well. > Furthermore, when I look online for Javadoc and examples, if I use a > current jar version, then my odds are better that I can implement what I > find online. https://javadoc.io/ or similar services? > I view it as simpler and safer to update from a "near" dependency than > letting a dependency acquire "bit rot" and upgrade later, especially > if an update means making adjustments. I want to make smaller > increments of adjustment rather than a larger set. Just like I prefer > to RERO instead of big bangs for releasing Commons components. I fully agree with you if nobody else depends on my stuff. I.e. I'm working on an application rather than a library. When working on a library I really want the user to be and stay in control. > Another way to look at this is to look at a large software stack: If every > library developer never updates dependencies, then your application, > through transitive dependencies will end up depending (virtually) on many > versions of the each library, which is much more likely to create jar hell > and other problems than the other edge case where everyone uses the latest > version (or a fixed version.) You are just hitting home on Stefan's almost 20 year old impression that automatic transitive dependency resolution is a BAD IDEA. ;-) I fully expect you (all of you) to ignore that last remark. > From a hand-waving-talking-over-beers-more-FOSS-y philosophical-POV, I'd > like to think that by eating our own current Apache dog food as well as > other FOSS dog food, we all help each other make our software better. We might help the developers of our dependencies but I'm afraid we are hurting our dependees - and I tend to care more for the latter. Stefan - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [all] When to update dependencies?
Hi All: I'd like to start here by flipping the story around by looking at what we do. Ideally, a PR comes in for a bug with its fix and unit test. This is the best case scenario. If the PR is green, we squash and merge, and tell folks, yes, it will be in the next release. At times, a message comes in that says something like "In version 1.1, I see this behavior, it's a bug!" To which we say, "Ah, yes, this was fixed in 1.5, please upgrade." Alternatively, we say "Ah, yes, that's a bug, may you provide a PR? It will be in the next release". Other times, we say "May you please try the latest version, see what happens, and tell us?" Then pick one of the above. Sometime I say: "I'm not going to fix this on top of 1.1 to create a 1.1.1, I'll fix it in the current code base and it will be in the next version." A CVE might cause an older version to be branched as in the 1.1.1 case above but I've not seen that done here. Now back to our code depending on other dependencies. My view is that there are bugs, you just have not hit them. If I find one in a dependency and it gets fixed, it's going to mean a new version of that dependency. Furthermore, when I look online for Javadoc and examples, if I use a current jar version, then my odds are better that I can implement what I find online. I view it as simpler and safer to update from a "near" dependency than letting a dependency acquire "bit rot" and upgrade later, especially if an update means making adjustments. I want to make smaller increments of adjustment rather than a larger set. Just like I prefer to RERO instead of big bangs for releasing Commons components. Another way to look at this is to look at a large software stack: If every library developer never updates dependencies, then your application, through transitive dependencies will end up depending (virtually) on many versions of the each library, which is much more likely to create jar hell and other problems than the other edge case where everyone uses the latest version (or a fixed version.) >From a hand-waving-talking-over-beers-more-FOSS-y philosophical-POV, I'd like to think that by eating our own current Apache dog food as well as other FOSS dog food, we all help each other make our software better. Gary On Fri, Jul 24, 2020 at 4:04 AM Stefan Bodewig wrote: > Hi all > > here I'd like to explain why I prefer not to update dependencies just > because we can. Maybe you can convince me that I'm wrong. I've tried to > make this point in different threads but either it has been lost or it > just wasn't worth discussing. > > First of all let me get a few things out of the way > > * I'm not talking about emails, I can deal with them > > * I don't care whether a bot or a human asks for a version update > > * I'm only talking about dependencies that are visible to our > users. Test time dependencies or versions of Maven plugins are > probably not an issue. Although Compress has mananaged to break its > OSGi bundle just by upgrading the parent POM in the past. > https://issues.apache.org/jira/browse/COMPRESS-498 > > All our components have downstream users. I.e. our dependencies become > somebody else's dependencies as well. > > Let's say commons-foo 1.1.0 depends on A 1.12.4 and bumps the dependency > to A 1.12.18 for commons-foo 1.2.0. > > When a user of commons-foo upgrades to 1.2.0 and hasn't defined their > dependency on A explicitly they will also upgrade A to 1.12.18. This may > be fine or it may cause problems. The new version of A may have made > incompatible changes that break the user's code or it may just have bugs > that were not present in A 1.12.4 and now raise their head. > > Of course the users can explicitly state a dependency on A 1.12.4 > themselves. But there is no guarantee commons-foo compiled against A > 1.12.18 will still work with A 1.12.4. > > About fifteen years ago Ant was bitten by StringBuffer adding a new > method append(StringBuffer) in Java 1.4 (if memory serves me > right). Code that called someStringBuffer.append(anotherStringBuffer) > compiled on Java 1.3 would call append(Object), but compiled on 1.4 it > would call the new version and thus could not run on Java 1.3. This is > the kind of change animal sniffer was invented to detect and the > --release option of javac deals with. There is no such tool helping us > with APIs that are not part of the Java classlib. > > Therefore I believe updating a dependency is a risk and we should leave > it to the users to decide which version they want to use. > > Unless we've got real reasons to update. Real reasons IMHO are security > issues, bugs in dependencies causing bugs in our code or when we really > want to use new features introduced in a new version. > > Outside of these good reasons I wouldn't want to ever update a > dependency. > > Stefan > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands,
Re: [all] When to update dependencies?
On Fri, Jul 24, 2020 at 6:03 AM Gilles Sadowski wrote: > 2020-07-24 11:25 UTC+02:00, Torsten Curdt : > > It still needs a person to decide to merge a PR for a new version. > > So this indeed is just about the dependency upgrade policies. > > > > But isn't that what the version definition is for? > > Ideally. > In practice, I think that all we can assume is that the version > string is a quick-glance summary of changes, for human > consumption. > > > I'd argue that 1.12.4 <-> 1.12.6 should be a compatible upgrade AND > > downgrade, > > 1.12.4 -> 1.20.0 not so much. > > > > But to avoid all this is why I usually try to inline dependencies for > > libraries as much as possible. Basically pretending to not have any. > > This of course depends on whether the dependency can be isolated that > way. > > > > Also a point I made many times. > > Just wanted to mention it - again :) > > I think that it would be great that "Commons" has a common > policy for dealing with this (so that we don't have to repeat > ourselves every now and then). > A long time ago, the "shade" feature seemed the perfect > answer to that problem. Yet, to avoid dependencies even > on another Commons component, several of us continue to > support the copy/paste option. > There are a few items I'd like to address in this thread so I'll start with the above. I would like to offer my view on shading and copy/paste as - to borrow Jeff Atwood's expression - a "coding horror". While you may think that there are still edge cases where this may seem to some as necessary or expedient, I hope to dissuade you from these practices. The first and most obvious horror to me is the copying and pasting of bugs and hard to maintain code. You may also end up taking along supporting methods and classes to support the code you copy. While this may seem to follow the YAGNI principle, my claim is that you can end up with an iceberg of new code, where the new API surface above the water does not end up exercising all of the supporting code you had to drag along that lies below. The question where I am most curious to hear back from the copy clan is: Do you also copy the unit tests that cover the code you copied? No? Coding horror #2. The third horror does not affect us within Apache but its general practice begs its adherents to answer: Did you check the license of the code you copied? No? Coding horror #3. Again, this is not an issue within Apache projects, but the question pops up time and time again at Apache (and at work): Can I copy this code I found outside of Apache? The fourth and final copy horror falls in the same category as the first and deserves its own mention: security; and can be best understood with this question: How often do you check for CVEs for the library containing the code you copied? Never? Coding horror #4. Shading has its limited use in my view, and I've used it to provide monolithic versions of applications, but beyond that, I've no use for it; but the pain it can cause is very real indeed. You can imagine all manner of jar-hell created by shading. For instance: - library L1 shades library ShadedA-1.0 and ShadedB-1.1. - library L2 shades library ShadedA-1.1 and ShadedB-1.0. - An app wants to use L1, L2, ShadedA-1.1, and ShadedB-1.1 but it can't no matter what classpath ordering it uses. - An app wants to use L1, L2, ShadedA-1.0, and ShadedB-1.0 but it can't no matter what classpath ordering it uses. Lastly, there is a special kind of shading jar hell that deserves its own section: the copied and re-packaged source/jar. In this lovely planned development managed by Hades, an entire jar has been copied then repackaged such that instead of being in its original Java package org.foo.library, it is now under org.other.bar.org.foo.library. Residents of this charming cottage have now the proud owners of every bug and CVE for that specific version of the source jar, without, and this is the kicker, the option of overriding the jar with a new version (I'll leave aside your option to take a new version of the jar and repackaging it yourself to put it in front of the shaded jar.) I've seen this in various libraries that shade bytecode manipulation libraries that you can't use on new Java versions. I hope this helps someone... Gary > Gilles > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > >
Re: [all] When to update dependencies?
On 2020-07-24, Torsten Curdt wrote: > It still needs a person to decide to merge a PR for a new version. > So this indeed is just about the dependency upgrade policies. Right. > But isn't that what the version definition is for? > I'd argue that 1.12.4 <-> 1.12.6 should be a compatible upgrade AND > downgrade, > 1.12.4 -> 1.20.0 not so much. As Gary pointed out else-thread most of the time we do not know how strict the team developing our dependency adheres to SemVer. Even if it was completely API compatible, we'd replace a version that worked for our users with a different version that may introduce problems. No matter how small the risk is, what is the benefit of upgrading if we don't need the new version ourselves? > But to avoid all this is why I usually try to inline dependencies for > libraries as much as possible. Basically pretending to not have any. Agreed, this is a different strategy that makes the whole question moot. > Also a point I made many times. > Just wanted to mention it - again :) ;-) Stefan - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [all] When to update dependencies?
2020-07-24 11:25 UTC+02:00, Torsten Curdt : > It still needs a person to decide to merge a PR for a new version. > So this indeed is just about the dependency upgrade policies. > > But isn't that what the version definition is for? Ideally. In practice, I think that all we can assume is that the version string is a quick-glance summary of changes, for human consumption. > I'd argue that 1.12.4 <-> 1.12.6 should be a compatible upgrade AND > downgrade, > 1.12.4 -> 1.20.0 not so much. > > But to avoid all this is why I usually try to inline dependencies for > libraries as much as possible. Basically pretending to not have any. > This of course depends on whether the dependency can be isolated that way. > > Also a point I made many times. > Just wanted to mention it - again :) I think that it would be great that "Commons" has a common policy for dealing with this (so that we don't have to repeat ourselves every now and then). A long time ago, the "shade" feature seemed the perfect answer to that problem. Yet, to avoid dependencies even on another Commons component, several of us continue to support the copy/paste option. Gilles - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [all] When to update dependencies?
It still needs a person to decide to merge a PR for a new version. So this indeed is just about the dependency upgrade policies. But isn't that what the version definition is for? I'd argue that 1.12.4 <-> 1.12.6 should be a compatible upgrade AND downgrade, 1.12.4 -> 1.20.0 not so much. But to avoid all this is why I usually try to inline dependencies for libraries as much as possible. Basically pretending to not have any. This of course depends on whether the dependency can be isolated that way. Also a point I made many times. Just wanted to mention it - again :)
