Re: Overlapping classes warnings
In general, I don't think that means you should exclude something; it's still needed. The problem is that commons config depends *only* on *beanutils-core 1.8.0* so it ends up managing up that artifact version only, and not the main beanutils one. In this particular instance, which I've seen before, it seems clear that beanutils-core *can* be excluded since it's a subset (see: the warning message!) and beanutils is an uber-artifact. But then we'd have to manage up beanutils to 1.8.0. On Thu, Apr 9, 2015 at 5:24 PM, Ted Yu yuzhih...@gmail.com wrote: commons-beanutils is brought in transitively: [INFO] | +- org.apache.hadoop:hadoop-common:jar:2.4.0:compile [INFO] | | +- commons-cli:commons-cli:jar:1.2:compile [INFO] | | +- xmlenc:xmlenc:jar:0.52:compile [INFO] | | +- commons-httpclient:commons-httpclient:jar:3.1:compile [INFO] | | +- commons-io:commons-io:jar:2.4:compile [INFO] | | +- commons-collections:commons-collections:jar:3.2.1:compile [INFO] | | +- commons-lang:commons-lang:jar:2.6:compile [INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile [INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile [INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile [INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile Looks like we can exclude commons-beanutils in pom.xml for hadoop-client dependency. e.g. in core/pom.xml : dependency groupIdorg.apache.hadoop/groupId artifactIdhadoop-client/artifactId exclusions exclusion groupIdjavax.servlet/groupId artifactIdservlet-api/artifactId /exclusion /exclusions /dependency On Thu, Apr 9, 2015 at 2:14 PM, Ritesh Kumar Singh riteshoneinamill...@gmail.com wrote: Though the warnings can be ignored, they add up in the log files while compiling other projects too. And there are a lot of those warnings. Any workaround? How do we modify the pom.xml file to exclude these unnecessary dependencies? On Fri, Apr 10, 2015 at 2:29 AM, Sean Owen so...@cloudera.com wrote: Generally, you can ignore these things. They mean some artifacts packaged other artifacts, and so two copies show up when all the JAR contents are merged. But here you do show a small dependency convergence problem; beanutils 1.7 is present but beanutills-core 1.8 is too even though these should be harmonized. I imagine one could be excluded; I imagine we could harmonize the version manually. In practice, I also imagine it doesn't cause any problem but feel free to propose a fix along those lines. On Thu, Apr 9, 2015 at 4:54 PM, Ritesh Kumar Singh riteshoneinamill...@gmail.com wrote: Hi, During compilation I get a lot of these: [WARNING] kryo-2.21.jar, reflectasm-1.07-shaded.jar define 23 overlappping classes: [WARNING] commons-beanutils-1.7.0.jar, commons-beanutils-core-1.8.0.jar define 82 overlappping classes: [WARNING] commons-beanutils-1.7.0.jar, commons-collections-3.2.1.jar, commons-beanutils-core-1.8.0.jar define 10 overlappping classes: And a lot of others. How do I fix these? - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Overlapping classes warnings
commons-beanutils is brought in transitively: [INFO] | +- org.apache.hadoop:hadoop-common:jar:2.4.0:compile [INFO] | | +- commons-cli:commons-cli:jar:1.2:compile [INFO] | | +- xmlenc:xmlenc:jar:0.52:compile [INFO] | | +- commons-httpclient:commons-httpclient:jar:3.1:compile [INFO] | | +- commons-io:commons-io:jar:2.4:compile [INFO] | | +- commons-collections:commons-collections:jar:3.2.1:compile [INFO] | | +- commons-lang:commons-lang:jar:2.6:compile [INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile [INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile [INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile [INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile Looks like we can exclude commons-beanutils in pom.xml for hadoop-client dependency. e.g. in core/pom.xml : dependency groupIdorg.apache.hadoop/groupId artifactIdhadoop-client/artifactId exclusions exclusion groupIdjavax.servlet/groupId artifactIdservlet-api/artifactId /exclusion /exclusions /dependency On Thu, Apr 9, 2015 at 2:14 PM, Ritesh Kumar Singh riteshoneinamill...@gmail.com wrote: Though the warnings can be ignored, they add up in the log files while compiling other projects too. And there are a lot of those warnings. Any workaround? How do we modify the pom.xml file to exclude these unnecessary dependencies? On Fri, Apr 10, 2015 at 2:29 AM, Sean Owen so...@cloudera.com wrote: Generally, you can ignore these things. They mean some artifacts packaged other artifacts, and so two copies show up when all the JAR contents are merged. But here you do show a small dependency convergence problem; beanutils 1.7 is present but beanutills-core 1.8 is too even though these should be harmonized. I imagine one could be excluded; I imagine we could harmonize the version manually. In practice, I also imagine it doesn't cause any problem but feel free to propose a fix along those lines. On Thu, Apr 9, 2015 at 4:54 PM, Ritesh Kumar Singh riteshoneinamill...@gmail.com wrote: Hi, During compilation I get a lot of these: [WARNING] kryo-2.21.jar, reflectasm-1.07-shaded.jar define 23 overlappping classes: [WARNING] commons-beanutils-1.7.0.jar, commons-beanutils-core-1.8.0.jar define 82 overlappping classes: [WARNING] commons-beanutils-1.7.0.jar, commons-collections-3.2.1.jar, commons-beanutils-core-1.8.0.jar define 10 overlappping classes: And a lot of others. How do I fix these?
Re: Overlapping classes warnings
Generally, you can ignore these things. They mean some artifacts packaged other artifacts, and so two copies show up when all the JAR contents are merged. But here you do show a small dependency convergence problem; beanutils 1.7 is present but beanutills-core 1.8 is too even though these should be harmonized. I imagine one could be excluded; I imagine we could harmonize the version manually. In practice, I also imagine it doesn't cause any problem but feel free to propose a fix along those lines. On Thu, Apr 9, 2015 at 4:54 PM, Ritesh Kumar Singh riteshoneinamill...@gmail.com wrote: Hi, During compilation I get a lot of these: [WARNING] kryo-2.21.jar, reflectasm-1.07-shaded.jar define 23 overlappping classes: [WARNING] commons-beanutils-1.7.0.jar, commons-beanutils-core-1.8.0.jar define 82 overlappping classes: [WARNING] commons-beanutils-1.7.0.jar, commons-collections-3.2.1.jar, commons-beanutils-core-1.8.0.jar define 10 overlappping classes: And a lot of others. How do I fix these? - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Overlapping classes warnings
Though the warnings can be ignored, they add up in the log files while compiling other projects too. And there are a lot of those warnings. Any workaround? How do we modify the pom.xml file to exclude these unnecessary dependencies? On Fri, Apr 10, 2015 at 2:29 AM, Sean Owen so...@cloudera.com wrote: Generally, you can ignore these things. They mean some artifacts packaged other artifacts, and so two copies show up when all the JAR contents are merged. But here you do show a small dependency convergence problem; beanutils 1.7 is present but beanutills-core 1.8 is too even though these should be harmonized. I imagine one could be excluded; I imagine we could harmonize the version manually. In practice, I also imagine it doesn't cause any problem but feel free to propose a fix along those lines. On Thu, Apr 9, 2015 at 4:54 PM, Ritesh Kumar Singh riteshoneinamill...@gmail.com wrote: Hi, During compilation I get a lot of these: [WARNING] kryo-2.21.jar, reflectasm-1.07-shaded.jar define 23 overlappping classes: [WARNING] commons-beanutils-1.7.0.jar, commons-beanutils-core-1.8.0.jar define 82 overlappping classes: [WARNING] commons-beanutils-1.7.0.jar, commons-collections-3.2.1.jar, commons-beanutils-core-1.8.0.jar define 10 overlappping classes: And a lot of others. How do I fix these?
Re: Overlapping classes warnings
I agree, but as I say, most are out of the control of Spark. They aren't because of unnecessary dependencies. On Thu, Apr 9, 2015 at 5:14 PM, Ritesh Kumar Singh riteshoneinamill...@gmail.com wrote: Though the warnings can be ignored, they add up in the log files while compiling other projects too. And there are a lot of those warnings. Any workaround? How do we modify the pom.xml file to exclude these unnecessary dependencies? On Fri, Apr 10, 2015 at 2:29 AM, Sean Owen so...@cloudera.com wrote: Generally, you can ignore these things. They mean some artifacts packaged other artifacts, and so two copies show up when all the JAR contents are merged. But here you do show a small dependency convergence problem; beanutils 1.7 is present but beanutills-core 1.8 is too even though these should be harmonized. I imagine one could be excluded; I imagine we could harmonize the version manually. In practice, I also imagine it doesn't cause any problem but feel free to propose a fix along those lines. On Thu, Apr 9, 2015 at 4:54 PM, Ritesh Kumar Singh riteshoneinamill...@gmail.com wrote: Hi, During compilation I get a lot of these: [WARNING] kryo-2.21.jar, reflectasm-1.07-shaded.jar define 23 overlappping classes: [WARNING] commons-beanutils-1.7.0.jar, commons-beanutils-core-1.8.0.jar define 82 overlappping classes: [WARNING] commons-beanutils-1.7.0.jar, commons-collections-3.2.1.jar, commons-beanutils-core-1.8.0.jar define 10 overlappping classes: And a lot of others. How do I fix these? - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Overlapping classes warnings
I found this jira https://jira.codehaus.org/browse/MSHADE-128 when googling for fixes. Wonder if it can fix anything here. But anyways, thanks for the help :) On Fri, Apr 10, 2015 at 2:46 AM, Sean Owen so...@cloudera.com wrote: I agree, but as I say, most are out of the control of Spark. They aren't because of unnecessary dependencies. On Thu, Apr 9, 2015 at 5:14 PM, Ritesh Kumar Singh riteshoneinamill...@gmail.com wrote: Though the warnings can be ignored, they add up in the log files while compiling other projects too. And there are a lot of those warnings. Any workaround? How do we modify the pom.xml file to exclude these unnecessary dependencies? On Fri, Apr 10, 2015 at 2:29 AM, Sean Owen so...@cloudera.com wrote: Generally, you can ignore these things. They mean some artifacts packaged other artifacts, and so two copies show up when all the JAR contents are merged. But here you do show a small dependency convergence problem; beanutils 1.7 is present but beanutills-core 1.8 is too even though these should be harmonized. I imagine one could be excluded; I imagine we could harmonize the version manually. In practice, I also imagine it doesn't cause any problem but feel free to propose a fix along those lines. On Thu, Apr 9, 2015 at 4:54 PM, Ritesh Kumar Singh riteshoneinamill...@gmail.com wrote: Hi, During compilation I get a lot of these: [WARNING] kryo-2.21.jar, reflectasm-1.07-shaded.jar define 23 overlappping classes: [WARNING] commons-beanutils-1.7.0.jar, commons-beanutils-core-1.8.0.jar define 82 overlappping classes: [WARNING] commons-beanutils-1.7.0.jar, commons-collections-3.2.1.jar, commons-beanutils-core-1.8.0.jar define 10 overlappping classes: And a lot of others. How do I fix these?
