yes in sbt assembly you can exclude jars (although i never had a need for this) and files in jars.
for example i frequently remove log4j.properties, because for whatever reason hadoop decided to include it making it very difficult to use our own logging config. On Tue, Feb 25, 2014 at 4:24 PM, Konstantin Boudnik <c...@apache.org> wrote: > On Fri, Feb 21, 2014 at 11:11AM, Patrick Wendell wrote: > > Kos - thanks for chiming in. Could you be more specific about what is > > available in maven and not in sbt for these issues? I took a look at > > the bigtop code relating to Spark. As far as I could tell [1] was the > > main point of integration with the build system (maybe there are other > > integration points)? > > > > > - in order to integrate Spark well into existing Hadoop stack it was > > > necessary to have a way to avoid transitive dependencies > duplications and > > > possible conflicts. > > > > > > E.g. Maven assembly allows us to avoid adding _all_ Hadoop libs > and later > > > merely declare Spark package dependency on standard Bigtop Hadoop > > > packages. And yes - Bigtop packaging means the naming and layout > would be > > > standard across all commercial Hadoop distributions that are worth > > > mentioning: ASF Bigtop convenience binary packages, and Cloudera or > > > Hortonworks packages. Hence, the downstream user doesn't need to > spend any > > > effort to make sure that Spark "clicks-in" properly. > > > > The sbt build also allows you to plug in a Hadoop version similar to > > the maven build. > > I am actually talking about an ability to exclude a set of dependencies > from an > assembly, similarly to what's happening in dependencySet sections of > assembly/src/main/assembly/assembly.xml > If there is a comparable functionality in Sbt, that would help quite a bit, > apparently. > > Cos > > > > - Maven provides a relatively easy way to deal with the jar-hell > problem, > > > although the original maven build was just Shader'ing everything > into a > > > huge lump of class files. Oftentimes ending up with classes > slamming on > > > top of each other from different transitive dependencies. > > > > AFIAK we are only using the shade plug-in to deal with conflict > > resolution in the assembly jar. These are dealt with in sbt via the > > sbt assembly plug-in in an identical way. Is there a difference? > > I am bringing up the Sharder, because it is an awful hack, which is can't > be > used in real controlled deployment. > > Cos > > > [1] > https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=blob;f=bigtop-packages/src/common/spark/do-component-build;h=428540e0f6aa56cd7e78eb1c831aa7fe9496a08f;hb=master >
