Tushar, I agree that all modules should inherit the version from the "parent pom" of the malhar repo. I think the benefits outweigh the cost of bumping versions of components that haven't actually changed. I'd love to get others feedback on this as well.
On another note, I plan on starting a spreadsheet/googledoc with the possible groupings of operators into these modules. Stay tuned... -Andy On Mon, Sep 21, 2015 at 11:51 PM, Tushar Gosavi <[email protected]> wrote: > +1 for the general idea > > Does these independent modules going to have independent versions? For > example, if there is no change in kafka operator between malhar 3.0 and > malhar 4.0, will we increment version of malhar-contrib-kafka to 4.0. I > have learned from my previous project that, It is easier to manage versions > if we make all modules at same version level for a release, even if there > is no change in a particular module. > > - Tushar. > > > > On Fri, Sep 18, 2015 at 12:18 AM, Timothy Farkas <[email protected]> > wrote: > > > I agree Andy's solution is better, but just for the sake of argument > > profiles can be inherited from a parent pom, so if the maven archetype > > defines a new project with a parent pom with the correct profiles > defined, > > then the desired profiles can be activated in the pom of the new project. > > It is no more complicated than adding additional dependencies to your > > project. > > > > On Thu, Sep 17, 2015 at 10:32 AM, Sandesh Hegde <[email protected] > > > > wrote: > > > > > Currently all the dependencies in Malhar-Contrib are marked as > optional. > > So > > > users have to already modify the existing POM to use it in their > project. > > > So restructuring should be fine. > > > > > > On Thu, Sep 17, 2015 at 11:29 AM Chetan Narsude < > [email protected]> > > > wrote: > > > > > > > The profiles are excellent when you are developing malhar-contrib. > > > Profiles > > > > do not work when you are using malhar-contrib. The problem Andy is > > > trying > > > > to solve is the later. If there is an elegant solution which I am > > missing > > > > using profiles, please correct me. > > > > > > > > The way Andy suggested is the way many successful projects do it. > Look > > at > > > > Netty as an example. > > > > > > > > +1 for that. > > > > > > > > > > > > -- > > > > Chetan > > > > > > > > > > > > > > > > On Thu, Sep 17, 2015 at 11:22 AM, Timothy Farkas < > [email protected]> > > > > wrote: > > > > > > > > > I think restructuring the project in that way would be the > > technically > > > > > correct thing to do, but if people are unwilling to accept the > change > > > in > > > > > project structure you could achieve something similar by using > maven > > > > > profiles. With profiles the project structure would remain as is. > > > > Profiles > > > > > could be added to the malhar pom, and a profile would define the > > > > > dependencies needed for different types of operators. For example > the > > > > hbase > > > > > profile would define the dependencies for the hbase operator. Then > > any > > > > > project using a malhar library would just activate the correct > > profile > > > in > > > > > it's pom, and the correct dependencies would be pulled in. > > > > > > > > > > > > > > > > > > > > http://maven.apache.org/guides/introduction/introduction-to-profiles.html > > > > > > > > > > On Thu, Sep 17, 2015 at 10:01 AM, Andy Perlitch < > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > I am currently assigned to MLHR-1843 > > > > > > <https://malhar.atlassian.net/browse/MLHR-1843>, which > essentially > > > > aims > > > > > to > > > > > > expose smaller, more consumable maven artifacts that would do > away > > > with > > > > > the > > > > > > need to manually include necessary dependencies based on the > > > operators > > > > in > > > > > > use. > > > > > > > > > > > > As an example, say I am building an app package that needs Kafka > > > input > > > > > and > > > > > > output operators, but I don't want all the other transitive > > > > dependencies > > > > > > that come via malhar-contrib. Currently I would need to specify > > > > > > malhar-contrib as a dependency, and add an exclusions block in > my > > > app > > > > > > package pom: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *<dependency> <groupId>com.datatorrent</groupId> > > > > > > <artifactId>malhar-contrib</artifactId> <version>3.0.0</version> > > > <!-- > > > > > so > > > > > > none of malhar-contrib's deps are included -->* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * <exclusions> <exclusion> <groupId>*</groupId> > > > > > > <artifactId>*</artifactId> </exclusion> > > > </exclusions></dependency>* > > > > > > > > > > > > Then, I would have to include the kafka library explicitly as a > > > > > dependency: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *<dependency> <groupId>org.apache.kafka</groupId> > > > > > > <artifactId>kafka_2.10</artifactId> > > > > > > <version>0.8.1.1</version></dependency>* > > > > > > > > > > > > Wouldn't it be nice if I could just put this in my pom?: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *<dependency> <groupId>com.datatorrent</groupId> > > > > > > <artifactId>malhar-contrib-kafka</artifactId> > > > > > > <version>3.0.0</version></dependency>* > > > > > > > > > > > > > > > > > > In order to make this possible, we will need to organize the > malhar > > > > > project > > > > > > into more granular modules (artifacts). Specifically, the > > > > malhar-contrib > > > > > > artifact would essentially just be a pom that specifies each > > smaller > > > > > module > > > > > > as a dependency: > > > > > > > > > > > > *<!-- in malhar-contrib's pom.xml: -->* > > > > > > > > > > > > *<modules> <module>kafka</module>* > > > > > > * <module>twitter</module>* > > > > > > * <module>redis</module>* > > > > > > > > > > > > * <!-- other smaller modules --></modules>* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *<dependency> <groupId>com.datatorrent</groupId> > > > > > > <artifactId>malhar-contrib-kafka</artifactId> > > > > > > <version>3.0.0</version></dependency>* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *<dependency> <groupId>com.datatorrent</groupId> > > > > > > <artifactId>malhar-contrib-twitter</artifactId> > > > > > > <version>3.0.0</version></dependency>* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *<dependency> <groupId>com.datatorrent</groupId> > > > > > > <artifactId>malhar-contrib-redis</artifactId> > > > > > > <version>3.0.0</version></dependency>* > > > > > > > > > > > > With these changes, there may be a risk of breaking backwards > > > > > > compatibility, however I think the gain in usability of malhar > > merits > > > > the > > > > > > effort to make this work. > > > > > > > > > > > > I am still relatively new to maven, so I would love to get some > > > > feedback > > > > > > from other devs about this! > > > > > > > > > > > > -- > > > > > > Regards, > > > > > > Andy Perlitch > > > > > > Software Engineer > > > > > > DataTorrent Inc > > > > > > (408)829-9319 > > > > > > > > > > > > > > > > > > > > > -- Regards, Andy Perlitch Software Engineer DataTorrent Inc (408)829-9319
