Where is number 4 needed? The current contrib package structure could remain even though we move them into modules right?
On Tue, Sep 29, 2015 at 12:19 AM, Andy Perlitch <[email protected]> wrote: > Hi all, > > This is a first cut at a plan to restructure malhar in a way that is more > portable and adherent to Maven's principles of modularity and dependency > management. > > Overview of Current Malhar Architecture > --------------------------------------------------------------- > The current malhar repo consists of several maven modules: > > * *malhar-library* > operators which do not require additional transitive dependencies beyond > what Apex and Hadoop require > * *malhar-contrib* > operators requiring other maven dependencies > * *malhar-demos* > demo applications > * *malhar-samples* > sample code showing example usage of malhar operators > * *malhar-apps* > apex applications (currently only logstream) > > > Proposed Changes > --------------------------------------------------------------- > > 1. *Scrub malhar-library for any operators needing additional dependencies* > `malhar-library` is intended to consist of only operators without extra > transitive dependencies. All operators should be checked for the necessity > of extra dependencies. > > 2. *Move operators from malhar-demos and malhar-apps into contrib (or > library if prudent)* > There are various operators in both of these modules that are general > enough to move into library or contrib. > > 3. *Create modules for all contrib subfolders* > All folders under `contrib/src/main/com/datatorrent/contrib/` should be > converted to modules of contrib and listed as such in `/contrib/pom.xml`. > Additionally, each of these smaller contrib modules will have its own > version and dependencies. > > 4. *Use the Shades Plugin to allow for backwards-compatible fully-qualified > class names* > This is made possible by shades class relocation > < > https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html > > > feature. This might be a bit error prone as well as confusing to use for > outside developers, but it must be done if these changes are to be made > prior to a major release. > > > > Let me know what you all think of this approach. > > Best, > Andy > > > On Tue, Sep 22, 2015 at 11:20 AM, Chetan Narsude <[email protected]> > wrote: > > > +1 > > > > On Tue, Sep 22, 2015 at 11:08 AM, Gaurav Gupta <[email protected]> > > wrote: > > > > > I agree with David.. Each artifact should have it's own version > > > > > > Thanks > > > -Gaurav > > > > > > On Tue, Sep 22, 2015 at 11:07 AM, David Yan <[email protected]> > > wrote: > > > > > > > I actually think that each baby artifact should have its own version, > > > > because each artifact has its own interface and its own life cycle, > > > > especially after we break up the giant library, applications will > > depend > > > on > > > > the baby artifacts instead of the giant library. For example if > there > > is > > > > no change in malhar-contrib-kafka (I think the name should actually > be > > > > apex-malhar-kafka), we should not confuse users by bumping the > version. > > > > > > > > David > > > > > > > > On Tue, Sep 22, 2015 at 9:03 AM, Andy Perlitch <[email protected] > > > > > > wrote: > > > > > > > > > Tushar, > > > > > > > > > > I agree that all modules should inherit the version from the > "parent > > > pom" > > > > > of the malhar repo. I think the benefits outweigh the cost of > bumping > > > > > versions of components that haven't actually changed. I'd love to > get > > > > > others feedback on this as well. > > > > > > > > > > On another note, I plan on starting a spreadsheet/googledoc with > the > > > > > possible groupings of operators into these modules. Stay tuned... > > > > > > > > > > -Andy > > > > > > > > > > On Mon, Sep 21, 2015 at 11:51 PM, Tushar Gosavi < > > > [email protected]> > > > > > wrote: > > > > > > > > > > > +1 for the general idea > > > > > > > > > > > > Does these independent modules going to have independent > versions? > > > For > > > > > > example, if there is no change in kafka operator between malhar > 3.0 > > > and > > > > > > malhar 4.0, will we increment version of malhar-contrib-kafka to > > > 4.0. I > > > > > > have learned from my previous project that, It is easier to > manage > > > > > versions > > > > > > if we make all modules at same version level for a release, even > if > > > > there > > > > > > is no change in a particular module. > > > > > > > > > > > > - Tushar. > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 18, 2015 at 12:18 AM, Timothy Farkas < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > I agree Andy's solution is better, but just for the sake of > > > argument > > > > > > > profiles can be inherited from a parent pom, so if the maven > > > > archetype > > > > > > > defines a new project with a parent pom with the correct > profiles > > > > > > defined, > > > > > > > then the desired profiles can be activated in the pom of the > new > > > > > project. > > > > > > > It is no more complicated than adding additional dependencies > to > > > your > > > > > > > project. > > > > > > > > > > > > > > On Thu, Sep 17, 2015 at 10:32 AM, Sandesh Hegde < > > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Currently all the dependencies in Malhar-Contrib are marked > as > > > > > > optional. > > > > > > > So > > > > > > > > users have to already modify the existing POM to use it in > > their > > > > > > project. > > > > > > > > So restructuring should be fine. > > > > > > > > > > > > > > > > On Thu, Sep 17, 2015 at 11:29 AM Chetan Narsude < > > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > The profiles are excellent when you are developing > > > > malhar-contrib. > > > > > > > > Profiles > > > > > > > > > do not work when you are using malhar-contrib. The problem > > Andy > > > > is > > > > > > > > trying > > > > > > > > > to solve is the later. If there is an elegant solution > which > > I > > > am > > > > > > > missing > > > > > > > > > using profiles, please correct me. > > > > > > > > > > > > > > > > > > The way Andy suggested is the way many successful projects > do > > > it. > > > > > > Look > > > > > > > at > > > > > > > > > Netty as an example. > > > > > > > > > > > > > > > > > > +1 for that. > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Chetan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 17, 2015 at 11:22 AM, Timothy Farkas < > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I think restructuring the project in that way would be > the > > > > > > > technically > > > > > > > > > > correct thing to do, but if people are unwilling to > accept > > > the > > > > > > change > > > > > > > > in > > > > > > > > > > project structure you could achieve something similar by > > > using > > > > > > maven > > > > > > > > > > profiles. With profiles the project structure would > remain > > as > > > > is. > > > > > > > > > Profiles > > > > > > > > > > could be added to the malhar pom, and a profile would > > define > > > > the > > > > > > > > > > dependencies needed for different types of operators. For > > > > example > > > > > > the > > > > > > > > > hbase > > > > > > > > > > profile would define the dependencies for the hbase > > operator. > > > > > Then > > > > > > > any > > > > > > > > > > project using a malhar library would just activate the > > > correct > > > > > > > profile > > > > > > > > in > > > > > > > > > > it's pom, and the correct dependencies would be pulled > in. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://maven.apache.org/guides/introduction/introduction-to-profiles.html > > > > > > > > > > > > > > > > > > > > On Thu, Sep 17, 2015 at 10:01 AM, Andy Perlitch < > > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > I am currently assigned to MLHR-1843 > > > > > > > > > > > <https://malhar.atlassian.net/browse/MLHR-1843>, which > > > > > > essentially > > > > > > > > > aims > > > > > > > > > > to > > > > > > > > > > > expose smaller, more consumable maven artifacts that > > would > > > do > > > > > > away > > > > > > > > with > > > > > > > > > > the > > > > > > > > > > > need to manually include necessary dependencies based > on > > > the > > > > > > > > operators > > > > > > > > > in > > > > > > > > > > > use. > > > > > > > > > > > > > > > > > > > > > > As an example, say I am building an app package that > > needs > > > > > Kafka > > > > > > > > input > > > > > > > > > > and > > > > > > > > > > > output operators, but I don't want all the other > > transitive > > > > > > > > > dependencies > > > > > > > > > > > that come via malhar-contrib. Currently I would need to > > > > specify > > > > > > > > > > > malhar-contrib as a dependency, and add an exclusions > > block > > > > in > > > > > > my > > > > > > > > app > > > > > > > > > > > package pom: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *<dependency> <groupId>com.datatorrent</groupId> > > > > > > > > > > > <artifactId>malhar-contrib</artifactId> > > > > > <version>3.0.0</version> > > > > > > > > <!-- > > > > > > > > > > so > > > > > > > > > > > none of malhar-contrib's deps are included -->* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * <exclusions> <exclusion> > <groupId>*</groupId> > > > > > > > > > > > <artifactId>*</artifactId> </exclusion> > > > > > > > > </exclusions></dependency>* > > > > > > > > > > > > > > > > > > > > > > Then, I would have to include the kafka library > > explicitly > > > > as a > > > > > > > > > > dependency: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *<dependency> <groupId>org.apache.kafka</groupId> > > > > > > > > > > > <artifactId>kafka_2.10</artifactId> > > > > > > > > > > > <version>0.8.1.1</version></dependency>* > > > > > > > > > > > > > > > > > > > > > > Wouldn't it be nice if I could just put this in my > pom?: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *<dependency> <groupId>com.datatorrent</groupId> > > > > > > > > > > > <artifactId>malhar-contrib-kafka</artifactId> > > > > > > > > > > > <version>3.0.0</version></dependency>* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In order to make this possible, we will need to > organize > > > the > > > > > > malhar > > > > > > > > > > project > > > > > > > > > > > into more granular modules (artifacts). Specifically, > the > > > > > > > > > malhar-contrib > > > > > > > > > > > artifact would essentially just be a pom that specifies > > > each > > > > > > > smaller > > > > > > > > > > module > > > > > > > > > > > as a dependency: > > > > > > > > > > > > > > > > > > > > > > *<!-- in malhar-contrib's pom.xml: -->* > > > > > > > > > > > > > > > > > > > > > > *<modules> <module>kafka</module>* > > > > > > > > > > > * <module>twitter</module>* > > > > > > > > > > > * <module>redis</module>* > > > > > > > > > > > > > > > > > > > > > > * <!-- other smaller modules --></modules>* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *<dependency> <groupId>com.datatorrent</groupId> > > > > > > > > > > > <artifactId>malhar-contrib-kafka</artifactId> > > > > > > > > > > > <version>3.0.0</version></dependency>* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *<dependency> <groupId>com.datatorrent</groupId> > > > > > > > > > > > <artifactId>malhar-contrib-twitter</artifactId> > > > > > > > > > > > <version>3.0.0</version></dependency>* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *<dependency> <groupId>com.datatorrent</groupId> > > > > > > > > > > > <artifactId>malhar-contrib-redis</artifactId> > > > > > > > > > > > <version>3.0.0</version></dependency>* > > > > > > > > > > > > > > > > > > > > > > With these changes, there may be a risk of breaking > > > backwards > > > > > > > > > > > compatibility, however I think the gain in usability of > > > > malhar > > > > > > > merits > > > > > > > > > the > > > > > > > > > > > effort to make this work. > > > > > > > > > > > > > > > > > > > > > > I am still relatively new to maven, so I would love to > > get > > > > some > > > > > > > > > feedback > > > > > > > > > > > from other devs about this! > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Regards, > > > > > > > > > > > Andy Perlitch > > > > > > > > > > > Software Engineer > > > > > > > > > > > DataTorrent Inc > > > > > > > > > > > (408)829-9319 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > Andy Perlitch > > > > > Software Engineer > > > > > DataTorrent Inc > > > > > (408)829-9319 > > > > > > > > > > > > > > > > > > -- > Regards, > Andy Perlitch > Software Engineer > DataTorrent Inc > (408)829-9319 >
