Hi everyone, I am currently assigned to MLHR-1843 <https://malhar.atlassian.net/browse/MLHR-1843>, which essentially aims to expose smaller, more consumable maven artifacts that would do away with the need to manually include necessary dependencies based on the operators in use.
As an example, say I am building an app package that needs Kafka input and output operators, but I don't want all the other transitive dependencies that come via malhar-contrib. Currently I would need to specify malhar-contrib as a dependency, and add an exclusions block in my app package pom: *<dependency> <groupId>com.datatorrent</groupId> <artifactId>malhar-contrib</artifactId> <version>3.0.0</version> <!-- so none of malhar-contrib's deps are included -->* * <exclusions> <exclusion> <groupId>*</groupId> <artifactId>*</artifactId> </exclusion> </exclusions></dependency>* Then, I would have to include the kafka library explicitly as a dependency: *<dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.10</artifactId> <version>0.8.1.1</version></dependency>* Wouldn't it be nice if I could just put this in my pom?: *<dependency> <groupId>com.datatorrent</groupId> <artifactId>malhar-contrib-kafka</artifactId> <version>3.0.0</version></dependency>* In order to make this possible, we will need to organize the malhar project into more granular modules (artifacts). Specifically, the malhar-contrib artifact would essentially just be a pom that specifies each smaller module as a dependency: *<!-- in malhar-contrib's pom.xml: -->* *<modules> <module>kafka</module>* * <module>twitter</module>* * <module>redis</module>* * <!-- other smaller modules --></modules>* *<dependency> <groupId>com.datatorrent</groupId> <artifactId>malhar-contrib-kafka</artifactId> <version>3.0.0</version></dependency>* *<dependency> <groupId>com.datatorrent</groupId> <artifactId>malhar-contrib-twitter</artifactId> <version>3.0.0</version></dependency>* *<dependency> <groupId>com.datatorrent</groupId> <artifactId>malhar-contrib-redis</artifactId> <version>3.0.0</version></dependency>* With these changes, there may be a risk of breaking backwards compatibility, however I think the gain in usability of malhar merits the effort to make this work. I am still relatively new to maven, so I would love to get some feedback from other devs about this! -- Regards, Andy Perlitch Software Engineer DataTorrent Inc (408)829-9319
