Dear Community, This is in relation to malhar cleanup work that is ongoing.
In one of the talks during Apache BigData Europe, I got to know about Spark-Packages (https://spark-packages.org/) (I believe lot of you must be aware of it). Spark package is basically functionality over and above and using Spark core functionality. The spark packages can initially present in someone's public repository and one could register that with https://spark-packages.org/ and later on as it matures and finds more use, it gets consumed in mainstream Spark repository and releases. I found this idea quite interesting to keep our apex-malhar releases cleaner. One could have extension to apex-malhar in their own repository and just register itself with Apache Apex. As it matures and find more and more use we can consume that in mainstream releases. Advantages to this are multiple: 1. The entry point for registering extensions with Apache Apex can be minimal. This way we get more indirect contributions. 2. Faster way to add more feature in the project. 3. We keep our releases cleaner. 4. One could progress on feature-set faster balancing both Apache Way as well as their own Enterprise Interests. Please share your thoughts on this. Thanks, Chinmay.
