Shade and relocate the external modules sounds ok as a short term solution.
For the long term we should consider something like the second option to add external modules without shipping uber jars. Thanks, Arun On 7/22/16, 6:07 AM, "Jungtaek Lim" <[email protected]> wrote: >Hi devs, > >AFAIK, we had been struggled to resolve dependency issues for storm-core. >As we all know, the strategy we have been using is shade & relocating. > >Now State and Storm SQL requires that some of external modules need to be >included to extlib, which is the classpath workers refer. > >http://issues.apache.org/jira/browse/STORM-1881 >https://issues.apache.org/jira/browse/STORM-1435 > >There're two issues here: >- We don't make uber jar for external modules so users need to find and >copy dependencies jars to extlib manually. >- External modules also use Guava and Jackson and so on which are origin of >version conflict issues. > >So we should apply the shade & relocating strategy for every external >modules (at least storm-redis, storm-kafka, storm-sql-core, >storm-sql-kafka), or introduce the way to add the dependency without adding >them to extlib. (like --packages and --jar for Spark) > >Please express your opinions about this. > >Thanks, >Jungtaek Lim (HeartSaVioR)
