I've made a design document for adding jars and maven artifacts at submission. Since we don't have any formats for this, I just borrowed KIP format. That's not we would like to adopt that format, it's just me.
https://docs.google.com/document/d/1f3Ed0Wa8aN2j7gtptT0BOKYMyBdpohLgERg4YebJJ7c/edit?usp=sharing Btw, I guess ASF policy requires that discussion history should be logged to Apache Infra. (So I only grant view permission who opens the doc via link.) So what's recommended way to do? Would we want to file an issue for JIRA with attaching design doc to PDF and discuss there? - Jungtaek Lim 2016년 7월 23일 (토) 오전 12:06, Jungtaek Lim <[email protected]>님이 작성: > Thought about it once more, and found that former approach still needs to > add 'provided' scope libraries to extlib directory.. > > Along with thinking former approach, I've experimented a bit of latter > approach by creating POC project. Since I don't know about copy and use ASF > project codes for personal use, I'd just share main class to see if this > POC and my theory makes sense for us. > https://gist.github.com/HeartSaVioR/3639a9ee829fe1203b4a085a0fb069d6 > 'zeppelin' package has some classes for transitive dependency resolver > (copied from Zeppelin), and 'spark' package has some classes for > classloader (yes also copied from Spark). > > Please share your experiences if you have knowledges regarding this area. > > - Jungtaek Lim > > 2016년 7월 22일 (금) 오후 10:59, Bobby Evans <[email protected]>님이 작성: > >> You can do that with a combination of the distributed cache and setting >> the classpath, just like with hadoop. It is not as clean as it could be, >> but it does work. - Bobby >> >> On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan <[email protected]> >> wrote: >> >> >> Shade and relocate the external modules sounds ok as a short term >> solution. >> >> For the long term we should consider something like the second option to >> add external modules without shipping uber jars. >> >> Thanks, >> Arun >> >> On 7/22/16, 6:07 AM, "Jungtaek Lim" <[email protected]> wrote: >> >> >Hi devs, >> > >> >AFAIK, we had been struggled to resolve dependency issues for storm-core. >> >As we all know, the strategy we have been using is shade & relocating. >> > >> >Now State and Storm SQL requires that some of external modules need to be >> >included to extlib, which is the classpath workers refer. >> > >> >http://issues.apache.org/jira/browse/STORM-1881 >> >https://issues.apache.org/jira/browse/STORM-1435 >> > >> >There're two issues here: >> >- We don't make uber jar for external modules so users need to find and >> >copy dependencies jars to extlib manually. >> >- External modules also use Guava and Jackson and so on which are origin >> of >> >version conflict issues. >> > >> >So we should apply the shade & relocating strategy for every external >> >modules (at least storm-redis, storm-kafka, storm-sql-core, >> >storm-sql-kafka), or introduce the way to add the dependency without >> adding >> >them to extlib. (like --packages and --jar for Spark) >> > >> >Please express your opinions about this. >> > >> >Thanks, >> >Jungtaek Lim (HeartSaVioR) >> >> >> >> >> > >
