Not yet. It would be good to continue discussion on this thread for now. Thanks, Jungtaek Lim (HeartSaVioR)
2016년 8월 1일 (월) 오후 1:38, Satish Duggana <[email protected]>님이 작성: > Hi, > Did we find a way to comment on the proposal? How can we proceed with this > discussion? > > Thanks, > Satish. > > > On Thu, Jul 28, 2016 at 9:42 AM, Jungtaek Lim <[email protected]> wrote: > > > I've made a design document for adding jars and maven artifacts at > > submission. > > Since we don't have any formats for this, I just borrowed KIP format. > > That's not we would like to adopt that format, it's just me. > > > > > > > https://docs.google.com/document/d/1f3Ed0Wa8aN2j7gtptT0BOKYMyBdpohLgERg4YebJJ7c/edit?usp=sharing > > > > Btw, I guess ASF policy requires that discussion history should be logged > > to Apache Infra. > > (So I only grant view permission who opens the doc via link.) > > > > So what's recommended way to do? Would we want to file an issue for JIRA > > with attaching design doc to PDF and discuss there? > > > > - Jungtaek Lim > > > > 2016년 7월 23일 (토) 오전 12:06, Jungtaek Lim <[email protected]>님이 작성: > > > > > Thought about it once more, and found that former approach still needs > to > > > add 'provided' scope libraries to extlib directory.. > > > > > > Along with thinking former approach, I've experimented a bit of latter > > > approach by creating POC project. Since I don't know about copy and use > > ASF > > > project codes for personal use, I'd just share main class to see if > this > > > POC and my theory makes sense for us. > > > https://gist.github.com/HeartSaVioR/3639a9ee829fe1203b4a085a0fb069d6 > > > 'zeppelin' package has some classes for transitive dependency resolver > > > (copied from Zeppelin), and 'spark' package has some classes for > > > classloader (yes also copied from Spark). > > > > > > Please share your experiences if you have knowledges regarding this > area. > > > > > > - Jungtaek Lim > > > > > > 2016년 7월 22일 (금) 오후 10:59, Bobby Evans <[email protected]>님이 > > 작성: > > > > > >> You can do that with a combination of the distributed cache and > setting > > >> the classpath, just like with hadoop. It is not as clean as it could > > be, > > >> but it does work. - Bobby > > >> > > >> On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan < > > [email protected]> > > >> wrote: > > >> > > >> > > >> Shade and relocate the external modules sounds ok as a short term > > >> solution. > > >> > > >> For the long term we should consider something like the second option > to > > >> add external modules without shipping uber jars. > > >> > > >> Thanks, > > >> Arun > > >> > > >> On 7/22/16, 6:07 AM, "Jungtaek Lim" <[email protected]> wrote: > > >> > > >> >Hi devs, > > >> > > > >> >AFAIK, we had been struggled to resolve dependency issues for > > storm-core. > > >> >As we all know, the strategy we have been using is shade & > relocating. > > >> > > > >> >Now State and Storm SQL requires that some of external modules need > to > > be > > >> >included to extlib, which is the classpath workers refer. > > >> > > > >> >http://issues.apache.org/jira/browse/STORM-1881 > > >> >https://issues.apache.org/jira/browse/STORM-1435 > > >> > > > >> >There're two issues here: > > >> >- We don't make uber jar for external modules so users need to find > and > > >> >copy dependencies jars to extlib manually. > > >> >- External modules also use Guava and Jackson and so on which are > > origin > > >> of > > >> >version conflict issues. > > >> > > > >> >So we should apply the shade & relocating strategy for every external > > >> >modules (at least storm-redis, storm-kafka, storm-sql-core, > > >> >storm-sql-kafka), or introduce the way to add the dependency without > > >> adding > > >> >them to extlib. (like --packages and --jar for Spark) > > >> > > > >> >Please express your opinions about this. > > >> > > > >> >Thanks, > > >> >Jungtaek Lim (HeartSaVioR) > > >> > > >> > > >> > > >> > > >> > > > > > > > > >
