Jungtaek, I’ve given you permissions to the Storm wiki so you can post the document there.
-Taylor > On Aug 1, 2016, at 12:46 AM, Jungtaek Lim <[email protected]> wrote: > > Not yet. It would be good to continue discussion on this thread for now. > > Thanks, > Jungtaek Lim (HeartSaVioR) > > 2016년 8월 1일 (월) 오후 1:38, Satish Duggana <[email protected]>님이 작성: > >> Hi, >> Did we find a way to comment on the proposal? How can we proceed with this >> discussion? >> >> Thanks, >> Satish. >> >> >> On Thu, Jul 28, 2016 at 9:42 AM, Jungtaek Lim <[email protected]> wrote: >> >>> I've made a design document for adding jars and maven artifacts at >>> submission. >>> Since we don't have any formats for this, I just borrowed KIP format. >>> That's not we would like to adopt that format, it's just me. >>> >>> >>> >> https://docs.google.com/document/d/1f3Ed0Wa8aN2j7gtptT0BOKYMyBdpohLgERg4YebJJ7c/edit?usp=sharing >>> >>> Btw, I guess ASF policy requires that discussion history should be logged >>> to Apache Infra. >>> (So I only grant view permission who opens the doc via link.) >>> >>> So what's recommended way to do? Would we want to file an issue for JIRA >>> with attaching design doc to PDF and discuss there? >>> >>> - Jungtaek Lim >>> >>> 2016년 7월 23일 (토) 오전 12:06, Jungtaek Lim <[email protected]>님이 작성: >>> >>>> Thought about it once more, and found that former approach still needs >> to >>>> add 'provided' scope libraries to extlib directory.. >>>> >>>> Along with thinking former approach, I've experimented a bit of latter >>>> approach by creating POC project. Since I don't know about copy and use >>> ASF >>>> project codes for personal use, I'd just share main class to see if >> this >>>> POC and my theory makes sense for us. >>>> https://gist.github.com/HeartSaVioR/3639a9ee829fe1203b4a085a0fb069d6 >>>> 'zeppelin' package has some classes for transitive dependency resolver >>>> (copied from Zeppelin), and 'spark' package has some classes for >>>> classloader (yes also copied from Spark). >>>> >>>> Please share your experiences if you have knowledges regarding this >> area. >>>> >>>> - Jungtaek Lim >>>> >>>> 2016년 7월 22일 (금) 오후 10:59, Bobby Evans <[email protected]>님이 >>> 작성: >>>> >>>>> You can do that with a combination of the distributed cache and >> setting >>>>> the classpath, just like with hadoop. It is not as clean as it could >>> be, >>>>> but it does work. - Bobby >>>>> >>>>> On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan < >>> [email protected]> >>>>> wrote: >>>>> >>>>> >>>>> Shade and relocate the external modules sounds ok as a short term >>>>> solution. >>>>> >>>>> For the long term we should consider something like the second option >> to >>>>> add external modules without shipping uber jars. >>>>> >>>>> Thanks, >>>>> Arun >>>>> >>>>> On 7/22/16, 6:07 AM, "Jungtaek Lim" <[email protected]> wrote: >>>>> >>>>>> Hi devs, >>>>>> >>>>>> AFAIK, we had been struggled to resolve dependency issues for >>> storm-core. >>>>>> As we all know, the strategy we have been using is shade & >> relocating. >>>>>> >>>>>> Now State and Storm SQL requires that some of external modules need >> to >>> be >>>>>> included to extlib, which is the classpath workers refer. >>>>>> >>>>>> http://issues.apache.org/jira/browse/STORM-1881 >>>>>> https://issues.apache.org/jira/browse/STORM-1435 >>>>>> >>>>>> There're two issues here: >>>>>> - We don't make uber jar for external modules so users need to find >> and >>>>>> copy dependencies jars to extlib manually. >>>>>> - External modules also use Guava and Jackson and so on which are >>> origin >>>>> of >>>>>> version conflict issues. >>>>>> >>>>>> So we should apply the shade & relocating strategy for every external >>>>>> modules (at least storm-redis, storm-kafka, storm-sql-core, >>>>>> storm-sql-kafka), or introduce the way to add the dependency without >>>>> adding >>>>>> them to extlib. (like --packages and --jar for Spark) >>>>>> >>>>>> Please express your opinions about this. >>>>>> >>>>>> Thanks, >>>>>> Jungtaek Lim (HeartSaVioR) >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >>
signature.asc
Description: Message signed with OpenPGP using GPGMail
