Guys, just to be clear, the JSON use in Spark is primarily to serve the web UI, not to do any heavy parsing. Given that, I wouldn’t update the library for performance, just for other reasons. If you bring in another library that uses Jackson, one problem is to make sure that it also works with other libraries we depend on that use Jackson, like Hadoop. You’d need to test it on multiple Hadoop versions (1.0.4 and up since that’s where we start) to make sure it doesn’t create a problem.
Matei On Feb 14, 2014, at 2:48 PM, Pascal Voitot Dev <pascal.voitot....@gmail.com> wrote: > No worry! > I didn't want to be too harsh too :D > > upper-case was just to make it clear! > > Pascal > > On Fri, Feb 14, 2014 at 11:21 PM, Evan Chan <e...@ooyala.com> wrote: > >> Pascal, >> >> Ah, I stand corrected, thanks. >> >> On Mon, Feb 10, 2014 at 11:49 PM, Pascal Voitot Dev >> <pascal.voitot....@gmail.com> wrote: >>> Evan, >>> >>> Excuse me but that's WRONG that play-json pulls all play deps! >>> >>> PLAY/JSON has NO HEAVY DEP ON PLAY! >>> >>> I personally worked to make it an independent module in play! >>> So play/json has just one big dep which is Jackson! >>> >>> I agree that jackson is the right way to go as a beginning. >>> But for scala developers, a higher thin layer like play/json is useful to >>> bring typesafety... >>> >>> Pascal >>> >>> On Tue, Feb 11, 2014 at 1:31 AM, Evan Chan <e...@ooyala.com> wrote: >>> >>>> By the way, I did a benchmark on JSON parsing performance recently. >>>> Based on that, spray-json was about 10x slower than the Jackson-based >>>> parsers. I recommend json4s-jackson, because jackson is almost >>>> certainly already a dependency of Sparks (many other Java libraries >>>> use it), so the dependencies are very lightweight. I didn't >>>> benchmark Argonaut or play-json, partly because play-json pulled in >>>> all the Play dependencies, tho as someone else commented in this >>>> thread, they plan to split it out. >>>> >>>> On Mon, Feb 10, 2014 at 12:22 AM, Pascal Voitot Dev >>>> <pascal.voitot....@gmail.com> wrote: >>>>> Hi again, >>>>> >>>>> If spark just need Json with serialization/deserialization of basic >>>>> structures and some potential simple validations for webUI, let's >> remind >>>>> that play/json (without any other dependency than Jackson) is about >>>> 200-300 >>>>> lines of code... the only dependency is jackson which is the best json >>>>> parser that I know. The rest of code is about typesafety & >> composition... >>>>> If Spark need some json veryveryvery performant JSON >> (de)serialization, >>>> we >>>>> will have to look on things like pickling and potentially some >> streaming >>>>> parsers (I think this is a domain under work right now...) >>>>> >>>>> Pascal >>>>> >>>>> >>>>> On Mon, Feb 10, 2014 at 1:50 AM, Will Benton <wi...@redhat.com> >> wrote: >>>>> >>>>>> Matei, sorry if I was unclear: I'm referring to downstream operating >>>>>> system distributions (like Fedora or Debian) that have policies >>>> requiring >>>>>> that all packages are built from source (using only tools already >>>> packaged >>>>>> in the distribution). So end-users (and distributions with different >>>>>> policies) don't have to build Lift to get the lift-json artifact, but >>>> it is >>>>>> a concern for many open-source communities. >>>>>> >>>>>> best, >>>>>> wb >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Matei Zaharia" <matei.zaha...@gmail.com> >>>>>>> To: dev@spark.incubator.apache.org >>>>>>> Sent: Sunday, February 9, 2014 4:29:20 PM >>>>>>> Subject: Re: proposal: replace lift-json with spray-json >>>>>>> >>>>>>> Will, why are you saying that downstream distributes need to build >>>> all of >>>>>>> Lift to package lift-json? Spark just downloads it from Maven >> Central, >>>>>> where >>>>>>> it's a JAR with no external dependencies. We don't have any >>>> dependency on >>>>>>> the rest of lift. >>>>>>> >>>>>>> Matei >>>>>>> >>>>>>> On Feb 9, 2014, at 11:28 AM, Will Benton <wi...@redhat.com> wrote: >>>>>>> >>>>>>>> lift-json is a nice library, but Lift is a pretty heavyweight >>>>>> dependency to >>>>>>>> track just for its JSON support. (lift-json is relatively >>>>>> self-contained >>>>>>>> as a dependency from an end-user's perspective, but downstream >>>>>>>> distributors need to build all of Lift in order to package the >> JSON >>>>>>>> support.) I understand that this has come up before (cf. >> SPARK-883) >>>>>> and >>>>>>>> that the uncertain future of JSON support in the Scala standard >>>>>> library is >>>>>>>> the motivator for relying on an external library. >>>>>>>> >>>>>>>> I'm proposing replacing lift-json in Spark with something more >>>>>> lightweight. >>>>>>>> I've evaluated apparent project liveness and dependency scope for >>>> most >>>>>> of >>>>>>>> the current Scala JSON libraries and believe the best candidate >> is >>>>>>>> spray-json (https://github.com/spray/spray-json), the JSON >> library >>>>>> used by >>>>>>>> the Spray HTTP toolkit. spray-json is Apache-licensed, actively >>>>>> developed, >>>>>>>> and builds and works independently of Spray with only one >> external >>>>>>>> dependency. >>>>>>>> >>>>>>>> It looks to me like a pretty straightforward change (although >>>>>>>> JsonProtocol.scala would be a little more verbose since it >> couldn't >>>> use >>>>>>>> the Lift JSON DSL), and I'd like to do it. I'm writing now to >> ask >>>> for >>>>>>>> some community feedback before making the change (and submitting >> a >>>> JIRA >>>>>>>> and PR). If no one has any serious objections (to the effort in >>>>>> general >>>>>>>> or to to the choice of spark-json in particular), I'll go ahead >> and >>>> do >>>>>> it, >>>>>>>> but if anyone has concerns, I'd be happy to discuss and address >> them >>>>>>>> before getting started. >>>>>>>> >>>>>>>> >>>>>>>> thanks, >>>>>>>> wb >>>>>>> >>>>>>> >>>>>> >>>> >>>> >>>> >>>> -- >>>> -- >>>> Evan Chan >>>> Staff Engineer >>>> e...@ooyala.com | >>>> >> >> >> >> -- >> -- >> Evan Chan >> Staff Engineer >> e...@ooyala.com | >>
