Re: How spark depends on Guava
On 23 May 2016, at 06:32, Todd <bit1...@163.com<mailto:bit1...@163.com>> wrote: Can someone please take alook at my question?I am spark-shell local mode and yarn-client mode.Spark code uses guava library,spark should have guava in place during run time. Thanks. At 2016-05-23 11:48:58, "Todd" <bit1...@163.com<mailto:bit1...@163.com>> wrote: Hi, In the spark code, guava maven dependency scope is provided, my question is, how spark depends on guava during runtime? I looked into the spark-assembly-1.6.1-hadoop2.6.1.jar,and didn't find class entries like com.google.common.base.Preconditions etc... Spark "shades" guava on import into the assembly; the libaries are moved to a new package and all internal references with it. This is because guava is a nightmare of backwards compatibility. if you want guava, decide which version you want and ask for it explicitly
Re: How spark depends on Guava
Hi Todd, It's used heavily for thread pool executors for one. Don't know about other uses. Jacek On 23 May 2016 5:49 a.m., "Todd" <bit1...@163.com> wrote: > Hi, > In the spark code, guava maven dependency scope is provided, my question > is, how spark depends on guava during runtime? I looked into the > spark-assembly-1.6.1-hadoop2.6.1.jar,and didn't find class entries like > com.google.common.base.Preconditions etc... >
Re:Re: How spark depends on Guava
Thanks Mat. When you look at the spark assembly jar(such as spark-assembly-1.6.0-hadoop2.6.1.jar), you will find that there is very few classes belonging to guava library. So i am wondering where guava library comes into play during run-time. At 2016-05-23 15:42:51, "Mat Schaffer" <m...@schaffer.me> wrote: I got curious so I tried sbt dependencyTree. Looks like Guava comes into spark core from a couple places. -Mat matschaffer.com On Mon, May 23, 2016 at 2:32 PM, Todd <bit1...@163.com> wrote: Can someone please take alook at my question?I am spark-shell local mode and yarn-client mode.Spark code uses guava library,spark should have guava in place during run time. Thanks. At 2016-05-23 11:48:58, "Todd" <bit1...@163.com> wrote: Hi, In the spark code, guava maven dependency scope is provided, my question is, how spark depends on guava during runtime? I looked into the spark-assembly-1.6.1-hadoop2.6.1.jar,and didn't find class entries like com.google.common.base.Preconditions etc...
How spark depends on Guava
Hi, In the spark code, guava maven dependency scope is provided, my question is, how spark depends on guava during runtime? I looked into the spark-assembly-1.6.1-hadoop2.6.1.jar,and didn't find class entries like com.google.common.base.Preconditions etc...
