Re: Use maxmind geoip lib to process ip on Spark/Spark Streaming

1).map(_.latitude).getOrElse(None).toString >> val longitude = >> (lookupResult._1).map(_.longitude).getOrElse(None).toString >> return List(countryName, city, latitude, longitude) >> } >> sc.addFile("/home/your_user/GeoLiteCity.dat") >> >>

Use maxmind geoip lib to process ip on Spark/Spark Streaming

Hi all, Currently, I sent nginx log to Kafka then I want to use Spark Streaming to parse the log and enrich the IP info with geoip libs from Maxmind. I found this one https://github.com/Sanoma-CDA/maxmind-geoip2-scala.git , but spark

Move Spark configuration from SPARK_CLASSPATH to spark-default.conf , HiveContext went wrong with Class com.hadoop.compression.lzo.LzoCodec not found

        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)         at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)         ... 68 more -- Zhun Shen Data Mining at LightnInTheBox.com Email: shenzhunal...@gmail.com | shenz

Got error “java.lang.IllegalAccessError when using HiveContext in Spark shell on AWS

Hi, When I try to use HiveContext in Spark shell on AWS, I got the error java.lang.IllegalAccessError: tried to access method com.google.common.collect.MapMaker.makeComputingMap(Lcom/google/common/base/Function;)Ljava/util/concurrent/ConcurrentMap. I follow the steps below to compile and install

Re: Got error “java.lang.IllegalAccessError when using HiveContext in Spark shell on AWS

. -- Zhun Shen Data Mining at LightnInTheBox.com Email: shenzhunal...@gmail.com | shenz...@yahoo.com Phone: 186 0627 7769 GitHub: https://github.com/shenzhun LinkedIn: http://www.linkedin.com/in/shenzhun On August 7, 2014 at 6:57:06 PM, Cheng Lian (lian.cs@gmail.com) wrote: Hey Zhun, Thanks