Here is the gradle build I was using originally:
apply plugin: 'java'
apply plugin: 'eclipse'
sourceCompatibility = 1.7
version = '0.0.1'
group = 'com.spark.testing'
repositories {
mavenCentral()
}
dependencies {
compile 'org.apache.spark:spark-core_2.10:1.0.0'
compile 'edu.stanford.nlp:stanford-corenlp:3.3.1'
compile group: 'edu.stanford.nlp', name: 'stanford-corenlp', version:
'3.3.1', classifier:'models'
compile files('lib/elasticsearch-hadoop-2.0.0.jar')
testCompile 'junit:junit:4.+'
testCompile group: "com.github.tlrx", name: "elasticsearch-test", version:
"1.2.1"
}
When I ran dependencyInsight on jackson, I got the following output:
C:\dev\workspace\SparkProject>gradle dependencyInsight --dependency
jackson-core
:dependencyInsight
com.fasterxml.jackson.core:jackson-core:2.3.0
\--- com.fasterxml.jackson.core:jackson-databind:2.3.0
+--- org.json4s:json4s-jackson_2.10:3.2.6
| \--- org.apache.spark:spark-core_2.10:1.0.0
| \--- compile
\--- com.codahale.metrics:metrics-json:3.0.0
\--- org.apache.spark:spark-core_2.10:1.0.0 (*)
org.codehaus.jackson:jackson-core-asl:1.0.1
\--- org.codehaus.jackson:jackson-mapper-asl:1.0.1
\--- org.apache.hadoop:hadoop-core:1.0.4
\--- org.apache.hadoop:hadoop-client:1.0.4
\--- org.apache.spark:spark-core_2.10:1.0.0
\--- compile
Version 1.0.1 of jackson-core-asl does not have the field
ALLOW_UNQUOTED_FIELD_NAMES, but later versions of it do.
On Sunday, July 6, 2014 4:28:56 PM UTC-4, Costin Leau wrote:
>
> Hi,
>
> Glad to see you sorted out the problem. Out of curiosity what version of
> jackson were you using and what was pulling it in? Can you share you maven
> pom/gradle build?
>
>
> On Sun, Jul 6, 2014 at 10:27 PM, Brian Thomas <[email protected]
> <javascript:>> wrote:
>
>> I figured it out, dependency issue in my classpath. Maven was pulling
>> down a very old version of the jackson jar. I added the following line to
>> my dependencies and the error went away:
>>
>> compile 'org.codehaus.jackson:jackson-mapper-asl:1.9.13'
>>
>>
>> On Friday, July 4, 2014 3:22:30 PM UTC-4, Brian Thomas wrote:
>>>
>>> I am trying to test querying elasticsearch using Apache Spark using
>>> elasticsearch-hadoop. I am just trying to do a query to the elasticsearch
>>> server and return the count of results.
>>>
>>> Below is my test class using the Java API:
>>>
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.io.MapWritable;
>>> import org.apache.hadoop.io.Text;
>>> import org.apache.spark.SparkConf;
>>> import org.apache.spark.api.java.JavaPairRDD;
>>> import org.apache.spark.api.java.JavaSparkContext;
>>> import org.apache.spark.serializer.KryoSerializer;
>>> import org.elasticsearch.hadoop.mr.EsInputFormat;
>>>
>>> import scala.Tuple2;
>>>
>>> public class ElasticsearchSparkQuery{
>>>
>>> public static int query(String masterUrl, String
>>> elasticsearchHostPort) {
>>> SparkConf sparkConfig = new SparkConf().setAppName("
>>> ESQuery").setMaster(masterUrl);
>>> sparkConfig.set("spark.serializer",
>>> KryoSerializer.class.getName());
>>> JavaSparkContext sparkContext = new
>>> JavaSparkContext(sparkConfig);
>>>
>>> Configuration conf = new Configuration();
>>> conf.setBoolean("mapred.map.tasks.speculative.execution",
>>> false);
>>> conf.setBoolean("mapred.reduce.tasks.speculative.execution",
>>> false);
>>> conf.set("es.nodes", elasticsearchHostPort);
>>> conf.set("es.resource", "media/docs");
>>> conf.set("es.query", "?q=*");
>>>
>>> JavaPairRDD<Text, MapWritable> esRDD =
>>> sparkContext.newAPIHadoopRDD(conf, EsInputFormat.class, Text.class,
>>> MapWritable.class);
>>> return (int) esRDD.count();
>>> }
>>> }
>>>
>>>
>>> When I try to run this I get the following error:
>>>
>>>
>>> 4/07/04 14:58:07 INFO executor.Executor: Running task ID 0
>>> 14/07/04 14:58:07 INFO storage.BlockManager: Found block broadcast_0
>>> locally
>>> 14/07/04 14:58:07 INFO rdd.NewHadoopRDD: Input split: ShardInputSplit
>>> [node=[5UATWUzmTUuNzhmGxXWy_w/S'byll|10.45.71.152:9200],shard=0]
>>> 14/07/04 14:58:07 WARN mr.EsInputFormat: Cannot determine task id...
>>> 14/07/04 14:58:07 ERROR executor.Executor: Exception in task ID 0
>>> java.lang.NoSuchFieldError: ALLOW_UNQUOTED_FIELD_NAMES
>>> at org.elasticsearch.hadoop.serialization.json.
>>> JacksonJsonParser.<clinit>(JacksonJsonParser.java:38)
>>> at org.elasticsearch.hadoop.serialization.ScrollReader.
>>> read(ScrollReader.java:75)
>>> at org.elasticsearch.hadoop.rest.RestRepository.scroll(
>>> RestRepository.java:267)
>>> at org.elasticsearch.hadoop.rest.ScrollQuery.hasNext(
>>> ScrollQuery.java:75)
>>> at org.elasticsearch.hadoop.mr.EsInputFormat$ShardRecordReader.next(
>>> EsInputFormat.java:319)
>>> at org.elasticsearch.hadoop.mr.EsInputFormat$ShardRecordReader.
>>> nextKeyValue(EsInputFormat.java:255)
>>> at org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(
>>> NewHadoopRDD.scala:122)
>>> at org.apache.spark.InterruptibleIterator.hasNext(
>>> InterruptibleIterator.scala:39)
>>> at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1014)
>>> at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847)
>>> at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:847)
>>> at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(
>>> SparkContext.scala:1080)
>>> at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(
>>> SparkContext.scala:1080)
>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.
>>> scala:111)
>>> at org.apache.spark.scheduler.Task.run(Task.scala:51)
>>> at org.apache.spark.executor.Executor$TaskRunner.run(
>>> Executor.scala:187)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>> ThreadPoolExecutor.java:1145)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>> ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Has anyone run into this issue with the JacksonJsonParser?
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/9c2b2f2e-5196-4a72-bfbc-4cd0fda9edf0%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/elasticsearch/9c2b2f2e-5196-4a72-bfbc-4cd0fda9edf0%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ecca33ea-b1e0-4196-84f0-c3c0838de786%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
