Hey, Has anyone experienced with such an issue? Perhaps Costin can help here?
Thanks! On Thursday, February 12, 2015 at 8:27:14 AM UTC+2, Itai Yaffe wrote: > > Hey, > I've recently started using Elasticsearch for Spark (Scala application). > I've added elasticsearch-spark_2.10 version 2.1.0.BUILD-SNAPSHOT to my > Spark application pom file, and used > org.apache.spark.rdd.RDD[String].saveJsonToEs() to send documents to > Elasticsearch. > When the documents are loaded to Elasticsearch, my metadata fields (e.g > id, index, etc.) are being loaded as part of the _source field. > Is there a way to exclude them from the _source? > I've tried using the new "es.mapping.exclude" configuration property > (added in this commit > <https://github.com/elasticsearch/elasticsearch-hadoop/commit/aae4f0460a23bac9567ea2ad335c74245a1ba069> > > - that's why I needed to take the latest build rather than using version > 2.1.0.Beta3), but it doesn't seem to have any affect (although I'm not sure > it's even possible to exclude fields I'm using for mapping, e.g " > es.mapping.id"). > > A code snippet (I'm using a single-node Elasticsearch cluster for testing > purposes and running the Spark app from my desktop) : > val conf = new SparkConf()... > conf.set("es.index.auto.create", "false") > conf.set("es.nodes.discovery", "false") > conf.set("es.nodes", "XXX:9200") > conf.set("es.update.script", "XXX") > conf.set("es.update.script.params", "param1:events") > conf.set("es.update.retry.on.conflict" , "2") > conf.set("es.write.operation", "upsert") > conf.set("es.input.json", "true") > val documentsRdd = ... > documentsRdd.saveJsonToEs("test/user", scala.collection.Map(" > es.mapping.id" -> "_id", "es.mapping.exclude" -> "_id")) > > The JSON looks like that : > { > "_id": "XXXX", > "_type": "user", > "_index": "test", > "params": { > "events": [ > { > ... > } > ] > } > > Thanks! > } > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/aea88dfb-8d4b-49d1-a236-8de6d513b4f6%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
