Hey,
I've recently started using Elasticsearch for Spark (Scala application).
I've added elasticsearch-spark_2.10 version 2.1.0.BUILD-SNAPSHOT to my
Spark application pom file, and used
org.apache.spark.rdd.RDD[String].saveJsonToEs() to send documents to
Elasticsearch.
When the documents are loaded to Elasticsearch, my metadata fields (e.g id,
index, etc.) are being loaded as part of the _source field.
Is there a way to exclude them from the _source?
I've tried using the new "es.mapping.exclude" configuration property (added
in this commit
<https://github.com/elasticsearch/elasticsearch-hadoop/commit/aae4f0460a23bac9567ea2ad335c74245a1ba069>
- that's why I needed to take the latest build rather than using version
2.1.0.Beta3), but it doesn't seem to have any affect (although I'm not sure
it's even possible to exclude fields I'm using for mapping, e.g
"es.mapping.id").
A code snippet (I'm using a single-node Elasticsearch cluster for testing
purposes and running the Spark app from my desktop) :
val conf = new SparkConf()...
conf.set("es.index.auto.create", "false")
conf.set("es.nodes.discovery", "false")
conf.set("es.nodes", "XXX:9200")
conf.set("es.update.script", "XXX")
conf.set("es.update.script.params", "param1:events")
conf.set("es.update.retry.on.conflict" , "2")
conf.set("es.write.operation", "upsert")
conf.set("es.input.json", "true")
val documentsRdd = ...
documentsRdd.saveJsonToEs("test/user",
scala.collection.Map("es.mapping.id" -> "_id", "es.mapping.exclude" ->
"_id"))
The JSON looks like that :
{
"_id": "XXXX",
"_type": "user",
"_index": "test",
"params": {
"events": [
{
...
}
]
}
Thanks!
}
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8055055f-8787-492b-97f4-144b2a7f7fce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
