Have a look at https://github.com/elastic/elasticsearch-hadoop#apache-spark
You can simply call the .saveToEs function to store your RDD data into ES. Thanks Best Regards On Thu, Oct 29, 2015 at 8:19 PM, Nipun Arora <[email protected]> wrote: > Hi, > > I am sending data to an elasticsearch deployment. The printing to file > seems to work fine, but I keep getting no-node found for ES when I send > data to it. I suspect there is some special way to handle the connection > object? Can anyone explain what should be changed here? > > Thanks > Nipun > > The following is the code block where I send the data > > > > > addMTSUnmatched.foreach( > new Function<JavaRDD<String>, Void>() { > @Override > public Void call(JavaRDD<String> stringJavaRDD) throws Exception { > List<String> list = stringJavaRDD.collect(); > for (String str : list){ > if(OnlineUtils.ESFlag) { > OnlineUtils.printToFile(str, 1, type1_outputFile, > OnlineUtils.client); > }else{ > OnlineUtils.printToFile(str, 1, type1_outputFile); > } > } > return null; > } > } > ); > > > public static void printToFile(String str, int type, File fileID, Client > client) throws IOException { > JSONObject obj = new JSONObject(str); > if (obj.has("message")) { > String message = obj.get("message").toString(); > obj.put("message", message.replace("'", "")); > } > obj.put("anomaly_type", type); > > ESUtils.putMessage(client,"necla","demo",obj.toString()); > > Files.append(str + "\n", fileID, Charset.defaultCharset()); > } > >
