Agreed. Reading is database could be an heavy operation. That’s why I always recommend if possible to send to elasticsearch data in the same « transaction » as you persist your data to the database. You often have a bean ready to be persisted in memory. Serializing to JSON with Jackson and then sending to elasticsearch does not cost much in term of CPU usage/memory and in term of line of codes.
But for that, you need to have an access to the application which produces the data. If not, I agree, you should not reinvent the wheel and use whatever ETL you want to read, transform and index data. Such as JDBC River. My advice about JDBC river would be to run it outside elasticsearch as a feeder: https://github.com/jprante/elasticsearch-river-jdbc#two-flavors-river-or-feeder <https://github.com/jprante/elasticsearch-river-jdbc#two-flavors-river-or-feeder> Best -- David Pilato | Technical Advocate | Elasticsearch.com @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr <https://twitter.com/elasticsearchfr> | @scrutmydocs <https://twitter.com/scrutmydocs> > Le 7 janv. 2015 à 18:47, [email protected] a écrit : > > The definitions of "real time" are quite vague. I also tend to use "real > time" for sub-second latencies with a guarantee of a maximum response time. > Other just mean by "real time" everything that is not batching, pushing data > over the wire as the changes happen, no matter how large the change is - a > process which can take huge delays in worst case, even minutes. > > Although it is possible to run a select query every second with JDBC plugin > crontab, this puts a heavy load on the system with not much gain, so I don't > recommend it. > > For instant replication, one would have to extend JDBC plugin with a special > queue on the side of the RDBMS and use proprietary driver API. Even with > such a queue the latency is often higher than a few seconds. > > Jörg > > > > On Wed, Jan 7, 2015 at 5:52 PM, David Pilato <[email protected] > <mailto:[email protected]>> wrote: > Well. I did not mention your plugin because he basically asked for "I want to > know how can I input the data that I'm getting in Elasticsearch in real > time.". > Real time aspect here seems to be important. > > Elasticsearch is not real time but near real time. > JDBC river is obviously not real time as IIRC it runs a job every x minutes > or so. > > > > -- > David Pilato | Technical Advocate | Elasticsearch.com > <http://elasticsearch.com/> > @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr > <https://twitter.com/elasticsearchfr> | @scrutmydocs > <https://twitter.com/scrutmydocs> > > > >> Le 7 janv. 2015 à 17:35, [email protected] >> <mailto:[email protected]> a écrit : >> >> You can use the JDBC plugin instead of reinventing the wheel. >> >> https://github.com/jprante/elasticsearch-river-jdbc >> <https://github.com/jprante/elasticsearch-river-jdbc> >> >> Jörg >> >> On Wed, Jan 7, 2015 at 5:28 PM, Marian Valero <[email protected] >> <mailto:[email protected]>> wrote: >> Thanks! >> >> I have a problem with jsonBuilder() method, eclipse don't found this, I do >> it that way: >> >> public static void main(String args[]) throws SQLException { >> >> int count=0; >> Node node = nodeBuilder().client(true).node(); >> Client client = node.client(); >> >> String url = "jdbc:oracle:thin:@eso:1521:eso"; >> >> Properties props = new Properties(); >> props.setProperty("user", "eso"); >> props.setProperty("password", "eso"); >> >> Connection conn = DriverManager.getConnection(url,props); >> >> String sql ="select * from responselog"; >> >> PreparedStatement preStatement = conn.prepareStatement(sql); >> >> ResultSet result = preStatement.executeQuery(); >> >> // Timestamp timestamp = null; >> >> BulkRequestBuilder bulkRequest = client.prepareBulk(); >> >> while(result.next() && count<1000){ >> count++; >> String id = result.getString("id"); >> String deliveryid = result.getString("deliveryid"); >> String msgid = result.getString("msgid"); >> String rspdate = result.getString("rspdate"); >> String parsedresponse = result.getString("parsedresponse"); >> String shortcode = result.getString("shortcode"); >> String insid = result.getString("insid"); >> String mobilenumber = result.getString("mobilenumber"); >> String rawresponse = result.getString("rawresponse"); >> >> XContentBuilder builder = jsonBuilder() >> .startObject() >> .field("deliveryid", deliveryid) >> .field("msgid", msgid) >> .field("message", "trying out Elasticsearch") >> .endObject(); >> >> >> // YOUR ORACLE stuff here >> // either use client#prepare, or use Requests# to directly >> build index/delete requests >> bulkRequest.add(client.prepareIndex("logs", "responselog", id) >> .setSource(jsonBuilder() >> .startObject() >> .field("deliveryid", deliveryid) >> .field("msgid", msgid) >> .field("rspdate", rspdate) >> .field("parsedresponse", parsedresponse) >> .field("shortcode", shortcode) >> .field("insid", insid) >> .field("mobilenumber", mobilenumber) >> .field("rawresponse", rawresponse) >> .endObject() >> ) >> ); >> >> System.out.println("Responselog from Oracle : "+ >> result.getString("id")+" "+result.getString("deliveryid")+" >> "+result.getString("msgid")+" "+result.getString("rspdate")+" >> "+result.getString("parsedresponse")+" "+result.getString("shortcode")+" >> "+result.getString("insid")+" "+result.getString("mobilenumber")+" >> "+result.getString("rawresponse")); >> } >> >> BulkResponse bulkResponse = bulkRequest.execute().actionGet(); >> System.out.println("done"); >> } >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/865f9ddc-517e-4e1b-aba4-8d7e2a2eebf1%40googlegroups.com?utm_medium=email&utm_source=footer>. >> >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFcx7gXvXzg83cu9dFx7%3DYtW%2BA_K%2BtBANUpdD-86uB8GQ%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFcx7gXvXzg83cu9dFx7%3DYtW%2BA_K%2BtBANUpdD-86uB8GQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/AA7A49A7-5BFC-4DFD-9745-BBEAAD60E603%40pilato.fr > > <https://groups.google.com/d/msgid/elasticsearch/AA7A49A7-5BFC-4DFD-9745-BBEAAD60E603%40pilato.fr?utm_medium=email&utm_source=footer>. > > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE0CUreVbvpQrtXE186yqjA%2BtiYPCfZkwK_3QxfoOnOSA%40mail.gmail.com > > <https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE0CUreVbvpQrtXE186yqjA%2BtiYPCfZkwK_3QxfoOnOSA%40mail.gmail.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8F8C7196-83E8-48BC-9AF3-94F866A65E8B%40pilato.fr. For more options, visit https://groups.google.com/d/optout.
