Yea, since you mentioned real time, I thought you were concerned about data coming in 'today.'
For older data, the only aspect of real time that applies is how soon you can search against ES once loaded. ES supports near real time so this isn't really an issue, and considering you have 10 year old data, I had assumed it's ok if the older data doesn't have to be readily available until the entire data set is loaded, especially since the intent for these type of things is to get an overall view of all the data points. Anyways, one thing I did in the past was to partition the historical data (in oracle) and then index the data into ES in time intervals. For example, store 1 year's worth of data in a single index. If doing it this way, you will have 10 indexes, and for query purposes, you can set aliases on the indexes and control your query using aliases. For instance, if most of your queries for analysis are for 'the last year', you can query against the index that contains data for 2014 and 2015. This way you don't query a giant index that can go back 10 years. Also, by breaking down into chunks, any indexing failures would not cause me to re-index the entire thing, which could take a very long time. Best wishes on your project! On Tuesday, January 13, 2015 at 1:07:31 PM UTC-8, Marian Valero wrote: > > Thanks Ed. > > Yes, but I have data from 10 years ago and I have to input this data too > for analyze that logs with the data that is insert every day. There are > millions of logs. > > 2015-01-13 16:22 GMT-04:30 Ed Kim <[email protected] <javascript:>>: > >> Not sure if this is an option for you, but if an application is feeding >> that log data into oracle, you could consider having that application also >> index into ElasticSearch. >> >> On Wednesday, January 7, 2015 at 5:46:18 AM UTC-8, Marian Valero wrote: >>> >>> I'm reading data from my oracle database to java with JDBC but I want to >>> know how can I input the data that I'm getting in Elasticsearch in real >>> time. >>> >>> Thanks. >>> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/cnkq84aom1M/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/86e937ac-af3f-43f7-8c1f-60c7517b8cbf%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/86e937ac-af3f-43f7-8c1f-60c7517b8cbf%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1fd5e8fb-117e-419b-b5b7-cda96a55abec%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
