Ok. I think that you need to use something else than the transport plugin for your specific use case which is not a database sync feature.
Any chance you could push from your source to couchbase AND elasticsearch at the same time? Or write a batch which reads couchbase and write to elasticsearch. I don't think there is yet an input for logstash but I might be wrong... -- David ;-) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > Le 14 janv. 2015 à 19:38, Behrad <[email protected]> a écrit : > > This is our persistent storage, but the working set is a multiple days, > older data only is needed for search and audits, the whole data is being > backed up and archived by a separate process, so we don't need extra large > hard disks (each billion records gets to 600GB) Redis is not designed for > persistency and Message Queues are slow when persistency is envolved, if we > were using a Message Queue, we also should have persist our data on a durable > external storage. I want to have a small working set of data of a week, and a > workhouse of multi months of past data. (I can't see why to keep multi > billions of passive data in Couchbase which should be searched by ES!?) > > 2015-01-14 21:02 GMT+03:30 David Pilato <[email protected]>: >> I see. Just wondering why you are sending things to couchbase then. Why not >> using Redis or RabbitMQ or any message queue System. I mean that it sounds >> like you are using couchbase as a queue for elasticsearch. >> >> Or if you want a Backup in couchbase, could you imagine sending your data >> from your application to couchbase and elasticsearch at the same time? >> >> I think you are trying to twist here the couchbase plugin which is designed >> to keep the database in sync with elasticsearch which is obviously not what >> you are looking for. >> >> My 2 cents. >> >> David >> >>> Le 14 janv. 2015 à 18:18, Behrad <[email protected]> a écrit : >>> >>> >>> >>> 2015-01-14 18:25 GMT+03:30 David Pilato <[email protected]>: >>>> Definitely scan and scroll API is for that. >>>> Some clients provide that feature out of the box. >>>> >>>> That said, reading from couchbase is something super easy. So I guess you >>>> could may be replicate your data from couchbase twice using 2 XDCR configs >>>> one for the weekly index and the other one for the year index. Is that >>>> something doable? >>> >>> Unfortunately not David, XDCR blinding syncs the source db with remotes >>> (any deleted data will also be deleted from the remote) and we can't let >>> the source db to grow (about 1.3 billion records/month) so we are using >>> couchbase TTLs to store a week or so, >>> however we'd like to have online searching/access to a multiple month (i >>> hope by utilizing ES) and I'm seeking the best Enterprise approach to this. >>> >>> However I haven't any clue for capacity planning of ES for 5, 6 billions of >>> records. >>>> >>>> >>>> >>>> -- >>>> David Pilato | Technical Advocate | Elasticsearch.com >>>> @dadoonet | @elasticsearchfr | @scrutmydocs >>>> >>>> >>>> >>>>> Le 14 janv. 2015 à 14:51, Behrad <[email protected]> a écrit : >>>>> >>>>> I'll recheck elasticsearch-transport-couchbase again David, another two >>>>> related questions comes to my mind to ask, >>>>> >>>>> 1) Is there any best practice to replicate/sync/duplicate an existing >>>>> index to a new index by ES? >>>>> 2) Considering I have an existing index with data persist enabled on ES, >>>>> Can I use that to create a new index with a customer defined mapping on >>>>> my data on ES without the need of my primary database? (does sroll api >>>>> help with this?) >>>>> >>>>> 2015-01-14 17:00 GMT+03:30 David Pilato <[email protected]>: >>>>>> May be you could use different mapping for the year index and the week >>>>>> index? >>>>>> One with ttl disabled and the other one with ttl enabled? >>>>>> >>>>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html >>>>>> >>>>>> -- >>>>>> David Pilato | Technical Advocate | Elasticsearch.com >>>>>> @dadoonet | @elasticsearchfr | @scrutmydocs >>>>>> >>>>>> >>>>>> >>>>>>> Le 14 janv. 2015 à 14:21, Behrad <[email protected]> a écrit : >>>>>>> >>>>>>> I am actually using Couchbase ES plugin David, this is my database >>>>>>> requirement that data has a weekly ttl, so when data is removed in >>>>>>> Couchbase, ES plugin also removes it (using Couchbase's XDCR) within >>>>>>> ES. >>>>>>> >>>>>>> 2015-01-14 16:43 GMT+03:30 David Pilato <[email protected]>: >>>>>>>> Don't use ttl for that kind of usage but prefer time based indices. >>>>>>>> Much more efficient. >>>>>>>> >>>>>>>> So the year index won't have any removal and you will end up to just >>>>>>>> remove weekly indices which is super fast and doesn't generate a lot >>>>>>>> of IO (but ttl does) >>>>>>>> >>>>>>>> HTH >>>>>>>> >>>>>>>> David >>>>>>>> >>>>>>>>> Le 14 janv. 2015 à 14:01, Behrad Zari <[email protected]> a écrit : >>>>>>>>> >>>>>>>>> My data contains a weekly ttl so my ES index will loose older data, >>>>>>>>> however I want to have another accumulative large index containing a >>>>>>>>> year of my data... How can I instruct ES/sync my second index without >>>>>>>>> affecting ttls in the first index? ( I don't want deleted docs from >>>>>>>>> first index to be deleted in the second) >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "elasticsearch" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to a topic in the >>>>>>>> Google Groups "elasticsearch" group. >>>>>>>> To unsubscribe from this topic, visit >>>>>>>> https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe. >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr. >>>>>>>> >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> --Behrad >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "elasticsearch" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>> an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in the >>>>>> Google Groups "elasticsearch" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr. >>>>>> >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> >>>>> -- >>>>> --Behrad >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "elasticsearch" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr. >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >>> -- >>> --Behrad >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com. >>> For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr. >> >> For more options, visit https://groups.google.com/d/optout. > > > > -- > --Behrad > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9E76ABA8-E485-4568-94CA-534F88A00DC8%40pilato.fr. For more options, visit https://groups.google.com/d/optout.
