2015-01-14 22:23 GMT+03:30 David Pilato <[email protected]>: > Ok. I think that you need to use something else than the transport plugin > for your specific use case which is not a database sync feature. > > Any chance you could push from your source to couchbase AND elasticsearch > at the same time? Or write a batch which reads couchbase and write to > elasticsearch. >
I can stream from Couchbase views into ES by a hand made script., but I was elaborating available toolset to achieve this automatically. > I don't think there is yet an input for logstash but I might be wrong... > > -- > David ;-) > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 14 janv. 2015 à 19:38, Behrad <[email protected]> a écrit : > > This is our persistent storage, but the working set is a multiple days, > older data only is needed for search and audits, the whole data is being > backed up and archived by a separate process, so we don't need extra large > hard disks (each billion records gets to 600GB) Redis is not designed for > persistency and Message Queues are slow when persistency is envolved, if we > were using a Message Queue, we also should have persist our data on a > durable external storage. I want to have a small working set of data of a > week, and a workhouse of multi months of past data. (I can't see why to > keep multi billions of passive data in Couchbase which should be searched > by ES!?) > > 2015-01-14 21:02 GMT+03:30 David Pilato <[email protected]>: > >> I see. Just wondering why you are sending things to couchbase then. Why >> not using Redis or RabbitMQ or any message queue System. I mean that it >> sounds like you are using couchbase as a queue for elasticsearch. >> >> Or if you want a Backup in couchbase, could you imagine sending your data >> from your application to couchbase and elasticsearch at the same time? >> >> I think you are trying to twist here the couchbase plugin which is >> designed to keep the database in sync with elasticsearch which is obviously >> not what you are looking for. >> >> My 2 cents. >> >> David >> >> Le 14 janv. 2015 à 18:18, Behrad <[email protected]> a écrit : >> >> >> >> 2015-01-14 18:25 GMT+03:30 David Pilato <[email protected]>: >> >>> Definitely scan and scroll API is for that. >>> Some clients provide that feature out of the box. >>> >>> That said, reading from couchbase is something super easy. So I guess >>> you could may be replicate your data from couchbase twice using 2 XDCR >>> configs one for the weekly index and the other one for the year index. Is >>> that something doable? >>> >> >> Unfortunately not David, XDCR blinding syncs the source db with remotes >> (any deleted data will also be deleted from the remote) and we can't let >> the source db to grow (about 1.3 billion records >> /month) so we are using couchbase TTLs to store a week or so, >> however we'd like to have online searching/access to a multiple month (i >> hope by utilizing ES) and I'm seeking the best Enterprise approach to this. >> >> However I haven't any clue for capacity planning of ES for 5, 6 billions >> of records. >> >>> >>> >>> >>> -- >>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com >>> <http://Elasticsearch.com>* >>> @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr >>> <https://twitter.com/elasticsearchfr> | @scrutmydocs >>> <https://twitter.com/scrutmydocs> >>> >>> >>> >>> Le 14 janv. 2015 à 14:51, Behrad <[email protected]> a écrit : >>> >>> I'll recheck elasticsearch-transport-couchbase again David, another two >>> related questions comes to my mind to ask, >>> >>> 1) Is there any best practice to replicate/sync/duplicate an existing >>> index to a new index by ES? >>> 2) Considering I have an existing index with data persist enabled on ES, >>> Can I use that to create a new index with a customer defined mapping on my >>> data on ES without the need of my primary database? (does sroll api help >>> with this?) >>> >>> 2015-01-14 17:00 GMT+03:30 David Pilato <[email protected]>: >>> >>>> May be you could use different mapping for the year index and the week >>>> index? >>>> One with ttl disabled and the other one with ttl enabled? >>>> >>>> >>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html >>>> >>>> -- >>>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com >>>> <http://elasticsearch.com/>* >>>> @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr >>>> <https://twitter.com/elasticsearchfr> | @scrutmydocs >>>> <https://twitter.com/scrutmydocs> >>>> >>>> >>>> >>>> Le 14 janv. 2015 à 14:21, Behrad <[email protected]> a écrit : >>>> >>>> I am actually using Couchbase ES plugin David, this is my database >>>> requirement that data has a weekly ttl, so when data is removed in >>>> Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES. >>>> >>>> 2015-01-14 16:43 GMT+03:30 David Pilato <[email protected]>: >>>> >>>>> Don't use ttl for that kind of usage but prefer time based indices. >>>>> Much more efficient. >>>>> >>>>> So the year index won't have any removal and you will end up to just >>>>> remove weekly indices which is super fast and doesn't generate a lot of IO >>>>> (but ttl does) >>>>> >>>>> HTH >>>>> >>>>> David >>>>> >>>>> Le 14 janv. 2015 à 14:01, Behrad Zari <[email protected]> a écrit : >>>>> >>>>> My data contains a weekly ttl so my ES index will loose older data, >>>>> however I want to have another accumulative large index containing a year >>>>> of my data... How can I instruct ES/sync my second index without affecting >>>>> ttls in the first index? ( I don't want deleted docs from first index to >>>>> be >>>>> deleted in the second) >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "elasticsearch" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr >>>>> <https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> --Behrad >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "elasticsearch" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr >>>> <https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> --Behrad >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr >>> <https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> --Behrad >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr >> <https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > --Behrad > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/9E76ABA8-E485-4568-94CA-534F88A00DC8%40pilato.fr > <https://groups.google.com/d/msgid/elasticsearch/9E76ABA8-E485-4568-94CA-534F88A00DC8%40pilato.fr?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- --Behrad -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFPQ35HeT9GotRPido6aZ%3D8665Ki%3D_uP%3DxT33gbkHovOA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
