I see. Just wondering why you are sending things to couchbase then. Why not using Redis or RabbitMQ or any message queue System. I mean that it sounds like you are using couchbase as a queue for elasticsearch.
Or if you want a Backup in couchbase, could you imagine sending your data from your application to couchbase and elasticsearch at the same time? I think you are trying to twist here the couchbase plugin which is designed to keep the database in sync with elasticsearch which is obviously not what you are looking for. My 2 cents. David > Le 14 janv. 2015 à 18:18, Behrad <[email protected]> a écrit : > > > > 2015-01-14 18:25 GMT+03:30 David Pilato <[email protected]>: >> Definitely scan and scroll API is for that. >> Some clients provide that feature out of the box. >> >> That said, reading from couchbase is something super easy. So I guess you >> could may be replicate your data from couchbase twice using 2 XDCR configs >> one for the weekly index and the other one for the year index. Is that >> something doable? > > Unfortunately not David, XDCR blinding syncs the source db with remotes (any > deleted data will also be deleted from the remote) and we can't let the > source db to grow (about 1.3 billion records/month) so we are using > couchbase TTLs to store a week or so, > however we'd like to have online searching/access to a multiple month (i hope > by utilizing ES) and I'm seeking the best Enterprise approach to this. > > However I haven't any clue for capacity planning of ES for 5, 6 billions of > records. >> >> >> >> -- >> David Pilato | Technical Advocate | Elasticsearch.com >> @dadoonet | @elasticsearchfr | @scrutmydocs >> >> >> >>> Le 14 janv. 2015 à 14:51, Behrad <[email protected]> a écrit : >>> >>> I'll recheck elasticsearch-transport-couchbase again David, another two >>> related questions comes to my mind to ask, >>> >>> 1) Is there any best practice to replicate/sync/duplicate an existing index >>> to a new index by ES? >>> 2) Considering I have an existing index with data persist enabled on ES, >>> Can I use that to create a new index with a customer defined mapping on my >>> data on ES without the need of my primary database? (does sroll api help >>> with this?) >>> >>> 2015-01-14 17:00 GMT+03:30 David Pilato <[email protected]>: >>>> May be you could use different mapping for the year index and the week >>>> index? >>>> One with ttl disabled and the other one with ttl enabled? >>>> >>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html >>>> >>>> -- >>>> David Pilato | Technical Advocate | Elasticsearch.com >>>> @dadoonet | @elasticsearchfr | @scrutmydocs >>>> >>>> >>>> >>>>> Le 14 janv. 2015 à 14:21, Behrad <[email protected]> a écrit : >>>>> >>>>> I am actually using Couchbase ES plugin David, this is my database >>>>> requirement that data has a weekly ttl, so when data is removed in >>>>> Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES. >>>>> >>>>> 2015-01-14 16:43 GMT+03:30 David Pilato <[email protected]>: >>>>>> Don't use ttl for that kind of usage but prefer time based indices. >>>>>> Much more efficient. >>>>>> >>>>>> So the year index won't have any removal and you will end up to just >>>>>> remove weekly indices which is super fast and doesn't generate a lot of >>>>>> IO (but ttl does) >>>>>> >>>>>> HTH >>>>>> >>>>>> David >>>>>> >>>>>>> Le 14 janv. 2015 à 14:01, Behrad Zari <[email protected]> a écrit : >>>>>>> >>>>>>> My data contains a weekly ttl so my ES index will loose older data, >>>>>>> however I want to have another accumulative large index containing a >>>>>>> year of my data... How can I instruct ES/sync my second index without >>>>>>> affecting ttls in the first index? ( I don't want deleted docs from >>>>>>> first index to be deleted in the second) >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "elasticsearch" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>> an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in the >>>>>> Google Groups "elasticsearch" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr. >>>>>> >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> >>>>> -- >>>>> --Behrad >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "elasticsearch" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr. >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >>> -- >>> --Behrad >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com. >>> For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr. >> >> For more options, visit https://groups.google.com/d/optout. > > > > -- > --Behrad > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr. For more options, visit https://groups.google.com/d/optout.
