This is our persistent storage, but the working set is a multiple days, older data only is needed for search and audits, the whole data is being backed up and archived by a separate process, so we don't need extra large hard disks (each billion records gets to 600GB) Redis is not designed for persistency and Message Queues are slow when persistency is envolved, if we were using a Message Queue, we also should have persist our data on a durable external storage. I want to have a small working set of data of a week, and a workhouse of multi months of past data. (I can't see why to keep multi billions of passive data in Couchbase which should be searched by ES!?)
2015-01-14 21:02 GMT+03:30 David Pilato <[email protected]>: > I see. Just wondering why you are sending things to couchbase then. Why > not using Redis or RabbitMQ or any message queue System. I mean that it > sounds like you are using couchbase as a queue for elasticsearch. > > Or if you want a Backup in couchbase, could you imagine sending your data > from your application to couchbase and elasticsearch at the same time? > > I think you are trying to twist here the couchbase plugin which is > designed to keep the database in sync with elasticsearch which is obviously > not what you are looking for. > > My 2 cents. > > David > > Le 14 janv. 2015 à 18:18, Behrad <[email protected]> a écrit : > > > > 2015-01-14 18:25 GMT+03:30 David Pilato <[email protected]>: > >> Definitely scan and scroll API is for that. >> Some clients provide that feature out of the box. >> >> That said, reading from couchbase is something super easy. So I guess you >> could may be replicate your data from couchbase twice using 2 XDCR configs >> one for the weekly index and the other one for the year index. Is that >> something doable? >> > > Unfortunately not David, XDCR blinding syncs the source db with remotes > (any deleted data will also be deleted from the remote) and we can't let > the source db to grow (about 1.3 billion records > /month) so we are using couchbase TTLs to store a week or so, > however we'd like to have online searching/access to a multiple month (i > hope by utilizing ES) and I'm seeking the best Enterprise approach to this. > > However I haven't any clue for capacity planning of ES for 5, 6 billions > of records. > >> >> >> >> -- >> *David Pilato* | *Technical Advocate* | *Elasticsearch.com >> <http://Elasticsearch.com>* >> @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr >> <https://twitter.com/elasticsearchfr> | @scrutmydocs >> <https://twitter.com/scrutmydocs> >> >> >> >> Le 14 janv. 2015 à 14:51, Behrad <[email protected]> a écrit : >> >> I'll recheck elasticsearch-transport-couchbase again David, another two >> related questions comes to my mind to ask, >> >> 1) Is there any best practice to replicate/sync/duplicate an existing >> index to a new index by ES? >> 2) Considering I have an existing index with data persist enabled on ES, >> Can I use that to create a new index with a customer defined mapping on my >> data on ES without the need of my primary database? (does sroll api help >> with this?) >> >> 2015-01-14 17:00 GMT+03:30 David Pilato <[email protected]>: >> >>> May be you could use different mapping for the year index and the week >>> index? >>> One with ttl disabled and the other one with ttl enabled? >>> >>> >>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-ttl-field.html >>> >>> -- >>> *David Pilato* | *Technical Advocate* | *Elasticsearch.com >>> <http://elasticsearch.com/>* >>> @dadoonet <https://twitter.com/dadoonet> | @elasticsearchfr >>> <https://twitter.com/elasticsearchfr> | @scrutmydocs >>> <https://twitter.com/scrutmydocs> >>> >>> >>> >>> Le 14 janv. 2015 à 14:21, Behrad <[email protected]> a écrit : >>> >>> I am actually using Couchbase ES plugin David, this is my database >>> requirement that data has a weekly ttl, so when data is removed in >>> Couchbase, ES plugin also removes it (using Couchbase's XDCR) within ES. >>> >>> 2015-01-14 16:43 GMT+03:30 David Pilato <[email protected]>: >>> >>>> Don't use ttl for that kind of usage but prefer time based indices. >>>> Much more efficient. >>>> >>>> So the year index won't have any removal and you will end up to just >>>> remove weekly indices which is super fast and doesn't generate a lot of IO >>>> (but ttl does) >>>> >>>> HTH >>>> >>>> David >>>> >>>> Le 14 janv. 2015 à 14:01, Behrad Zari <[email protected]> a écrit : >>>> >>>> My data contains a weekly ttl so my ES index will loose older data, >>>> however I want to have another accumulative large index containing a year >>>> of my data... How can I instruct ES/sync my second index without affecting >>>> ttls in the first index? ( I don't want deleted docs from first index to be >>>> deleted in the second) >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com >>>> <https://groups.google.com/d/msgid/elasticsearch/08f4979d-ccd4-431a-96b9-3e2911c9ae80%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "elasticsearch" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr >>>> <https://groups.google.com/d/msgid/elasticsearch/48E15A95-E2E6-4347-9040-A7789466A378%40pilato.fr?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> --Behrad >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CAJFrgeH_NJHp7rJWNXppQzASw6E8Xsh9trxshFaXows%2Bpr3BjQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr >>> <https://groups.google.com/d/msgid/elasticsearch/EBBC7F7B-F82F-401B-898B-84C76D6F5F80%40pilato.fr?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> --Behrad >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAJFrgeEAoSg_ppRreOpOchx9By3ezD%2BYT06VR84FJ4jMs6PmUg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr >> <https://groups.google.com/d/msgid/elasticsearch/884A1E22-F487-4487-AC86-82FA757E171D%40pilato.fr?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > --Behrad > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAJFrgeFLfY6vhMZvfrBbB2NHau_c%3DyZHiobP97oyUocKeoCyjQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/9Spk2d84FLk/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr > <https://groups.google.com/d/msgid/elasticsearch/D3C7950E-0B3C-46B8-9CB0-7D293FDC6990%40pilato.fr?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- --Behrad -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJFrgeG5aevQ5N_bh8obo-tg62RTSnQP91BEb%3Dd8Ab6pxnjk%3Dg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
