Thanks for the information. To keep the indexes in near real time sync I'll
end up creating thousands of indexes every day alone with this method,
correct?
On Mon, Apr 21, 2014 at 1:37 PM, Andrew Selden <
andrew.sel...@elasticsearch.com> wrote:
> Hi,
>
> I can see how the documentation may not be as clear as it should be on
> this point. Let me try to explain. If you refer back to [1], you will see
> an example restore command:
>
> $ curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore" -d '{
> "indices": "index_1,index_2",
> "ignore_unavailable": "true",
> "include_global_state": false,
> "rename_pattern": "index_(.+)",
> "rename_replacement": "restored_index_$1"
> }'
>
> The important parts to notice are: rename_pattern and rename_replacement.
> What this means is that, for any index in the repository named with the
> prefix ‘index_’, it will be restored to the cluster under the new name
> ‘restored_index_$1’, where $1 is the name of the original index. So
> “index_1" gets restored to “restored_index_1", meanwhile you can keep
> index_1 open and servicing requests.
>
> The basic operational workflow is that you would perform the restore
> operation with a rename. Once the restore is complete, do some sanity
> checking to confirm that you have in fact restored what you intended to.
> Finally, to make “restored_index_1” live and take “index_1” offline without
> impacting production service, you would use an atomic rename alias command
> [2].
>
> This assumes you have been using aliases to manage your indices. For
> example, typically you would set up an alias to point to a ‘logical’ index
> to mask one or more ‘physical’ indices. This is highly recommended as it
> gives you the flexibility to do things like hot-swap restored/live indices
> as in the above example.
>
> Hope that helps.
>
> Andrew
>
> 1.
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
> 2.
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html
>
> On Apr 21, 2014, at 12:36 PM, Mohit Anchlia <mohitanch...@gmail.com>
> wrote:
>
> This document seem to suggest that to do the restore elasticsearch need to
> close the index, which means index is not available during that time while
> restore is occurring?
>
> On Mon, Apr 21, 2014 at 12:31 PM, Andrew Selden <
> andrew.sel...@elasticsearch.com> wrote:
>
>> Hi,
>>
>> Snapshot/restore is fine to run on an active, production system. The
>> indices are still available for normal indexing and searching operations.
>> You can throttle the snapshot operation if you are concerned about
>> impacting production quality of service [1].
>>
>> I would suggest avoiding using scroll as a backup strategy. Keeping a
>> scroll open for a long time can be taxing on an actively used production
>> cluster. You are much better off using the snapshot/restore functionality.
>>
>> Andrew
>>
>> 1.
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html
>>
>>
>> On Apr 21, 2014, at 9:23 AM, Mohit Anchlia <mohitanch...@gmail.com>
>> wrote:
>>
>> Our data centers are a actively used and need to be highly available.
>> Does snapshot and restore happen in the background and indexes are still
>> available to the clients for read/write?
>>
>> Can Scroll be used to read indexes continuously and insert/update them?
>>
>> On Mon, Apr 21, 2014 at 5:57 AM, Alexander Reelsen <a...@spinscale.de>wrote:
>>
>>> hey,
>>>
>>> snapshot and restore might be an option, if you are willing to accept a
>>> lag a few minutes behind. another option might be to index/update/delete
>>> all your data into two different clusters, so you dont need replication at
>>> all?
>>>
>>>
>>> --Alex
>>>
>>>
>>> On Fri, Apr 18, 2014 at 10:19 PM, Mohit Anchlia
>>> <mohitanch...@gmail.com>wrote:
>>>
>>>> As I understand there is currently no feature that does async
>>>> replication between 2 clusters or even within the same cluster, but we have
>>>> a need to write one. What would be the best way to do it in elasticsearch?
>>>> I was thinking of leveraging Scroll for this.
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "elasticsearch" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrUdDdiqy62yHaaS6bJJ08_txDCNNXR8rGr%3DRGY8gAv-Q%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAOT3TWrUdDdiqy62yHaaS6bJJ08_txDCNNXR8rGr%3DRGY8gAv-Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearch+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9G8539fUmrzLWgkXoSYAEyUYkjUbyjeBiNVhBdarWTFA%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9G8539fUmrzLWgkXoSYAEyUYkjUbyjeBiNVhBdarWTFA%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpqVdoSWv552vh3ZOL7siBiMUCGRUV5NcBwAFoWPisVLQ%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAOT3TWpqVdoSWv552vh3ZOL7siBiMUCGRUV5NcBwAFoWPisVLQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearch+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/B852C0E8-A2BE-4C23-B14A-EE2593A4D7ED%40elasticsearch.com<https://groups.google.com/d/msgid/elasticsearch/B852C0E8-A2BE-4C23-B14A-EE2593A4D7ED%40elasticsearch.com?utm_medium=email&utm_source=footer>
>> .
>>
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAOT3TWr14cNv05GAjoxtXVtUY4-uPzaxt200tp_spDNXvO_tyg%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAOT3TWr14cNv05GAjoxtXVtUY4-uPzaxt200tp_spDNXvO_tyg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/C9E4589A-FBCB-4FC3-9492-35B8A722EB20%40elasticsearch.com<https://groups.google.com/d/msgid/elasticsearch/C9E4589A-FBCB-4FC3-9492-35B8A722EB20%40elasticsearch.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAOT3TWqY0dYmnJtSbReCqaBwXe6Bj9iUyEqPXB0RBXzDiX7UUQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
