Currently this is ran on 4 nodes, with the possibility of adding new nodes.
I get nothing in the logs.
I don't completely understand this for my use case:
https://github.com/elasticsearch/elasticsearch/pull/5180 .
Ideally what I would like, would be not to create so many aliases, but to
have a template doing the work for me. The template could hold the data
pertaining the filtering and routing. The template could be very simple,
for a request:
host:9200/user_{id} => this would automatically match the template:
"user_*" and use its options. Also, this would very much simplify my work
later on. As the server is alive and a new user would appear, the template
would automatically use the templates settings, instead of me checking if
the alias exists and then adding the alias.
This would allow me to create 1 template instead of so many similar
aliases. Or maybe this is already implemented?
Thanks!
W dniu czwartek, 6 marca 2014 13:49:29 UTC-8 użytkownik David Pilato
napisał:
>
> I have never seen that number of aliases. That means you have 5 million
> users?
> Nice project ;-)
>
> I guess here that the cluster state is getting so big that it takes more
> and more time to update it and copy it to all nodes.
>
> BTW how many nodes you have for those 200 shards?
>
> Do you see anything in logs?
>
> Thinking it loud.
> Wondering if creating some alias template could help here to minimize the
> cluster state size?
> Something like what you exactly describe:
> {
> 'index' : 'index_name',
> 'alias' : 'user_{user_id}',
> 'filter' : {
> 'term' : {
> 'user' : '{user_id}',
> },
> ),
> 'routing' => 'r{user_id}'
> }
>
> It looks somehow similar to what Luca just did with
> https://github.com/elasticsearch/elasticsearch/pull/5180
>
> Someone else has an idea?
>
> --
> David ;-)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
>
> Le 6 mars 2014 à 22:15, Michał Zgliczyński <[email protected] <javascript:>>
> a écrit :
>
> First of all, thank you for building ElasticSearch. It is a truly awesome
> product.
>
> I am trying to use the "User data flow". For this, I create a single index
> and multiple aliases inside of it. In my use case, I have about 5mln
> aliases to add.
>
> The alias structure roughly looks like this:
> {
> 'index' : 'index_name',
> 'alias' : 'user_' + user_id,
> 'filter' : {
> 'term' : {
> 'user' : user_id,
> },
> ),
> 'routing' => 'r' + user_id,
> }
>
> I create a server with this setup:
> {
> "index_name" : {
> "settings" : {
> "index.number_of_replicas" : "1",
> "index.number_of_shards" : "100",
> }
> }
> }
>
>
> Adding aliases works reasonably well for up to about 100k aliases, but it
> slows down for later updates.
>
> The following timings are shown after creating an index and then adding
> aliases. No other operations were performed during that time on the
> cluster and index.
> These are the times needed to send and add aliases in batches of 5000:
> batch: 5000 - time: 2311ms
> batch: 5000 - time: 4096ms
> batch: 5000 - time: 6022ms
> batch: 5000 - time: 8127ms
> batch: 5000 - time: 10174ms
> batch: 5000 - time: 11403ms
> batch: 5000 - time: 13126ms
> batch: 5000 - time: 14335ms
> batch: 5000 - time: 16500ms
> batch: 5000 - time: 20663ms
> batch: 5000 - time: 23002ms
> batch: 5000 - time: 24457ms
> batch: 5000 - time: 26375ms
> batch: 5000 - time: 28984ms
> batch: 5000 - time: 30559ms
> batch: 5000 - time: 32234ms
> batch: 5000 - time: 35098ms
> batch: 5000 - time: 38922ms
> batch: 5000 - time: 41776ms
> batch: 5000 - time: 53402ms
> batch: 5000 - time: 58600ms
> batch: 5000 - time: 65567ms
> batch: 5000 - time: 79885ms
> batch: 5000 - time: 89900ms
> batch: 5000 - time: 89368ms
> batch: 5000 - time: 104109ms
>
> As you can see, it gradually slows down. Is this expected? Looks like the
> addition time grows linearly to the amount of aliases. Is that correct?
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected] <javascript:>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/d8a03b09-9ed2-49c7-9ca8-a2285478d933%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/d8a03b09-9ed2-49c7-9ca8-a2285478d933%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/de1e8dce-a559-4c3a-98c1-e87a5eed46c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
