First of all, thank you for building ElasticSearch. It is a truly awesome
product.
I am trying to use the "User data flow". For this, I create a single index
and multiple aliases inside of it. In my use case, I have about 5mln
aliases to add.
The alias structure roughly looks like this:
{
'index' : 'index_name',
'alias' : 'user_' + user_id,
'filter' : {
'term' : {
'user' : user_id,
},
),
'routing' => 'r' + user_id,
}
I create a server with this setup:
{
"index_name" : {
"settings" : {
"index.number_of_replicas" : "1",
"index.number_of_shards" : "100",
}
}
}
Adding aliases works reasonably well for up to about 100k aliases, but it
slows down for later updates.
The following timings are shown after creating an index and then adding
aliases. No other operations were performed during that time on the
cluster and index.
These are the times needed to send and add aliases in batches of 5000:
batch: 5000 - time: 2311ms
batch: 5000 - time: 4096ms
batch: 5000 - time: 6022ms
batch: 5000 - time: 8127ms
batch: 5000 - time: 10174ms
batch: 5000 - time: 11403ms
batch: 5000 - time: 13126ms
batch: 5000 - time: 14335ms
batch: 5000 - time: 16500ms
batch: 5000 - time: 20663ms
batch: 5000 - time: 23002ms
batch: 5000 - time: 24457ms
batch: 5000 - time: 26375ms
batch: 5000 - time: 28984ms
batch: 5000 - time: 30559ms
batch: 5000 - time: 32234ms
batch: 5000 - time: 35098ms
batch: 5000 - time: 38922ms
batch: 5000 - time: 41776ms
batch: 5000 - time: 53402ms
batch: 5000 - time: 58600ms
batch: 5000 - time: 65567ms
batch: 5000 - time: 79885ms
batch: 5000 - time: 89900ms
batch: 5000 - time: 89368ms
batch: 5000 - time: 104109ms
As you can see, it gradually slows down. Is this expected? Looks like the
addition time grows linearly to the amount of aliases. Is that correct?
Thanks!
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d8a03b09-9ed2-49c7-9ca8-a2285478d933%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
