Thanks everyone for replying. As it turns out, all our problems stemmed
from our index schema.
Since our app was heavily modeled after social networks, we had to store
our users' followers, and their IDs. To do that, each of our users had an
array called "follower_ids" -- the IDs of the people who are following our
user. Now, that's all fine, until a user like the NBA, or Pepsi comes in
with millions and millions of followers, and that array turns into an
immensely giant array. We also tried turning the array into a nested object
of [{id: 1,}, {id: 2}, ...], but because of the indexing strategy, that
ended up even worse.
We've pinpointed the problem to the part in which we add IDs to the
follower_ids array. Ultimately, we swapped the schema around -- instead of
storing a giant "follower_ids", we started storing "following_ids" --
meaning, each person's document stores which users they follow.
Our current schema works great! CPU never goes above 25%, OS Load stays
consistent, and our cluster is functioning super fast.
On Sunday, August 3, 2014 2:55:21 AM UTC-4, smonasco wrote:
>
> Also is it always the master node that goes awry?
> On Aug 3, 2014 12:54 AM, "smonasco" <[email protected] <javascript:>>
> wrote:
>
>> What jumps out at me is your that the CPU work you're doing seems to be
>> very index related, your garbage collections are trying hard on the errant
>> machine and not getting anywhere and you have a lot of deleted docs.
>>
>> Tell us about your indexing strategy? Tell us things like routing, how
>> bursty it is and maybe why you have so many deleted docs.
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "elasticsearch" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/elasticsearch/e1RBjvFSKGU/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> [email protected] <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/2a79ccc0-d2cd-4d27-b66e-715d709c838b%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eecf7019-3ecc-4ad2-805b-a6c16cdd2122%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
