Try upgrading Elasticsearch -- it's up to 6.0 release just a few week ago now -- its (and Lucene's) memory usage has decreased over time.
The _uid field in particular will always be costly, unfortunately. Since it's a primary key, every term will be unique, and the term index has to work hard to store all the prefixes for those keys. Mike McCandless http://blog.mikemccandless.com On Sat, Dec 2, 2017 at 9:42 PM, Bingtao Yin <ybts...@gmail.com> wrote: > Thanks, mike. I'm facing a similar problem. > I'm running a 2.0 elasticsearch cluster, and find the fst of _uid field > takes a lot of memory. The _uid field is not analyzed and generated by > elasticsearch, which also has high cardinality. > Is there any ways to reduce memory cost for _uid field? Thanks. > > > 2017-11-29 5:47 GMT+08:00 elirev <elirevac...@gmail.com>: > > > Thanks Mike . > > I did not find any clear way to know it its FST or Norm , or > something > > else ( unless i miss something ) the fact the FST is an in memory prefix > > index lead me to think it using most of the heap . > > Our mapping is normal with around of 200 columns one of the columns is > > nested object with limited amount of objects (up to 4 instances ) , we > > are using monthly base indexes (keep 6 month open ) . In last month i > see > > dramatic extra allocation on the segment memory (around 30% where in > > regulare month is around 5%) , the only change i see is that the > nested > > object is now include avg 8 instances ) , this increases the amount of > > the hidden document we have now on the index (about more then twice) . > > When we optimize the index the amount of allocation memory was reduced > (we > > see it only after rolling restart the nodes ) . > > > > If you don't mind i have few question : > > 1) Do you know about an way to figure out which component is taking > all > > this memory . > > 2) Do you see relation between the fact that the nested objects was > > increases to the extra memory allocation we have ? > > 3) Did FST memory usage is impacted by the fact we optimize the > > problematic > > index and why we see it only after restarting ES service > > > > Thanks mike > > > > . > > > > > > > > > > > > > > -- > > Sent from: http://lucene.472066.n3.nabble.com/Lucene-Java-Users- > > f532864.html > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > >
