HI Richard, it certainly does seem like there is a 10000 item limit in play here. I don't know enough about elastic search to give you good guidance on why that is, but hopefully other forum readers can help out.
One thing that I don't believe is in the documentation right now (though it would be good to add it) is that your arches elasticsearch installation is automatically setup up with a head plugin, so you can view your current elastic search instance. go to localhost:9200/_plugin/head, and you'll get an in depth overview of the everything that is going on with elasticsearch. I don't know if you can manage limitations there, but it's definitely worth exploring. After reading a bit of this article <https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index>, and looking at the elasticsearch config files (located in my_hip_app/my_hip_app/elasticsearch/elasticsearch-1.4.1/config) that are included in your app by default, the first thing I'd try is increasing the number of shards, and seeing if that helps at all. At any rate, I'm guessing that your problem will be fixed with a bit of configuration optimization for the high volume of resources you have. Looking forward to hearing about your progress... On Monday, December 7, 2015 at 10:44:43 AM UTC-6, Richard Jennings wrote: > > Hi All, > > I've encountered two problems which I would appreciate some guidance: > > 1) Elasticsearch is not detecting all of the records in our database. For > instance, we know we have 500 buildings in the database but only 300 come > up when searching for them. This is the same across the board for all > nodes. I'm fairly certain that whenever I bulk-loaded data that > Elasticsearch was always running. Is there a way to re-index our data so > that they all show up without having to do a full reinstall, as I don't > wish to export the data out at this stage? > > 2) I've just re-booted arches and have now discovered that Elasticsearch > is not working properly for another reason. Previously I was able to > generate saved searches but now it no longer gives a running log of > activity in the database, which means I can no longer copy and past > searches and put them into the popular searches. It has paused at the > position in the copied text below: > > [2015-12-07 15:27:23,268][WARN ][index.mapper ] [Derrick > Slegers Speed] [entity] Type [HISTORICAL_EVENT.E5] contains a '.', it is > recommended not to include it within a type name > > [2015-12-07 15:27:23,472][INFO ][gateway ] [Derrick > Slegers Speed] recovered [6] indices into cluster_state > > [2015-12-07 15:27:23,624][WARN ][index.mapper ] [Derrick > Slegers Speed] [resource] Type [HERITAGE_RESOURCE_GROUP.E27] contains a > '.', it is recommended not to include it within a type name > > > When I was able to do this before the problem of the non-indexed data was > already present. > > I'm not sure if either of these issues relate to an earlier issue of mine > concerning memory, which I am currently looking in to: > > > https://groups.google.com/forum/#!searchin/archesproject/elasticsearch%7Csort:date/archesproject/ZWLvadW3Szc/nrRly3BJEwAJ > > (I have not had the chance to upgrade yet to the latest versions of Arches > 3 and HIP as I need to keep the current version of our database live for > the time being) > > Any help with these issues is much appreciated! > > Best wishes > > Richard > -- -- To post, send email to [email protected]. To unsubscribe, send email to [email protected]. For more information, visit https://groups.google.com/d/forum/archesproject?hl=en --- You received this message because you are subscribed to the Google Groups "Arches Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
