Hi Adam, Thanks very for your speedy reply!
This is what I got from running python manage.py packages -o index_database : operation: index_database package: eamena deleting index : resource_relations deleting index : entity indexes cleared entitytypes indexed creating term index indexing concepts concepts indexed indexing concepts required 172 seconds Concept Index Results: Status: failed, In Database: 2426, Indexed: 2422 Indexing 0 ACTIVITY.E7 resources Indexing 0 ACTOR.E39 resources Indexing 0 HERITAGE_RESOURCE.E18 resources Indexing 83879 HERITAGE_RESOURCE_GROUP.E27 resources Domains matching query does not exist. : 25 Indexing 0 HISTORICAL_EVENT.E5 resources Indexing 0 INFORMATION_RESOURCE.E73 resources Resource Index Results: Status: failed, Resource Type: HERITAGE_RESOURCE_GROUP.E27, In Database: 83879, Indexed: 10000 Status: Passed, Resource Type: ACTIVITY.E7, In Database: 0, Indexed: 0 Status: Passed, Resource Type: ACTOR.E39, In Database: 0, Indexed: 0 Status: Passed, Resource Type: INFORMATION_RESOURCE.E73, In Database: 0, Indexed: 0 Status: Passed, Resource Type: HERITAGE_RESOURCE.E18, In Database: 0, Indexed: 0 Status: Passed, Resource Type: HISTORICAL_EVENT.E5, In Database: 0, Indexed: 0 Running this has un-frozen elasticsearch and now all resource types show - thank you! There is still a problem with the indexing. I've been making quite a few customisations to HERITAGE_RESOURCE_GROUP.E27 to suit the needs of our project. We have so far only been using this resource type to record our data. I read through your recent post where you changed the name of the resource type id, but I retained the names in ours. Is there any significance that it has indexed exactly 10000 of our 83879 records before failing? Is there a maximum setting default that is set to 10000 that needs changing somewhere? With regard to what I mentioned about saved searches, I read through your documentation again and it was of course by copying and pasting the URL and not what appeared in the elasticsearch terminal that enabled the completion of the popular searches, not from what appeared in the logs in the terminal windows. Sorry for the confusion there. What I have noticed, however, is that the logs that previously displayed in the terminals now do not. This is perhaps not related to the indexing as this has worked in the past when the indexing has not, but it is something else to explore. Any further advice on what settings to change to get the index working again is welcome. I uploaded data via .arches files for a few entity ids, with the idea that we would enter in the remaining data via the arches front-end forms. I did the upload process piecemeal and used more than a dozen .arches files. Sometimes in the batches uploaded there was accidental duplication of a few records. I mention all of this in case the manner the data was entered has caused the problem. Many thanks, Richard On Monday, December 7, 2015 at 8:19:37 PM UTC, Adam Cox wrote: > > Hi Richard, I don't have a complete answer, but hopefully this will be of > some help. > > 1) you can run python manage.py packages -o index_database to recreate > the indices. It will not delete existing database entries. Not sure it'll > solve the problem, but worth trying. > > 2) from the message you posted, it just looks like elastic search has > frozen up during it's initialization process. The message you printed are > the correct output that you will always see when you start elasticsearch, > but it should list through all resource types, not just the first two. > Unless... have you modified the names of any of the resources? As per this > post <https://groups.google.com/forum/#!topic/archesproject/n4k9kB0KGws> > I made a little while ago, things are a little tricky with elasticsearch > indexing, so there could be something there. At any rate, the memory issue > you had before definitely could be a factor, because maybe elastic search > just froze from lack of resources. > > Good luck! > Adam > > On Monday, December 7, 2015 at 10:44:43 AM UTC-6, Richard Jennings wrote: >> >> Hi All, >> >> I've encountered two problems which I would appreciate some guidance: >> >> 1) Elasticsearch is not detecting all of the records in our database. For >> instance, we know we have 500 buildings in the database but only 300 come >> up when searching for them. This is the same across the board for all >> nodes. I'm fairly certain that whenever I bulk-loaded data that >> Elasticsearch was always running. Is there a way to re-index our data so >> that they all show up without having to do a full reinstall, as I don't >> wish to export the data out at this stage? >> >> 2) I've just re-booted arches and have now discovered that Elasticsearch >> is not working properly for another reason. Previously I was able to >> generate saved searches but now it no longer gives a running log of >> activity in the database, which means I can no longer copy and past >> searches and put them into the popular searches. It has paused at the >> position in the copied text below: >> >> [2015-12-07 15:27:23,268][WARN ][index.mapper ] [Derrick >> Slegers Speed] [entity] Type [HISTORICAL_EVENT.E5] contains a '.', it is >> recommended not to include it within a type name >> >> [2015-12-07 15:27:23,472][INFO ][gateway ] [Derrick >> Slegers Speed] recovered [6] indices into cluster_state >> >> [2015-12-07 15:27:23,624][WARN ][index.mapper ] [Derrick >> Slegers Speed] [resource] Type [HERITAGE_RESOURCE_GROUP.E27] contains a >> '.', it is recommended not to include it within a type name >> >> >> When I was able to do this before the problem of the non-indexed data was >> already present. >> >> I'm not sure if either of these issues relate to an earlier issue of mine >> concerning memory, which I am currently looking in to: >> >> >> https://groups.google.com/forum/#!searchin/archesproject/elasticsearch%7Csort:date/archesproject/ZWLvadW3Szc/nrRly3BJEwAJ >> >> (I have not had the chance to upgrade yet to the latest versions of >> Arches 3 and HIP as I need to keep the current version of our database live >> for the time being) >> >> Any help with these issues is much appreciated! >> >> Best wishes >> >> Richard >> > -- -- To post, send email to [email protected]. To unsubscribe, send email to [email protected]. For more information, visit https://groups.google.com/d/forum/archesproject?hl=en --- You received this message because you are subscribed to the Google Groups "Arches Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
