Hi Joel,
Looking at the error, it looks like you have a record in the database that
doesn't exist in the entity index. I'm not sure how the index got into
that state. It would be nice to do a little more digging.
Using the entityid in the error, can you find the report for that resource.
It should be located at:
http://localhost:8000/reports/974edbec-5664-4151-9018-a23216245478
If the report comes up successfully, maybe you can go to the form and
"re-save" it (which should trigger a re-index of the resource). The form
should be found at http://localhost:8000/entity/{you'll have to find out
the entitytypeid from the report}/
<http://localhost:9200/entity/HERITAGE_RESOURCE.E18/5a7bf49a-1541-433a-95c9-07e4c98dacd6>
974edbec-5664-4151-9018-a23216245478
<http://localhost:8000/reports/974edbec-5664-4151-9018-a23216245478> (make
sure to login).
If you can re-save it, you can try running the patch again. Assuming that
was the only issue, then you should be set.
If you see the issue again with a different resource, I've attached a new
patch file that you can run instead. Simply place the file into the
location where you pip installed arches (and then under
management/patches). Then run the command referencing the new patch.
It'll print out a list of resource ids that it can't find in the index.
Hopefully there aren't many.
I'm still a little perplexed as to how your index got out of sync. Do you
have any ideas of how that might have happened? Was elasticsearch not
running while you were adding resources via the ui? Did you happen to
delete a document from elasticsearch directly?
Please let us know, because if there's a bug I'd like to fix it!
Cheers,
Alexei
<http://localhost:8000/reports/974edbec-5664-4151-9018-a23216245478>On
Thursday, November 19, 2015 at 3:39:29 AM UTC-8, Joel Aldor wrote:
> Hello Alexei,
>
> Tried to apply the patch but it gives me this error:
>
> (ENV2)ubuntu@ip-172-31-28-36:~/Projects/phm2app$ python manage.py
> install_patch -s upgrade_to_v3_1_1
>
> INSTALLING PYSHP MODULE
> -----------------------
> You are using pip version 7.1.0, however version 7.1.2 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
> Requirement already satisfied (use --upgrade to upgrade): pyshp in
> /home/ubuntu/Projects/ENV2/lib/python2.7/site-packages
>
> UPDATING ENTITY INDEX
> ---------------------
> GET http://localhost:9200/entity/_all/5a7bf49a-1541-433a-95c9-07e4c98dacd6
> [status:200 request:0.002s]
> No handlers could be found for logger "elasticsearch.trace"
> PUT
> http://localhost:9200/entity/HERITAGE_RESOURCE.E18/5a7bf49a-1541-433a-95c9-07e4c98dacd6
>
> [status:200 request:0.040s]
> GET /entity/_all/974edbec-5664-4151-9018-a23216245478 [status:404
> request:0.001s]
> Traceback (most recent call last):
> File "manage.py", line 28, in <module>
> execute_from_command_line(sys.argv)
> File
> "/home/ubuntu/Projects/ENV2/local/lib/python2.7/site-packages/django/core/management/__init__.py",
>
> line 399, in execute_from_command_line
> utility.execute()
> File
> "/home/ubuntu/Projects/ENV2/local/lib/python2.7/site-packages/django/core/management/__init__.py",
>
> line 392, in execute
> self.fetch_command(subcommand).run_from_argv(self.argv)
> File
> "/home/ubuntu/Projects/ENV2/local/lib/python2.7/site-packages/django/core/management/base.py",
>
> line 242, in run_from_argv
> self.execute(*args, **options.__dict__)
> File
> "/home/ubuntu/Projects/ENV2/local/lib/python2.7/site-packages/django/core/management/base.py",
>
> line 285, in execute
> output = self.handle(*args, **options)
> File
> "/home/ubuntu/Projects/ENV2/local/lib/python2.7/site-packages/arches/management/commands/install_patch.py",
>
> line 39, in handle
> self.run_patch_file(options['source'])
> File
> "/home/ubuntu/Projects/ENV2/local/lib/python2.7/site-packages/arches/management/commands/install_patch.py",
>
> line 49, in run_patch_file
> import_module('arches.management.patches.%s' % module_name)
> File
> "/home/ubuntu/Projects/ENV2/local/lib/python2.7/site-packages/django/utils/importlib.py",
>
> line 40, in import_module
> __import__(name)
> File
> "/home/ubuntu/Projects/ENV2/local/lib/python2.7/site-packages/arches/management/patches/upgrade_to_v3_1_1.py",
>
> line 187, in <module>
> main()
> File
> "/home/ubuntu/Projects/ENV2/local/lib/python2.7/site-packages/arches/management/patches/upgrade_to_v3_1_1.py",
>
> line 163, in main
> indexed_resource = se.search(index='entity', id=resourceid)
> File
> "/home/ubuntu/Projects/ENV2/local/lib/python2.7/site-packages/arches/app/search/search.py",
>
> line 85, in search
> return self.es.get(**kwargs)
> File
> "/home/ubuntu/Projects/ENV2/local/lib/python2.7/site-packages/elasticsearch/client/utils.py",
>
> line 69, in _wrapped
> return func(*args, params=params, **kwargs)
> File
> "/home/ubuntu/Projects/ENV2/local/lib/python2.7/site-packages/elasticsearch/client/__init__.py",
>
> line 319, in get
> params=params)
> File
> "/home/ubuntu/Projects/ENV2/local/lib/python2.7/site-packages/elasticsearch/transport.py",
>
> line 307, in perform_request
> status, headers, data = connection.perform_request(method, url,
> params, body, ignore=ignore, timeout=timeout)
> File
> "/home/ubuntu/Projects/ENV2/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py",
>
> line 89, in perform_request
> self._raise_error(response.status, raw_data)
> File
> "/home/ubuntu/Projects/ENV2/local/lib/python2.7/site-packages/elasticsearch/connection/base.py",
>
> line 105, in _raise_error
> raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code,
> error_message, additional_info)
> elasticsearch.exceptions.NotFoundError: TransportError(404,
> u'{"_index":"entity","_type":"_all","_id":"974edbec-5664-4151-9018-a23216245478","found":false}')
> (ENV2)ubuntu@ip-172-31-28-36:~/Projects/phm2app$
>
> Let me know how to proceed with this. Thank you.
>
> Regards,
>
> Joel
>
> On Saturday, November 14, 2015 at 9:07:26 AM UTC+8, Alexei Peters wrote:
>>
>> Hi Everyone,
>> The Arches team has been busy trying to improve Arches and fix a few bugs
>> as well.
>> Below you'll find a listing of all the changes that are included in the
>> latest releases.
>>
>> Users are encouraged to update at their earliest convenience. Completely
>> re-installing Arches and Arches-HIP is the easiest way to accomplish this.
>>
>> If you can't completely re-install Arches (because you have data in the
>> system that you want to preserve) then you'll need to upgrade by running
>> the following commands:
>>
>> pip install arches --upgrade
>> pip install arches-hip --upgrade
>>
>> After upgrading you'll need to apply a patch by running this command from
>> an activated environment (make sure Elasticsearch is running):
>>
>> (on linux) --> sudo python manage.py install_patch -s upgrade_to_v3_1_1
>>
>> (on windows) --> python manage.py install_patch -s upgrade_to_v3_1_1
>>
>>
>> Cheers,
>> The Arches Team
>>
>> Listing of changes in the latest releases:
>>
>> Arches 3.1.1
>>
>> - Added permissions change instructions to linux setup script
>> - Added related resources as csv to export
>> - Added date type and multipart features to export
>> - Specified the version of virtualenv to use because of issues
>> running the arches-app command with more recent versions on linux.
>> - Added logic to support loading resources with a hard-coded value
>> type conceptid.
>> - Added support for specifying a particular value_type for an export
>> column whenthe entity being exported is of domain type. Previously we
>> could
>> only specify a primary and secondary export column - if the entitytypeid
>> was not an e55.
>> - Enabled the importer to accept uuid formatted legacy ids as the
>> entityid for aresource
>> - Add sort order to default value types.
>> - Adding graph export tool
>> - Adding arches json exporter and importer
>> - Allow users to view related resources without being logged in
>> - Make sure get_context method always returns a valid concept
>> - Fix file upload during the save of a .arches file
>> - Fix JSON output of a concept
>> - Fix time filter to work with branch list mods
>> - Fix link color in lists
>> - Change authentication logic. Add anonymous user. Create read and
>> edit groups. Add anonymous and admin user to respective groups. Cascade
>> changes to templates.
>> - Clean logic in auth method to not have duplicative logic to the
>> setAnonymous user middleware
>> - Add unique constraint for legacyoids in concepts.concepts table.
>> Add more information to entity type has no business table error.
>> - Fix html error that reverted the map zoom and position updates in
>> the ui
>> - Add function to populate concepts legacyoid when adding concepts
>> from RDM.
>> - Trigger 'delete' event AFTER removing the branch (this fixes an
>> issue in the time-filter.js) where a delete wouldn't refresh the search
>> - Fixed bug that kept numbers from exporting to CSV.
>> - Fix a few missellings in the docs
>> - Fix bug where clicking a link to edit a form entry would cause that
>> entry to disappear and not be entered into the editing section of the
>> form,
>> depends on update of knockout library
>> - Add deprecation decorator, mark user_can_edit context processor as
>> deprecated
>>
>>
>> Hip 1.0.4
>>
>> - Update resource graphs
>> - Confirm we're using a string in a regex match expression
>> - Fix time filter to work with branch list mods in arches
>>
>> Lots of updates to the docs (thanks to Adam Cox!)
>>
>> - Add note on running ES as daemon, fix other grammar
>> - Add integrity section to reports
>> - Slight change in S3 bucket info
>> - New docs section on converting to .arches from MS Excel
>> - Add business table name for management_recommendation.e89 node in
>> heritage_resource graph.
>> - Add warning about S3 file deletion problem, number some existing
>> lists.
>> - Remake all tables straight from graph csvs, except Activity (to
>> retain the descriptions. addressing issue in forum with incorrect nodes
>> in
>> some tables
>> - Reformat/color nodes in all gephi files, export all graphs as pdfs
>> in letter/legal and A4/B4, add notes on export and record of colors used
>>
>>
--
-- To post, send email to [email protected]. To unsubscribe, send
email to [email protected]. For more information,
visit https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
from django.db import connection, transaction
from django.contrib.auth.models import User, Group
from arches.app.search.search_engine_factory import SearchEngineFactory
import pip
import time
from arches.app.utils.betterJSONSerializer import JSONSerializer, JSONDeserializer
def main():
sql = """ALTER TABLE concepts.concepts
ALTER COLUMN conceptid DROP DEFAULT;
ALTER TABLE concepts.concepts
ALTER COLUMN legacyoid SET NOT NULL;
ALTER TABLE concepts.concepts
DROP CONSTRAINT IF EXISTS unique_concepts_legacyoid;
ALTER TABLE concepts.concepts
ADD CONSTRAINT unique_concepts_legacyoid UNIQUE (legacyoid);
CREATE OR REPLACE VIEW ontology.vw_export_nodes AS
SELECT foo.assettype,
foo.node AS label,
(foo.assettype || ':'::text) || foo.node AS id,
foo.mergenodeid AS mergenode,
foo.businesstable AS businesstablename
FROM ( SELECT m.entitytypeidfrom AS assettype,
r.entitytypedomain AS node,
m.mergenodeid,
( SELECT entity_types.businesstablename
FROM data.entity_types
WHERE entity_types.entitytypeid = r.entitytypedomain) AS businesstable
FROM ontology.mapping_steps ms
JOIN ontology.mappings m ON m.mappingid = ms.mappingid
JOIN ontology.rules r ON r.ruleid = ms.ruleid
UNION
SELECT m.entitytypeidfrom,
r.entitytyperange AS node,
m.mergenodeid,
( SELECT entity_types.businesstablename
FROM data.entity_types
WHERE entity_types.entitytypeid = r.entitytyperange) AS businesstable
FROM ontology.mapping_steps ms
JOIN ontology.mappings m ON m.mappingid = ms.mappingid
JOIN ontology.rules r ON r.ruleid = ms.ruleid) foo
WHERE (foo.node <> ALL (ARRAY['ARCHES_RECORD.E31'::text, 'CREATION_EVENT.E65'::text, 'UPDATE_EVENT.E65'::text, 'COMPILER.E82'::text, 'COMPILER_PERSON.E21'::text, 'REFERENCE_NUMBER_(INTERNAL).E42'::text, 'TIME-SPAN_UPDATE_EVENT.E52'::text, 'TIME-SPAN_CREATION_EVENT.E52'::text, 'DATE_OF_COMPILATION.E50'::text, 'DATE_OF_LAST_UPDATE.E50'::text])) AND foo.node <> foo.assettype
ORDER BY foo.assettype, foo.node;
ALTER TABLE ontology.vw_export_nodes
OWNER TO postgres;
CREATE OR REPLACE VIEW ontology.vw_export_edges AS
SELECT m.entitytypeidfrom AS assettype,
(m.entitytypeidfrom || ':'::text) || r.entitytypedomain AS source,
(m.entitytypeidfrom || ':'::text) || r.entitytyperange AS target,
r.propertyid AS label
FROM ontology.mapping_steps ms
JOIN ontology.mappings m ON m.mappingid = ms.mappingid
JOIN ontology.rules r ON r.ruleid = ms.ruleid
WHERE (m.entitytypeidfrom <> ALL (ARRAY['ARCHES_RECORD.E31'::text, 'CREATION_EVENT.E65'::text, 'UPDATE_EVENT.E65'::text, 'COMPILER.E82'::text, 'COMPILER_PERSON.E21'::text, 'REFERENCE_NUMBER_(INTERNAL).E42'::text, 'TIME-SPAN_UPDATE_EVENT.E52'::text, 'TIME-SPAN_CREATION_EVENT.E52'::text, 'DATE_OF_COMPILATION.E50'::text, 'DATE_OF_LAST_UPDATE.E50'::text])) AND (r.entitytypedomain <> ALL (ARRAY['ARCHES_RECORD.E31'::text, 'CREATION_EVENT.E65'::text, 'UPDATE_EVENT.E65'::text, 'COMPILER.E82'::text, 'COMPILER_PERSON.E21'::text, 'REFERENCE_NUMBER_(INTERNAL).E42'::text, 'TIME-SPAN_UPDATE_EVENT.E52'::text, 'TIME-SPAN_CREATION_EVENT.E52'::text, 'DATE_OF_COMPILATION.E50'::text, 'DATE_OF_LAST_UPDATE.E50'::text])) AND (r.entitytyperange <> ALL (ARRAY['ARCHES_RECORD.E31'::text, 'CREATION_EVENT.E65'::text, 'UPDATE_EVENT.E65'::text, 'COMPILER.E82'::text, 'COMPILER_PERSON.E21'::text, 'REFERENCE_NUMBER_(INTERNAL).E42'::text, 'TIME-SPAN_UPDATE_EVENT.E52'::text, 'TIME-SPAN_CREATION_EVENT.E52'::text, 'DATE_OF_COMPILATION.E50'::text, 'DATE_OF_LAST_UPDATE.E50'::text])) AND m.entitytypeidto = r.entitytyperange
ORDER BY m.entitytypeidfrom;
ALTER TABLE ontology.vw_export_edges
OWNER TO postgres;
INSERT INTO concepts.d_valuetypes SELECT 'sortorder', 'undefined', null, 'arches', 'text'
WHERE NOT EXISTS (SELECT 1 FROM concepts.d_valuetypes WHERE valuetype = 'sortorder');
CREATE OR REPLACE FUNCTION concepts.concpets_ins()
RETURNS trigger AS
$BODY$
DECLARE
v_uuid uuid = public.uuid_generate_v1mc();
BEGIN
--Provides CONCEPTID for RDM inserts and cases where ETL conceptid is not a UUID
IF NEW.CONCEPTID IS NULL THEN
NEW.CONCEPTID := v_uuid;
END IF;
-- Supports RDM where no concpetid or legacyoid is fed in
IF NEW.CONCEPTID IS NULL AND (NEW.LEGACYOID IS NULL OR NEW.LEGACYOID = '') THEN
NEW.LEGACYOID = v_uuid::text;
END IF;
-- I would assume that two cases below are handled in python code by being explicit about insert values for both columns... just coding defensively here. ABL.
-- Supports where ETL provided conceptid is a UUID and will be kept, but no LEGACYOID provided.
IF NEW.CONCEPTID IS NOT NULL and (NEW.LEGACYOID is null or NEW.LEGACYOID = '') THEN
NEW.LEGACYOID = NEW.CONCEPTID::text;
END IF;
-- Supports where ETL'ed conceptid is not a UUID. Populates original "concpetid" as LEGACYOID.
IF NEW.LEGACYOID IS NOT NULL OR NEW.LEGACYOID != '' then
NEW.LEGACYOID = NEW.LEGACYOID;
END IF;
RETURN NEW;
END$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION concepts.concpets_ins()
OWNER TO postgres;
-- Trigger: concepts_ins_tgr on concepts.concepts
DROP TRIGGER IF EXISTS concepts_ins_tgr ON concepts.concepts;
CREATE TRIGGER concepts_ins_tgr
BEFORE INSERT
ON concepts.concepts
FOR EACH ROW
EXECUTE PROCEDURE concepts.concpets_ins();"""
with transaction.atomic():
#import arches.management.patches.upgrade_to_v3_0_4
cursor = connection.cursor()
cursor.execute(sql)
anonymous_user, created = User.objects.get_or_create(username='anonymous')
if created:
anonymous_user.set_password('')
read_group, created = Group.objects.get_or_create(name='read')
anonymous_user.groups.add(read_group)
edit_group, created = Group.objects.get_or_create(name='edit')
admin_user = User.objects.get(username='admin')
admin_user.groups.add(edit_group)
admin_user.groups.add(read_group)
print '\nINSTALLING PYSHP MODULE'
print '-----------------------'
pip.main(['install', 'pyshp'])
print '\nUPDATING ENTITY INDEX'
print '---------------------'
# Add numbers array to resources that do not have them. Move numbers data from child_entities to numbers array in index.
resourceid_sql = "SELECT entityid FROM data.entities WHERE entitytypeid IN (SELECT distinct(entitytypeid) FROM data.entity_types WHERE isresource =True);"
cursor.execute(resourceid_sql)
resourceids = []
for val in cursor.fetchall():
resourceids.append(val[0])
start = time.time()
records = 0
err_found = True
se = SearchEngineFactory().create()
for resourceid in resourceids:
try:
indexed_resource = se.search(index='entity', id=resourceid)
except Exception as err:
if err_found:
print "The following resources were found in the database that don't exist in the 'entity' index:"
err_found = False
print resourceid
continue
if 'numbers' not in indexed_resource['_source']:
indexed_resource['_source']['numbers'] = []
else:
pass
for child_entity in indexed_resource['_source']['child_entities']:
if child_entity['businesstablename'] == 'numbers':
index_resource['_source']['numbers'].append(child_entity)
indexed_resource['_source']['child_entities'].remove(child_entity)
## Reindex resource here.
se.index_data(index='entity',doc_type=indexed_resource['_type'], body=indexed_resource['_source'], id=indexed_resource['_id'])
records+=1
# if records%500 == 0:
# print '%s records processed'%str(records)
print '%s records updated' % str(records)
# print 'Patch took %s seconds to run.'%str(time.time() - start)
print "\npatch '%s' successfully applied." % __name__
main()
