Ah, ok, so that all looks good. You could even remove the engine, schemas
and port lines from your file as they are set in the Arches file identical
to how you have them here.
Could you now try the following? First, in a browser navigate to one of the
resources in your database that you are trying to export, and copy the UUID
in the address bar (this is the resource's unique id).
Now, in the command line with the new virtual env activated, run
python manage.py shell
to open the python interpreter with all of your django settings.
Now try:
from arches.app.models.resource import Resource
a = Resource.get("<paste in the uuid here")
to see if you get an error there.
For context, I believe the error you are seeing is from this try/except
block, and I'm not sure what is causing it, but it would be good to narrow
it down...
https://github.com/archesproject/arches/blob/stable/3.x/arches/app/utils/data_management/resources/formats/archesjson.py#L32
Adam
On Mon, Apr 22, 2019 at 3:26 PM Joel Aldor <[email protected]> wrote:
> DATABASES['default']['NAME'] = 'arches_phm2app'
> DATABASES['default']['HOST'] = 'philippineheritagemap.org'
> DATABASES['default']['ENGINE'] = 'django.contrib.gis.db.backends.postgis'
> DATABASES['default']['PORT'] = '5432'
> DATABASES['default']['USER'] = 'postgres'
> ROOT_URLCONF = 'phm2app.urls'
> DATABASES['default']['POSTGIS_TEMPLATE'] = 'template_postgis_20'
> DATABASES['default']['PASSWORD'] = 'XXXXXXXX'
> DATABASES['default']['SCHEMAS'] = 'public,data,ontology,concepts'
>
>
> On Tuesday, April 23, 2019 at 4:22:12 AM UTC+8, Adam Cox wrote:
>>
>> Hi Joel, can you please post the db settings you have now just so we're
>> on the same page (sans sensitive info of course)?
>>
>> On Mon, Apr 22, 2019 at 3:06 PM Joel Aldor <[email protected]> wrote:
>>
>>> Hi Adam,
>>>
>>> I just did that, and it threw the same info and same export files as the
>>> last one.
>>>
>>> Regards,
>>>
>>> Joel
>>>
>>> On Tuesday, April 23, 2019 at 3:35:34 AM UTC+8, Adam Cox wrote:
>>>>
>>>> Ok, I guess it depends on whether you overwrote the entire DATABASES
>>>> variable, or if you just changed a few specific values. As an example of
>>>> the latter, you can see here how the 'default' db definition isn't
>>>> overwritten entirely, only a few of its settings (this is preferable).
>>>>
>>>> DATABASES['default']['HOST'] = 'yourdbdomain.com'
>>>> DATABASES['default']['USER'] = 'username'
>>>> DATABASES['default']['PASSWORD'] = 'XXXXXXXX'
>>>>
>>>> However, if you have something like this in settings.py:
>>>>
>>>> DATABASES = {
>>>> "default": {
>>>> 'ENGINE': 'django.contrib.gis.db.backends.postgis',
>>>> 'NAME': 'arches',
>>>> 'USER': ' username',
>>>> 'PASSWORD': ' XXXXXXXX ',
>>>> 'HOST': ' yourdbdomain.com',
>>>> 'PORT': '5432',
>>>> 'SCHEMAS': 'public,data,ontology,concepts',
>>>> 'POSTGIS_TEMPLATE': 'template_postgis_20',
>>>> }
>>>> }
>>>>
>>>> you are completely overwriting any of the db parameters that Arches
>>>> defines by default. In this case you'll need to make sure that all of the
>>>> necessary parameters (shown above) are included in your new definition, not
>>>> just the ones you want to change. Specifically, I'm thinking that you don't
>>>> have the SCHEMAS parameter.
>>>>
>>>> Adam
>>>>
>>>> On Mon, Apr 22, 2019 at 2:18 PM Joel Aldor <[email protected]> wrote:
>>>>
>>>>> Hi Adam, I did change the DATABASES setting to point to our phm2app
>>>>> production instance, as well as the ROOT_URLCONF setting. But I don't see
>>>>> a
>>>>> settings_local.py file on my 3.1.2 installation.
>>>>>
>>>>> On Tuesday, April 23, 2019 at 1:21:52 AM UTC+8, Adam Cox wrote:
>>>>>>
>>>>>> Hi Joel, have you changed your DATABASES setting at all? I have a
>>>>>> guess at what the issue may be, but it would be best to start with
>>>>>> knowing
>>>>>> exactly what is in your settings.py and settings_local.py files for
>>>>>> databases (excluding passwords of course).
>>>>>>
>>>>>> On Sat, Apr 20, 2019 at 8:12 PM Joel Aldor <[email protected]> wrote:
>>>>>>
>>>>>>> Hi Adam,
>>>>>>>
>>>>>>> OK, so I found out that the subsequent runs of the export_resources
>>>>>>> command gave me which libraries are missing, and I installed rdflib and
>>>>>>> SPARQLWrapper via pip. Now I tried to run again the command and it
>>>>>>> gives me
>>>>>>> this info:
>>>>>>>
>>>>>>> (ENV4)ubuntu@PHMARCHES3PROD:~/Projects/phmapp312test$ python manage.py
>>>>>>> packages -o export_resources -d v3fullexport.json
>>>>>>> operation: export_resources
>>>>>>> package: phmapp312test
>>>>>>> Writing 5290 HERITAGE_RESOURCE.E18 resources
>>>>>>> relation "strings" does not exist
>>>>>>> LINE 1: SELECT "strings"."entityid", "strings"."val" FROM "strings"
>>>>>>> ...
>>>>>>> ^
>>>>>>> : 5274
>>>>>>> Writing 3712 INFORMATION_RESOURCE.E73 resources
>>>>>>> relation "strings" does not exist
>>>>>>> LINE 1: SELECT "strings"."entityid", "strings"."val" FROM "strings"
>>>>>>> ...
>>>>>>> ^
>>>>>>> : 3712
>>>>>>> Writing 2 ACTIVITY.E7 resources
>>>>>>> relation "strings" does not exist
>>>>>>> LINE 1: SELECT "strings"."entityid", "strings"."val" FROM "strings"
>>>>>>> ...
>>>>>>> ^
>>>>>>> : 2
>>>>>>> Writing 4 HISTORICAL_EVENT.E5 resources
>>>>>>> relation "strings" does not exist
>>>>>>> LINE 1: SELECT "strings"."entityid", "strings"."val" FROM "strings"
>>>>>>> ...
>>>>>>> ^
>>>>>>> : 4
>>>>>>> Writing 7 HERITAGE_RESOURCE_GROUP.E27 resources
>>>>>>> relation "domains" does not exist
>>>>>>> LINE 1: SELECT "domains"."entityid", "domains"."val" FROM "domains"
>>>>>>> ...
>>>>>>> ^
>>>>>>> : 6
>>>>>>> Writing 8 ACTOR.E39 resources
>>>>>>> relation "rules" does not exist
>>>>>>> LINE 1: ...les"."entitytyperange", "rules"."propertyid" FROM "rules"
>>>>>>> WH...
>>>>>>> ^
>>>>>>> : 8
>>>>>>>
>>>>>>> The subsequent v3fullexport.json file has more strings, but most of
>>>>>>> it has "null" on it:
>>>>>>>
>>>>>>> {"resources":[{"form_groups":null,"child_entities":[],"label":"",
>>>>>>> "value":"","entitytypeid":"HERITAGE_RESOURCE.E18","entityid":
>>>>>>> "70576571-58f2-4922-aac4-3033cd724a15","property":"",
>>>>>>> "businesstablename":""},{"form_groups":null,"child_entities":[],
>>>>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18",
>>>>>>> "entityid":"216a290b-9bc6-4a3c-8066-7e0b776a4b6d","property":"",
>>>>>>> "businesstablename":""},{"form_groups":null,"child_entities":[],
>>>>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18",
>>>>>>> "entityid":"fc2a1c29-0d5e-44c9-b7ff-73fd3f3712ba","property":"",
>>>>>>> "businesstablename":""},{"form_groups":null,"child_entities":[],
>>>>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18",
>>>>>>> "entityid":"5470886e-92da-48c8-9e7e-0cb9f6a0f3cf","property":"",
>>>>>>> "businesstablename":""},{"form_groups":null,"child_entities":[],
>>>>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18",
>>>>>>> "entityid":"a67a5cb7-57e6-4d61-8d13-6bf27e327dab","property":"",
>>>>>>> "businesstablename":""},{"form_groups":null,"child_entities":[],
>>>>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18",
>>>>>>> "entityid":"188c9c73-242c-4dba-bebb-d72f7924303d","property":"",
>>>>>>> "businesstablename":""},{"form_groups":null,"child_entities":[],
>>>>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18",
>>>>>>> "entityid":"9dc90525-2f98-493e-97b9-73b9234defbe","property":"",
>>>>>>> "businesstablename":""},{"form_groups":null,"child_entities":[],
>>>>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18",
>>>>>>> "entityid":"56e375cf-3ff7-47fd-a198-15dcd107d984","property":"",
>>>>>>> "businesstablename":""},{"form_groups":null,"child_entities":[],
>>>>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18",
>>>>>>> "entityid":"30cc57e1-0425-48a6-9ee3-a2445a93d6a9","property":"",
>>>>>>> "businesstablename":""},{"form_groups":null,"child_entities":[],
>>>>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18",
>>>>>>> "entityid":"bfd47a61-4f6f-49ca-ad35-33c5a8e148b2","property":"",
>>>>>>> "businesstablename":""},{"form_groups":null,"child_entities":[],
>>>>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18",
>>>>>>> "entityid":"a396985d-79d2-426a-be2e-c170d2e4ca34","property":"",
>>>>>>> "businesstablename":""},{"form_groups":null,"child_entities":[],
>>>>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18",
>>>>>>> "entityid":"6ed80815-33d2-4dff-a39c-451126529756","property":"",
>>>>>>> "businesstablename":""},{"form_groups":null,"child_entities":[],
>>>>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18",
>>>>>>> "entityid":"92c85fd4-f41f-4464-a6f3-4ac907951de8","property":"",
>>>>>>> "businesstablename":""},{"form_groups":null,"child_entities":[],
>>>>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18",
>>>>>>> "entityid":"dec01562-3061-43e3-819c-f33f3da1b02f","property":"",
>>>>>>> "businesstablename":""},{"form_groups":null,"child_entities":[],
>>>>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18",
>>>>>>> "entityid":"671c624b-bb2e-47d4-ac70-3938314f8781","property":"",
>>>>>>> "businesstablename":""},{"form_groups":null,"child_entities":[],
>>>>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18",
>>>>>>> "entityid":"20c362a4-e7d1-4fe1-995a-9151fefa9921","property":"",
>>>>>>> "businesstablename":""},{"form_groups":null,"child_entities":[],
>>>>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE_GROUP.E27",
>>>>>>> "entityid":"9a003427-6a66-4283-b420-6a5744d7cfb9","property":"",
>>>>>>> "businesstablename":""}]}
>>>>>>>
>>>>>>> Not sure at this point what to do next. Please let me know what to
>>>>>>> look further.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Joel
>>>>>>>
>>>>>>> On Sunday, April 21, 2019 at 9:01:39 AM UTC+8, Joel Aldor wrote:
>>>>>>>>
>>>>>>>> Hi Adam,
>>>>>>>>
>>>>>>>> Tried installing Pillow, version installed is 6.0.0. Then I ran the
>>>>>>>> export_resources command again. Still produced the same export files
>>>>>>>> with
>>>>>>>> empty resources file.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Joel
>>>>>>>>
>>>>>>>> On Sunday, April 21, 2019 at 8:50:38 AM UTC+8, Adam Cox wrote:
>>>>>>>>>
>>>>>>>>> Hi Joel, if you have the new virtual environment activated you
>>>>>>>>> want have to change the settings file at all.
>>>>>>>>>
>>>>>>>>> The PIL error indicates that that python library (and potentially
>>>>>>>>> others) didn't install correctly... Not sure why. You could try pip
>>>>>>>>> installing PIL. I can't look it up right now but it may be actually be
>>>>>>>>> called Pillow in pip.
>>>>>>>>>
>>>>>>>>> Adam
>>>>>>>>>
>>>>>>>>> On Sat, Apr 20, 2019, 7:37 PM Joel Aldor <[email protected]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Adam,
>>>>>>>>>>
>>>>>>>>>> I'd like to think we need to explicitly point the settings.py to
>>>>>>>>>> our phm2app instance in order for the export_resources command to run
>>>>>>>>>> properly, but I'm not sure which parts I will need to edit, apart
>>>>>>>>>> from the
>>>>>>>>>> DATABASES settings. You think this might be the problem?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Joel
>>>>>>>>>>
>>>>>>>>>> On Sunday, April 21, 2019 at 8:25:41 AM UTC+8, Joel Aldor wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Adam,
>>>>>>>>>>>
>>>>>>>>>>> I tried to install arches v3.1.2 and pointed the instance into
>>>>>>>>>>> our existing database on settings.py, then I ran the
>>>>>>>>>>> export_resources
>>>>>>>>>>> command, which produced those export files. However, the console
>>>>>>>>>>> update
>>>>>>>>>>> gave me this info:
>>>>>>>>>>>
>>>>>>>>>>> (ENV4)ubuntu@PHMARCHES3PROD:~/Projects/phmapp312test$ python
>>>>>>>>>>> manage.py packages -o export_resources -d v3fullexport.json
>>>>>>>>>>> operation: export_resources
>>>>>>>>>>> package: phmapp312test
>>>>>>>>>>> Writing 5290 HERITAGE_RESOURCE.E18 resources
>>>>>>>>>>> No module named PIL : 5290
>>>>>>>>>>> Writing 3712 INFORMATION_RESOURCE.E73 resources
>>>>>>>>>>> No module named PIL : 3712
>>>>>>>>>>> Writing 2 ACTIVITY.E7 resources
>>>>>>>>>>> No module named PIL : 2
>>>>>>>>>>> Writing 4 HISTORICAL_EVENT.E5 resources
>>>>>>>>>>> No module named PIL : 4
>>>>>>>>>>> Writing 7 HERITAGE_RESOURCE_GROUP.E27 resources
>>>>>>>>>>> No module named PIL : 7
>>>>>>>>>>> Writing 8 ACTOR.E39 resources
>>>>>>>>>>> No module named PIL : 8
>>>>>>>>>>>
>>>>>>>>>>> When I checked the v3fullexport.json file it was almost empty.
>>>>>>>>>>> All it has inside the file is this string:
>>>>>>>>>>>
>>>>>>>>>>> {"resources":[]}
>>>>>>>>>>>
>>>>>>>>>>> The v3fullexport.relations file is full of all the relationships.
>>>>>>>>>>>
>>>>>>>>>>> I have a feeling I might be missing something here. Please let
>>>>>>>>>>> me know what I should do further. Thank you.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Joel
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Saturday, April 20, 2019 at 8:59:12 PM UTC+8, Adam Cox wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Joel, indeed, exporting the v3 resources is a crucial step
>>>>>>>>>>>> in the migration process!!
>>>>>>>>>>>>
>>>>>>>>>>>> I'm unable to look into this deeply right now but I have a
>>>>>>>>>>>> feeling that this means you don't have the latest version of
>>>>>>>>>>>> Arches 3,
>>>>>>>>>>>> which is 3.1.2. When I can, I'll look back through the version
>>>>>>>>>>>> history and
>>>>>>>>>>>> see where that command was introduced.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm the meantime, you could create a new virtual environment
>>>>>>>>>>>> and explicitly install arches 3.1.2 into that environment with
>>>>>>>>>>>>
>>>>>>>>>>>> pip install arches==3.1.2
>>>>>>>>>>>>
>>>>>>>>>>>> Then activate that new environment and try exporting again.
>>>>>>>>>>>> This would leave your old env untouched.
>>>>>>>>>>>>
>>>>>>>>>>>> Adam
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Apr 20, 2019, 6:23 AM Joel Aldor <[email protected]>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> We're supposed to migrate our v3 data into v4 this Easter
>>>>>>>>>>>>> weekend using the instructions from this link
>>>>>>>>>>>>> <https://arches.readthedocs.io/en/stable/exporting-v3-data/>,
>>>>>>>>>>>>> but we ran into a problem. Apparently, our v3 instance doesn't
>>>>>>>>>>>>> have the
>>>>>>>>>>>>> export_resources option when we execute this command:
>>>>>>>>>>>>>
>>>>>>>>>>>>> (ENV)ubuntu@PHMARCHES3PROD:~/Projects/phm2app$ python manage.py
>>>>>>>>>>>>> packages -o export_resources -d v3fullexport.json
>>>>>>>>>>>>> Usage: manage.py packages [options]
>>>>>>>>>>>>>
>>>>>>>>>>>>> manage.py: error: option -o: invalid choice:
>>>>>>>>>>>>> 'export_resources' (choose from 'setup', 'install', 'setup_db'
>>>>>>>>>>>>> , 'start_elasticsearch', 'setup_elasticsearch',
>>>>>>>>>>>>> 'build_permissions', 'livereload', 'load_resources',
>>>>>>>>>>>>> 'remove_resources', 'load_concept_scheme', 'index_database',
>>>>>>>>>>>>> 'export_resource_graphs')
>>>>>>>>>>>>>
>>>>>>>>>>>>> How can I export the business data from here?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Joel
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> -- To post, send email to [email protected]. To
>>>>>>>>>>>>> unsubscribe, send email to [email protected]. For
>>>>>>>>>>>>> more information, visit
>>>>>>>>>>>>> https://groups.google.com/d/forum/archesproject?hl=en
>>>>>>>>>>>>> ---
>>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>>> Google Groups "Arches Project" group.
>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>>>> it, send an email to [email protected].
>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>> -- To post, send email to [email protected]. To
>>>>>>>>>> unsubscribe, send email to [email protected]. For more
>>>>>>>>>> information, visit
>>>>>>>>>> https://groups.google.com/d/forum/archesproject?hl=en
>>>>>>>>>> ---
>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>> Google Groups "Arches Project" group.
>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>>> send an email to [email protected].
>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>
>>>>>>>>> --
>>>>>>> -- To post, send email to [email protected]. To
>>>>>>> unsubscribe, send email to [email protected]. For more
>>>>>>> information, visit
>>>>>>> https://groups.google.com/d/forum/archesproject?hl=en
>>>>>>> ---
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Arches Project" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to [email protected].
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>> --
>>>>> -- To post, send email to [email protected]. To unsubscribe,
>>>>> send email to [email protected]. For more information, visit
>>>>> https://groups.google.com/d/forum/archesproject?hl=en
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Arches Project" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>> --
>>> -- To post, send email to [email protected]. To unsubscribe,
>>> send email to [email protected]. For more information, visit
>>> https://groups.google.com/d/forum/archesproject?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Arches Project" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> -- To post, send email to [email protected]. To unsubscribe,
> send email to [email protected]. For more
> information, visit https://groups.google.com/d/forum/archesproject?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Arches Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/d/optout.
>
--
-- To post, send email to [email protected]. To unsubscribe, send
email to [email protected]. For more information,
visit https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
