Hi Joel, can you please post the db settings you have now just so we're on the same page (sans sensitive info of course)?
On Mon, Apr 22, 2019 at 3:06 PM Joel Aldor <[email protected]> wrote: > Hi Adam, > > I just did that, and it threw the same info and same export files as the > last one. > > Regards, > > Joel > > On Tuesday, April 23, 2019 at 3:35:34 AM UTC+8, Adam Cox wrote: >> >> Ok, I guess it depends on whether you overwrote the entire DATABASES >> variable, or if you just changed a few specific values. As an example of >> the latter, you can see here how the 'default' db definition isn't >> overwritten entirely, only a few of its settings (this is preferable). >> >> DATABASES['default']['HOST'] = 'yourdbdomain.com' >> DATABASES['default']['USER'] = 'username' >> DATABASES['default']['PASSWORD'] = 'XXXXXXXX' >> >> However, if you have something like this in settings.py: >> >> DATABASES = { >> "default": { >> 'ENGINE': 'django.contrib.gis.db.backends.postgis', >> 'NAME': 'arches', >> 'USER': ' username', >> 'PASSWORD': ' XXXXXXXX ', >> 'HOST': ' yourdbdomain.com', >> 'PORT': '5432', >> 'SCHEMAS': 'public,data,ontology,concepts', >> 'POSTGIS_TEMPLATE': 'template_postgis_20', >> } >> } >> >> you are completely overwriting any of the db parameters that Arches >> defines by default. In this case you'll need to make sure that all of the >> necessary parameters (shown above) are included in your new definition, not >> just the ones you want to change. Specifically, I'm thinking that you don't >> have the SCHEMAS parameter. >> >> Adam >> >> On Mon, Apr 22, 2019 at 2:18 PM Joel Aldor <[email protected]> wrote: >> >>> Hi Adam, I did change the DATABASES setting to point to our phm2app >>> production instance, as well as the ROOT_URLCONF setting. But I don't see a >>> settings_local.py file on my 3.1.2 installation. >>> >>> On Tuesday, April 23, 2019 at 1:21:52 AM UTC+8, Adam Cox wrote: >>>> >>>> Hi Joel, have you changed your DATABASES setting at all? I have a guess >>>> at what the issue may be, but it would be best to start with knowing >>>> exactly what is in your settings.py and settings_local.py files for >>>> databases (excluding passwords of course). >>>> >>>> On Sat, Apr 20, 2019 at 8:12 PM Joel Aldor <[email protected]> wrote: >>>> >>>>> Hi Adam, >>>>> >>>>> OK, so I found out that the subsequent runs of the export_resources >>>>> command gave me which libraries are missing, and I installed rdflib and >>>>> SPARQLWrapper via pip. Now I tried to run again the command and it gives >>>>> me >>>>> this info: >>>>> >>>>> (ENV4)ubuntu@PHMARCHES3PROD:~/Projects/phmapp312test$ python manage.py >>>>> packages -o export_resources -d v3fullexport.json >>>>> operation: export_resources >>>>> package: phmapp312test >>>>> Writing 5290 HERITAGE_RESOURCE.E18 resources >>>>> relation "strings" does not exist >>>>> LINE 1: SELECT "strings"."entityid", "strings"."val" FROM "strings" >>>>> ... >>>>> ^ >>>>> : 5274 >>>>> Writing 3712 INFORMATION_RESOURCE.E73 resources >>>>> relation "strings" does not exist >>>>> LINE 1: SELECT "strings"."entityid", "strings"."val" FROM "strings" >>>>> ... >>>>> ^ >>>>> : 3712 >>>>> Writing 2 ACTIVITY.E7 resources >>>>> relation "strings" does not exist >>>>> LINE 1: SELECT "strings"."entityid", "strings"."val" FROM "strings" >>>>> ... >>>>> ^ >>>>> : 2 >>>>> Writing 4 HISTORICAL_EVENT.E5 resources >>>>> relation "strings" does not exist >>>>> LINE 1: SELECT "strings"."entityid", "strings"."val" FROM "strings" >>>>> ... >>>>> ^ >>>>> : 4 >>>>> Writing 7 HERITAGE_RESOURCE_GROUP.E27 resources >>>>> relation "domains" does not exist >>>>> LINE 1: SELECT "domains"."entityid", "domains"."val" FROM "domains" >>>>> ... >>>>> ^ >>>>> : 6 >>>>> Writing 8 ACTOR.E39 resources >>>>> relation "rules" does not exist >>>>> LINE 1: ...les"."entitytyperange", "rules"."propertyid" FROM "rules" >>>>> WH... >>>>> ^ >>>>> : 8 >>>>> >>>>> The subsequent v3fullexport.json file has more strings, but most of it >>>>> has "null" on it: >>>>> >>>>> {"resources":[{"form_groups":null,"child_entities":[],"label":"", >>>>> "value":"","entitytypeid":"HERITAGE_RESOURCE.E18","entityid": >>>>> "70576571-58f2-4922-aac4-3033cd724a15","property":"", >>>>> "businesstablename":""},{"form_groups":null,"child_entities":[], >>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18", >>>>> "entityid":"216a290b-9bc6-4a3c-8066-7e0b776a4b6d","property":"", >>>>> "businesstablename":""},{"form_groups":null,"child_entities":[], >>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18", >>>>> "entityid":"fc2a1c29-0d5e-44c9-b7ff-73fd3f3712ba","property":"", >>>>> "businesstablename":""},{"form_groups":null,"child_entities":[], >>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18", >>>>> "entityid":"5470886e-92da-48c8-9e7e-0cb9f6a0f3cf","property":"", >>>>> "businesstablename":""},{"form_groups":null,"child_entities":[], >>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18", >>>>> "entityid":"a67a5cb7-57e6-4d61-8d13-6bf27e327dab","property":"", >>>>> "businesstablename":""},{"form_groups":null,"child_entities":[], >>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18", >>>>> "entityid":"188c9c73-242c-4dba-bebb-d72f7924303d","property":"", >>>>> "businesstablename":""},{"form_groups":null,"child_entities":[], >>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18", >>>>> "entityid":"9dc90525-2f98-493e-97b9-73b9234defbe","property":"", >>>>> "businesstablename":""},{"form_groups":null,"child_entities":[], >>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18", >>>>> "entityid":"56e375cf-3ff7-47fd-a198-15dcd107d984","property":"", >>>>> "businesstablename":""},{"form_groups":null,"child_entities":[], >>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18", >>>>> "entityid":"30cc57e1-0425-48a6-9ee3-a2445a93d6a9","property":"", >>>>> "businesstablename":""},{"form_groups":null,"child_entities":[], >>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18", >>>>> "entityid":"bfd47a61-4f6f-49ca-ad35-33c5a8e148b2","property":"", >>>>> "businesstablename":""},{"form_groups":null,"child_entities":[], >>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18", >>>>> "entityid":"a396985d-79d2-426a-be2e-c170d2e4ca34","property":"", >>>>> "businesstablename":""},{"form_groups":null,"child_entities":[], >>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18", >>>>> "entityid":"6ed80815-33d2-4dff-a39c-451126529756","property":"", >>>>> "businesstablename":""},{"form_groups":null,"child_entities":[], >>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18", >>>>> "entityid":"92c85fd4-f41f-4464-a6f3-4ac907951de8","property":"", >>>>> "businesstablename":""},{"form_groups":null,"child_entities":[], >>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18", >>>>> "entityid":"dec01562-3061-43e3-819c-f33f3da1b02f","property":"", >>>>> "businesstablename":""},{"form_groups":null,"child_entities":[], >>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18", >>>>> "entityid":"671c624b-bb2e-47d4-ac70-3938314f8781","property":"", >>>>> "businesstablename":""},{"form_groups":null,"child_entities":[], >>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE.E18", >>>>> "entityid":"20c362a4-e7d1-4fe1-995a-9151fefa9921","property":"", >>>>> "businesstablename":""},{"form_groups":null,"child_entities":[], >>>>> "label":"","value":"","entitytypeid":"HERITAGE_RESOURCE_GROUP.E27", >>>>> "entityid":"9a003427-6a66-4283-b420-6a5744d7cfb9","property":"", >>>>> "businesstablename":""}]} >>>>> >>>>> Not sure at this point what to do next. Please let me know what to >>>>> look further. >>>>> >>>>> Thanks, >>>>> >>>>> Joel >>>>> >>>>> On Sunday, April 21, 2019 at 9:01:39 AM UTC+8, Joel Aldor wrote: >>>>>> >>>>>> Hi Adam, >>>>>> >>>>>> Tried installing Pillow, version installed is 6.0.0. Then I ran the >>>>>> export_resources command again. Still produced the same export files with >>>>>> empty resources file. >>>>>> >>>>>> Regards, >>>>>> >>>>>> Joel >>>>>> >>>>>> On Sunday, April 21, 2019 at 8:50:38 AM UTC+8, Adam Cox wrote: >>>>>>> >>>>>>> Hi Joel, if you have the new virtual environment activated you want >>>>>>> have to change the settings file at all. >>>>>>> >>>>>>> The PIL error indicates that that python library (and potentially >>>>>>> others) didn't install correctly... Not sure why. You could try pip >>>>>>> installing PIL. I can't look it up right now but it may be actually be >>>>>>> called Pillow in pip. >>>>>>> >>>>>>> Adam >>>>>>> >>>>>>> On Sat, Apr 20, 2019, 7:37 PM Joel Aldor <[email protected]> wrote: >>>>>>> >>>>>>>> Hi Adam, >>>>>>>> >>>>>>>> I'd like to think we need to explicitly point the settings.py to >>>>>>>> our phm2app instance in order for the export_resources command to run >>>>>>>> properly, but I'm not sure which parts I will need to edit, apart from >>>>>>>> the >>>>>>>> DATABASES settings. You think this might be the problem? >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Joel >>>>>>>> >>>>>>>> On Sunday, April 21, 2019 at 8:25:41 AM UTC+8, Joel Aldor wrote: >>>>>>>>> >>>>>>>>> Hi Adam, >>>>>>>>> >>>>>>>>> I tried to install arches v3.1.2 and pointed the instance into our >>>>>>>>> existing database on settings.py, then I ran the export_resources >>>>>>>>> command, >>>>>>>>> which produced those export files. However, the console update gave >>>>>>>>> me this >>>>>>>>> info: >>>>>>>>> >>>>>>>>> (ENV4)ubuntu@PHMARCHES3PROD:~/Projects/phmapp312test$ python >>>>>>>>> manage.py packages -o export_resources -d v3fullexport.json >>>>>>>>> operation: export_resources >>>>>>>>> package: phmapp312test >>>>>>>>> Writing 5290 HERITAGE_RESOURCE.E18 resources >>>>>>>>> No module named PIL : 5290 >>>>>>>>> Writing 3712 INFORMATION_RESOURCE.E73 resources >>>>>>>>> No module named PIL : 3712 >>>>>>>>> Writing 2 ACTIVITY.E7 resources >>>>>>>>> No module named PIL : 2 >>>>>>>>> Writing 4 HISTORICAL_EVENT.E5 resources >>>>>>>>> No module named PIL : 4 >>>>>>>>> Writing 7 HERITAGE_RESOURCE_GROUP.E27 resources >>>>>>>>> No module named PIL : 7 >>>>>>>>> Writing 8 ACTOR.E39 resources >>>>>>>>> No module named PIL : 8 >>>>>>>>> >>>>>>>>> When I checked the v3fullexport.json file it was almost empty. All >>>>>>>>> it has inside the file is this string: >>>>>>>>> >>>>>>>>> {"resources":[]} >>>>>>>>> >>>>>>>>> The v3fullexport.relations file is full of all the relationships. >>>>>>>>> >>>>>>>>> I have a feeling I might be missing something here. Please let me >>>>>>>>> know what I should do further. Thank you. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Joel >>>>>>>>> >>>>>>>>> >>>>>>>>> On Saturday, April 20, 2019 at 8:59:12 PM UTC+8, Adam Cox wrote: >>>>>>>>>> >>>>>>>>>> Hi Joel, indeed, exporting the v3 resources is a crucial step in >>>>>>>>>> the migration process!! >>>>>>>>>> >>>>>>>>>> I'm unable to look into this deeply right now but I have a >>>>>>>>>> feeling that this means you don't have the latest version of Arches >>>>>>>>>> 3, >>>>>>>>>> which is 3.1.2. When I can, I'll look back through the version >>>>>>>>>> history and >>>>>>>>>> see where that command was introduced. >>>>>>>>>> >>>>>>>>>> I'm the meantime, you could create a new virtual environment and >>>>>>>>>> explicitly install arches 3.1.2 into that environment with >>>>>>>>>> >>>>>>>>>> pip install arches==3.1.2 >>>>>>>>>> >>>>>>>>>> Then activate that new environment and try exporting again. This >>>>>>>>>> would leave your old env untouched. >>>>>>>>>> >>>>>>>>>> Adam >>>>>>>>>> >>>>>>>>>> On Sat, Apr 20, 2019, 6:23 AM Joel Aldor <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> We're supposed to migrate our v3 data into v4 this Easter >>>>>>>>>>> weekend using the instructions from this link >>>>>>>>>>> <https://arches.readthedocs.io/en/stable/exporting-v3-data/>, >>>>>>>>>>> but we ran into a problem. Apparently, our v3 instance doesn't have >>>>>>>>>>> the >>>>>>>>>>> export_resources option when we execute this command: >>>>>>>>>>> >>>>>>>>>>> (ENV)ubuntu@PHMARCHES3PROD:~/Projects/phm2app$ python manage.py >>>>>>>>>>> packages -o export_resources -d v3fullexport.json >>>>>>>>>>> Usage: manage.py packages [options] >>>>>>>>>>> >>>>>>>>>>> manage.py: error: option -o: invalid choice: 'export_resources' >>>>>>>>>>> (choose from 'setup', 'install', 'setup_db', >>>>>>>>>>> 'start_elasticsearch', 'setup_elasticsearch', >>>>>>>>>>> 'build_permissions', 'livereload', 'load_resources', >>>>>>>>>>> 'remove_resources', 'load_concept_scheme', 'index_database', >>>>>>>>>>> 'export_resource_graphs') >>>>>>>>>>> >>>>>>>>>>> How can I export the business data from here? >>>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> >>>>>>>>>>> Joel >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> -- To post, send email to [email protected]. To >>>>>>>>>>> unsubscribe, send email to [email protected]. For more >>>>>>>>>>> information, visit >>>>>>>>>>> https://groups.google.com/d/forum/archesproject?hl=en >>>>>>>>>>> --- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "Arches Project" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>> -- To post, send email to [email protected]. To >>>>>>>> unsubscribe, send email to [email protected]. For more >>>>>>>> information, visit >>>>>>>> https://groups.google.com/d/forum/archesproject?hl=en >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Arches Project" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> -- >>>>> -- To post, send email to [email protected]. To unsubscribe, >>>>> send email to [email protected]. For more information, visit >>>>> https://groups.google.com/d/forum/archesproject?hl=en >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Arches Project" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>> -- To post, send email to [email protected]. To unsubscribe, >>> send email to [email protected]. For more information, visit >>> https://groups.google.com/d/forum/archesproject?hl=en >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Arches Project" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > -- To post, send email to [email protected]. To unsubscribe, > send email to [email protected]. For more > information, visit https://groups.google.com/d/forum/archesproject?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Arches Project" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- -- To post, send email to [email protected]. To unsubscribe, send email to [email protected]. For more information, visit https://groups.google.com/d/forum/archesproject?hl=en --- You received this message because you are subscribed to the Google Groups "Arches Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
