Hi Adam and Adam,
Just to update you on my export question. As a quick fix we commented out
the sort function in the following relevant section of code, which we found
in format.py:
def concatenate_value_lists(self, template_record):
"""
If multiple values are found for a column, joins them into a
semi-colon concatenated string.
"""
for k, v in template_record.iteritems():
if type(v) == list:
# v.sort()
try:
template_record[k] = ("; ").join(v)
except:
unicode_vals = [unicode(x) for x in v]
template_record[k] = ("; ").join(vals)
return template_record
This leaves the arrays in the same order that the team entered them into
the forms, meaning we know that the the third value in Resource A
corresponds with the third value in Resource B where these resources are
linked by the same Add button We can then manipulate the data outside of
arches as we prepare to convert it into .arches format. I will read more
about batch uploading as I know that you have written more on this since I
uploaded our data originally Adam C, but I am wondering whether .arches can
read arrays or whether we need to work with Group IDs.
Best wishes
Richard
On Thursday, January 14, 2016 at 7:51:17 PM UTC, Adam Cox wrote:
>
> HI Adam and Richard, along these same lines I posted this issue a while
> ago: https://github.com/archesproject/arches/issues/267. The big
> advantage I see in dumping everything into a .arches/.relations file is
> that you could then use any manner programming/basic find and replace
> operations to make alterations across the whole database. Then you can
> reload it with any new node names or conceptids that you've incorporated.
>
> The big disadvantage is that a .arches file of the size that Richard is
> working with would be monumentally large and the loading process would take
> a *lot *of time and memory. Also, manipulating a text file that size may
> not be feasible at all.
>
> Adam
>
> On Thursday, January 14, 2016 at 9:28:14 AM UTC-6, Richard Jennings wrote:
>>
>> Hi Adam,
>>
>> Thanks for your reply. I had some success and have spent the day
>> exploring our database within Postgres via pgAdmin! I re-read the Arches
>> documentation entitled 'Physical Data Model' and looked at your
>> ontology-resources-reference data figure and now understand that the data
>> is stored in Postgres within 'strings' 'domains' 'entity type' etc tables,
>> rather than tables such as 'Heritage Resource Group' or other fields like
>> this which I was initially searching for. These are stored within
>> Schemas/data/tables, which I saw yesterday but did not make the connection
>> that this is where all of our data resides. I ran queries such as 'SELECT *
>> FROM data.strings' and the data appeared! It is very instructive to look at
>> how it all fits together in the back-end. It helps me understand
>> conceptually about Arches. I can also see that UUIDs in the 'relations'
>> table are present and link measurement unit to type to value.
>>
>> Hopefully you have some thoughts on the export configuration file as it
>> would be great to be able to export the data out in the manner expressed in
>> my first email.
>>
>> I will definitely look at the back-up options now.
>>
>> Many thanks,
>>
>> Richard
>>
>>
>> On Thursday, January 14, 2016 at 3:03:20 PM UTC, Adam Lodge wrote:
>>>
>>> Richard,
>>>
>>> I am putting some thought to this for you and will try to respond
>>> shortly with a more complete answer.
>>>
>>> But, first things first… this may be obvious, but before you drop your
>>> postgres database containing production data, please be sure to use the
>>> backup command in postgres to take an “as-is” snapshot of your database.
>>> This will give you a backout plan that at least keeps the current state of
>>> your production database intact just in case things go haywire in your
>>> export/import workflow.
>>>
>>> Documentation on the Postgres backup command here:
>>> http://www.postgresql.org/docs/9.1/static/backup.html
>>> Also - there is UI within PgAdmin that uses the backup/restore
>>> functionality referenced above. It’s very easy to use.
>>>
>>> Best,
>>> Adam
>>>
>>> --
>>> Adam Lodge
>>> Geospatial Systems Consultant
>>> Farallon Geographics
>>>
>>> On Wednesday, January 13, 2016 at 12:11 PM, Richard Jennings wrote:
>>>
>>> Hi All,
>>>
>>> I hope you are well and Happy New Year! I am in the process of modifying
>>> our Arches database and would like some help please. We created a version
>>> around two months ago which is on a server and in production mode with many
>>> records entered. In the meantime, I have made some design changes
>>> (concurrently in a separate offline version) in which I have added some new
>>> resource nodes, which means from my understanding means that we need to
>>> re-drop the database to apply the changes. This will overwrite the existing
>>> data so I need to export this data out first. I have now finished the
>>> modifications and am ready to begin this phase.
>>>
>>> The following tasks are ahead as I see it:
>>>
>>> 1. Export the data out of the live database on the server (.csv file
>>> using json)
>>> 2. Overwrite the database on the sever (once the data is safely
>>> exported) by installing the most recent Arches and HIP updates on to the
>>> server (I've not done this yet while I have been in the middle of the
>>> changes)
>>> 3. Drop the new version of the database on the server
>>> 4. Copy over the new resource graphs, authority files and template
>>> changes etc that I've made on the offline version on to the clean new
>>> Arches database on the server
>>> 5. Reconfigure the exported data into .arches format
>>> 6. Batch upload into the new database
>>>
>>> Problems that I welcome that your advice:
>>>
>>> 1. Data export. Is it possible to retain groupings made during data
>>> entry? For example in the Measurement form, if there are multiple
>>> measurements taken of a given resource, such as length, width and height,
>>> each with different associated units and values, when I export the values
>>> in the json file, it gives one row per ArchesID, and separates the values
>>> in the different variables numerically or alphabetically via semicolons.
>>> The problem we have is that the relationship is not shown between them. For
>>> instance, we don't know which unit corresponds to which measurement value.
>>> (Forgive me if you set this out more clearly in the latest upgrades). It
>>> has me wondering as well, when you have multiple nodes that are linked by
>>> the single Add button, such as for Measurements or Components, name and
>>> name type, how are these relationships saved in the database if there is
>>> more than one set of values added? Is there a group ID assigned to them in
>>> the same way that we use a Group ID when we do the batch uploads?
>>>
>>> 2. To explore these queries further I thought I would consult the
>>> backend of the database via pgAdmin III. However, while I can locate the
>>> arches database that I dropped, I am unsure whereabouts within this
>>> database that our data is stored.
>>>
>>> Many thanks in advance and thanks for all your hard work.
>>>
>>> Best wishes
>>>
>>> Richard
>>>
>>> --
>>> -- To post, send email to [email protected]. To unsubscribe,
>>> send email to [email protected]. For more information,
>>> visit https://groups.google.com/d/forum/archesproject?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Arches Project" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
--
-- To post, send email to [email protected]. To unsubscribe, send
email to [email protected]. For more information,
visit https://groups.google.com/d/forum/archesproject?hl=en
---
You received this message because you are subscribed to the Google Groups
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
