Re: [Arches] Arches 5.0 Docker setup problems

[Dennis Wuthrich]

Hi David,

In Arches 5.0, ontologies are loaded as part of a package.  See 
https://github.com/archesproject/arches-her/tree/master/arches_her/pkg for 
an example of how ontologies are defined.  These are automatically loaded 
as part of the package load command.

Hope this helps.

-Dennis

On Saturday, March 7, 2020 at 3:13:44 PM UTC-8, David Osborne wrote:
>
> Hi Ben
>
> Thanks, that was it: starting the elasticsearch and db containers first 
> was the solution.
>
> My docker-compose file asked for the default resource models and concepts. 
> Although the concepts loaded ok, the branches and models didn't, with a 
> missing ontology, although I can't see how to enable whatever's missing:
>
> arches   | Running: python manage.py packages -o import_graphs
> arches   | operation: import_graphs
> arches   | /web_root/arches/arches/db/graphs/branches/Information 
> Resource Copyright.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Event Phase 
> Type.json
> arches   | Ontology matching query does not exist.
> arches   | 
> /web_root/arches/arches/db/graphs/branches/Description.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Place 1.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Resource 
> Type Classification.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Publication 
> Event.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/External 
> Identifier.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Heritage 
> Resource Group Phase Type.json
> arches   | Ontology matching query does not exist.
> arches   | 
> /web_root/arches/arches/db/graphs/branches/Appellation.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Modification 
> Event.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Heritage 
> Resource Phase Type.json
> arches   | Ontology matching query does not exist.
> arches   | 
> /web_root/arches/arches/db/graphs/branches/Component.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Settings 
> System Defaults.json
> arches   | Ontology matching query does not exist.
> arches   | 
> /web_root/arches/arches/db/graphs/branches/Evaluation.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Activity 
> Phase.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Place 3.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Resource 
> Update Event.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Keyword.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Beginning of 
> Existence.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Settings 
> Map.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Actor 
> Phase.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Right.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Title.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Information 
> Carrier.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Information 
> Resource Creation Event.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Settings 
> Saved Searches.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Settings 
> Time Search.json
> arches   | Ontology matching query does not exist.
> arches   | /web_root/arches/arches/db/graphs/branches/Language.json
> arches   | Ontology 

Re: [Arches] Arches 5.0 Docker setup problems

[David Osborne]

Hi Ben

Thanks, that was it: starting the elasticsearch and db containers first was 
the solution.

My docker-compose file asked for the default resource models and concepts. 
Although the concepts loaded ok, the branches and models didn't, with a 
missing ontology, although I can't see how to enable whatever's missing:

arches   | Running: python manage.py packages -o import_graphs
arches   | operation: import_graphs
arches   | /web_root/arches/arches/db/graphs/branches/Information 
Resource Copyright.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Event Phase 
Type.json
arches   | Ontology matching query does not exist.
arches   | 
/web_root/arches/arches/db/graphs/branches/Description.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Place 1.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Resource Type 
Classification.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Publication 
Event.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/External 
Identifier.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Heritage 
Resource Group Phase Type.json
arches   | Ontology matching query does not exist.
arches   | 
/web_root/arches/arches/db/graphs/branches/Appellation.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Modification 
Event.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Heritage 
Resource Phase Type.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Component.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Settings 
System Defaults.json
arches   | Ontology matching query does not exist.
arches   | 
/web_root/arches/arches/db/graphs/branches/Evaluation.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Activity 
Phase.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Place 3.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Resource 
Update Event.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Keyword.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Beginning of 
Existence.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Settings 
Map.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Actor 
Phase.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Right.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Title.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Information 
Carrier.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Information 
Resource Creation Event.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Settings 
Saved Searches.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Settings Time 
Search.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Language.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/End of 
Existence.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Settings 
Basic Search.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Existence.json
arches   | Ontology matching query does not exist.
arches   | /web_root/arches/arches/db/graphs/branches/Temporal 

Re: [Arches] Arches 5.0 Docker setup problems

[Ben O'Steen]

Im going to guess that the elasticsearch container is not ready and the
migration script fails when it attempts to run the first time through and
Arches ends up in a bad state which it generally can never recover from.

docker-compose down -v

This will kill and delete all the badly formed docker containers created by
a failed initial run.

docker-compose up --no-start

This will initialize the containers and network for the stack.

docker-compose start elasticsearch db

This will start the elasticsearch instance and the database.

docker logs -f elasticsearch

This will let you watch what it is doing. Ctrl-c to get back to normal.
I've found that if you run antivirus checkers and have some network
slowness, it can take a while (30s to 1min) for elasticsearch to actually
be ready. It sadly lies on its health check command and reports "green"
while it is still doing its random clustering things.

docker-compose up

After elasticsearch has finished doing whatever it does on start up that
takes so long, run this and watch to make sure there are no connection
errors.


Ben

On Thu, 5 Mar 2020, 07:44 David Osborne, 
wrote:

> Thanks, Ben: your comment helped me make some progress (to another error
> ;-)
>
> The problem is that the Readme.md in the docker directory of the Github
> archesproject/arches repo (file last revised 17 months ago!) is inaccurate
> or vague when it says "Create a new folder for your custom Arches code"
> without specifying where. I created one alongside my cloned arches repo,
> with the result that the path to /docker-entrypoint-initdb.d/init.sql did
> not exist, so Docker created directories instead.
>
> Having located the arches/install/init-unix.sql file referred to in the db
> service, I worked out that it's easiest to run docker-compose in the cloned
> arches directory itself, so I edited the docker-compose.yml file, and that
> file and the resulting output from docker-compose is in
> https://gist.github.com/daosborne/bc83812e33d9396d679af54f31565895 (with
> the lines from couchdb and elasticsearch edited out from the docker-compose
> log as not relevant).
>
> It gets further but has some problems (partly because I want the default
> graphs and concepts installed):
>
>1. error setting up db (traceback starting line 133)
>2. database tables graphs, ontologies, concepts, d_value_types don't
>exist (lines 222--648)
>3. db connection error during migrations (line 734)
>4. db column nodes.fieldname doesn't exist (lines 1123, 1185)
>
> I had also tried setting a project name, as suggested in the Readme but
> abandoned that because the directory was missing from /web_root (and
> produces a spurious error message about the project name clashing with a
> Python module).
>
> I hoped that using Docker would be a quick and easy way of getting a new
> Arches installation running but it seems it still has some rough edges. As
> I need to get some development done to make some slides for a conference
> paper, I think I'll go back to the tried and tested method of setting up a
> Vagrant VM!
>
> Cheers
> David
>
> On Wednesday, 4 March 2020 00:33:38 UTC, Ben O'Steen wrote:
>>
>> Hi,
>>
>> I think the key error is on line 63 of the gist you posted. For some
>> reason, it is interpreting file bind mount as a directory and not as a
>> file. If I run the default docker compose file here, and start up the db
>> container, this is the log output I get:
>>
>> $  docker logs db
>> Add rule to pg_hba: 0.0.0.0/0
>> Add rule to pg_hba: replication replicator
>> Setup master database
>> 2020-03-04 00:28:12.848 UTC [25] LOG:  starting PostgreSQL 12.1 (Debian
>> 12.1-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6)
>> 8.3.0, 64-bit
>> 2020-03-04 00:28:12.849 UTC [25] LOG:  listening on IPv4 address
>> "127.0.0.1", port 5432
>> 2020-03-04 00:28:12.855 UTC [25] LOG:  listening on Unix socket
>> "/var/run/postgresql/.s.PGSQL.5432"
>> 2020-03-04 00:28:12.928 UTC [36] LOG:  database system was shut down at
>> 2019-12-03 16:35:30 UTC
>> 2020-03-04 00:28:12.931 UTC [37] postgres@postgres FATAL:  the database
>> system is starting up
>> psql: error: could not connect to server: FATAL:  the database system is
>> starting up
>> 2020-03-04 00:28:12.948 UTC [25] LOG:  database system is ready to accept
>> connections
>>   List of databases
>>Name|  Owner   | Encoding | Collate |  Ctype  |   Access privileges
>>
>> ---+--+--+-+-+---
>>  postgres  | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
>>  template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres
>>+
>>|  |  | | |
>> postgres=CTc/postgres
>>  template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres
>>+
>>|  |  | | |
>> postgres=CTc/postgres
>> (3 rows)
>>
>> postgres ready
>> Setup postgres User:Password
>> Creating superuser 

Re: [Arches] Arches 5.0 Docker setup problems

[David Osborne]

Thanks, Ben: your comment helped me make some progress (to another error ;-)

The problem is that the Readme.md in the docker directory of the Github 
archesproject/arches repo (file last revised 17 months ago!) is inaccurate 
or vague when it says "Create a new folder for your custom Arches code" 
without specifying where. I created one alongside my cloned arches repo, 
with the result that the path to /docker-entrypoint-initdb.d/init.sql did 
not exist, so Docker created directories instead.

Having located the arches/install/init-unix.sql file referred to in the db 
service, I worked out that it's easiest to run docker-compose in the cloned 
arches directory itself, so I edited the docker-compose.yml file, and that 
file and the resulting output from docker-compose is in 
https://gist.github.com/daosborne/bc83812e33d9396d679af54f31565895 (with 
the lines from couchdb and elasticsearch edited out from the docker-compose 
log as not relevant).

It gets further but has some problems (partly because I want the default 
graphs and concepts installed):

   1. error setting up db (traceback starting line 133)
   2. database tables graphs, ontologies, concepts, d_value_types don't 
   exist (lines 222--648)
   3. db connection error during migrations (line 734)
   4. db column nodes.fieldname doesn't exist (lines 1123, 1185)

I had also tried setting a project name, as suggested in the Readme but 
abandoned that because the directory was missing from /web_root (and 
produces a spurious error message about the project name clashing with a 
Python module).

I hoped that using Docker would be a quick and easy way of getting a new 
Arches installation running but it seems it still has some rough edges. As 
I need to get some development done to make some slides for a conference 
paper, I think I'll go back to the tried and tested method of setting up a 
Vagrant VM!

Cheers
David

On Wednesday, 4 March 2020 00:33:38 UTC, Ben O'Steen wrote:
>
> Hi,
>
> I think the key error is on line 63 of the gist you posted. For some 
> reason, it is interpreting file bind mount as a directory and not as a 
> file. If I run the default docker compose file here, and start up the db 
> container, this is the log output I get:
>
> $  docker logs db
> Add rule to pg_hba: 0.0.0.0/0
> Add rule to pg_hba: replication replicator
> Setup master database
> 2020-03-04 00:28:12.848 UTC [25] LOG:  starting PostgreSQL 12.1 (Debian 
> 12.1-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 
> 8.3.0, 64-bit
> 2020-03-04 00:28:12.849 UTC [25] LOG:  listening on IPv4 address 
> "127.0.0.1", port 5432
> 2020-03-04 00:28:12.855 UTC [25] LOG:  listening on Unix socket 
> "/var/run/postgresql/.s.PGSQL.5432"
> 2020-03-04 00:28:12.928 UTC [36] LOG:  database system was shut down at 
> 2019-12-03 16:35:30 UTC
> 2020-03-04 00:28:12.931 UTC [37] postgres@postgres FATAL:  the database 
> system is starting up
> psql: error: could not connect to server: FATAL:  the database system is 
> starting up
> 2020-03-04 00:28:12.948 UTC [25] LOG:  database system is ready to accept 
> connections
>   List of databases
>Name|  Owner   | Encoding | Collate |  Ctype  |   Access privileges
>
> ---+--+--+-+-+---
>  postgres  | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
>  template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres 
>  +
>|  |  | | | 
> postgres=CTc/postgres
>  template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres 
>  +
>|  |  | | | 
> postgres=CTc/postgres
> (3 rows)
>
> postgres ready
> Setup postgres User:Password
> Creating superuser postgres
> ALTER ROLE
> Creating replication user replicator
> CREATE ROLE
> postgres db already exists
>   List of databases
>Name|  Owner   | Encoding | Collate |  Ctype  |   Access privileges
>
> ---+--+--+-+-+---
>  postgres  | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
>  template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres 
>  +
>|  |  | | | 
> postgres=CTc/postgres
>  template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres 
>  +
>|  |  | | | 
> postgres=CTc/postgres
> (3 rows)
>
> /docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
> CREATE DATABASE
> UPDATE 1
> You are now connected to database "template_postgis" as user "postgres".
> CREATE EXTENSION
> GRANT
> GRANT
> GRANT
> CREATE EXTENSION
>
> 2020-03-04 00:28:16.764 UTC [25] LOG:  received smart shutdown request
> 2020-03-04 00:28:16.783 UTC [25] LOG:  background worker "logical 
> replication launcher" (PID 44) exited with exit code 1
> 2020-03-04 00:28:16.791 UTC [39] LOG:  shutting down
> 

Re: [Arches] Arches 5.0 Docker setup problems

[Ben O'Steen]

Hi,

I think the key error is on line 63 of the gist you posted. For some
reason, it is interpreting file bind mount as a directory and not as a
file. If I run the default docker compose file here, and start up the db
container, this is the log output I get:

$  docker logs db
Add rule to pg_hba: 0.0.0.0/0
Add rule to pg_hba: replication replicator
Setup master database
2020-03-04 00:28:12.848 UTC [25] LOG:  starting PostgreSQL 12.1 (Debian
12.1-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6)
8.3.0, 64-bit
2020-03-04 00:28:12.849 UTC [25] LOG:  listening on IPv4 address
"127.0.0.1", port 5432
2020-03-04 00:28:12.855 UTC [25] LOG:  listening on Unix socket
"/var/run/postgresql/.s.PGSQL.5432"
2020-03-04 00:28:12.928 UTC [36] LOG:  database system was shut down at
2019-12-03 16:35:30 UTC
2020-03-04 00:28:12.931 UTC [37] postgres@postgres FATAL:  the database
system is starting up
psql: error: could not connect to server: FATAL:  the database system is
starting up
2020-03-04 00:28:12.948 UTC [25] LOG:  database system is ready to accept
connections
  List of databases
   Name|  Owner   | Encoding | Collate |  Ctype  |   Access privileges
---+--+--+-+-+---
 postgres  | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
 template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres
 +
   |  |  | | | postgres=CTc/postgres
 template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres
 +
   |  |  | | | postgres=CTc/postgres
(3 rows)

postgres ready
Setup postgres User:Password
Creating superuser postgres
ALTER ROLE
Creating replication user replicator
CREATE ROLE
postgres db already exists
  List of databases
   Name|  Owner   | Encoding | Collate |  Ctype  |   Access privileges
---+--+--+-+-+---
 postgres  | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
 template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres
 +
   |  |  | | | postgres=CTc/postgres
 template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres
 +
   |  |  | | | postgres=CTc/postgres
(3 rows)

/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
CREATE DATABASE
UPDATE 1
You are now connected to database "template_postgis" as user "postgres".
CREATE EXTENSION
GRANT
GRANT
GRANT
CREATE EXTENSION

2020-03-04 00:28:16.764 UTC [25] LOG:  received smart shutdown request
2020-03-04 00:28:16.783 UTC [25] LOG:  background worker "logical
replication launcher" (PID 44) exited with exit code 1
2020-03-04 00:28:16.791 UTC [39] LOG:  shutting down
2020-03-04 00:28:16.866 UTC [25] LOG:  database system is shut down
Postgres initialisation process completed  restarting in foreground


Why it is different for you I cannot tell right now. I'm using
docker-compose version 1.24.1 (docker engine 19.03.5).

Ben

On Tue, 3 Mar 2020 at 15:13, David Osborne 
wrote:

> I need a new Arches installation to develop some stuff for a conference
> presentation, so I thought I'd take the opportunity to not only use the
> latest version but also run it under Docker.
>
> Following the instructions at
> https://github.com/archesproject/arches/blob/master/docker/Readme.md, I
> got as far as Quick Start step 6, docker-compose up, but there were
> problems creating the database for my *archesdemo* project.
>
> I've uploaded my docker-compose.yml file and the results of running 
> docker-compose
> up into a gist at
> https://gist.github.com/daosborne/cd501c56dc117e066b4f118f15af7e57 to
> show the problem. Is it obvious what I've missed or is wrong?
>
> I'm running Docker Desktop 2.2.0.3 on MacOS 10.14.6 Mojave.
>
> Thanks!
> David
>
> --
> -- To post, send email to archesproject@googlegroups.com. To unsubscribe,
> send email to archesproject+unsubscr...@googlegroups.com. For more
> information, visit https://groups.google.com/d/forum/archesproject?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Arches Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to archesproject+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/archesproject/5f42603a-ca04-4b8f-8e90-92af93d20b40%40googlegroups.com
> 
> .
>

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To