Re: [Arches] Arches 5.0 Docker setup problems

[Ben O'Steen]

Im going to guess that the elasticsearch container is not ready and the
migration script fails when it attempts to run the first time through and
Arches ends up in a bad state which it generally can never recover from.

docker-compose down -v

This will kill and delete all the badly formed docker containers created by
a failed initial run.

docker-compose up --no-start

This will initialize the containers and network for the stack.

docker-compose start elasticsearch db

This will start the elasticsearch instance and the database.

docker logs -f elasticsearch

This will let you watch what it is doing. Ctrl-c to get back to normal.
I've found that if you run antivirus checkers and have some network
slowness, it can take a while (30s to 1min) for elasticsearch to actually
be ready. It sadly lies on its health check command and reports "green"
while it is still doing its random clustering things.

docker-compose up

After elasticsearch has finished doing whatever it does on start up that
takes so long, run this and watch to make sure there are no connection
errors.


Ben

On Thu, 5 Mar 2020, 07:44 David Osborne, 
wrote:

> Thanks, Ben: your comment helped me make some progress (to another error
> ;-)
>
> The problem is that the Readme.md in the docker directory of the Github
> archesproject/arches repo (file last revised 17 months ago!) is inaccurate
> or vague when it says "Create a new folder for your custom Arches code"
> without specifying where. I created one alongside my cloned arches repo,
> with the result that the path to /docker-entrypoint-initdb.d/init.sql did
> not exist, so Docker created directories instead.
>
> Having located the arches/install/init-unix.sql file referred to in the db
> service, I worked out that it's easiest to run docker-compose in the cloned
> arches directory itself, so I edited the docker-compose.yml file, and that
> file and the resulting output from docker-compose is in
> https://gist.github.com/daosborne/bc83812e33d9396d679af54f31565895 (with
> the lines from couchdb and elasticsearch edited out from the docker-compose
> log as not relevant).
>
> It gets further but has some problems (partly because I want the default
> graphs and concepts installed):
>
>1. error setting up db (traceback starting line 133)
>2. database tables graphs, ontologies, concepts, d_value_types don't
>exist (lines 222--648)
>3. db connection error during migrations (line 734)
>4. db column nodes.fieldname doesn't exist (lines 1123, 1185)
>
> I had also tried setting a project name, as suggested in the Readme but
> abandoned that because the directory was missing from /web_root (and
> produces a spurious error message about the project name clashing with a
> Python module).
>
> I hoped that using Docker would be a quick and easy way of getting a new
> Arches installation running but it seems it still has some rough edges. As
> I need to get some development done to make some slides for a conference
> paper, I think I'll go back to the tried and tested method of setting up a
> Vagrant VM!
>
> Cheers
> David
>
> On Wednesday, 4 March 2020 00:33:38 UTC, Ben O'Steen wrote:
>>
>> Hi,
>>
>> I think the key error is on line 63 of the gist you posted. For some
>> reason, it is interpreting file bind mount as a directory and not as a
>> file. If I run the default docker compose file here, and start up the db
>> container, this is the log output I get:
>>
>> $  docker logs db
>> Add rule to pg_hba: 0.0.0.0/0
>> Add rule to pg_hba: replication replicator
>> Setup master database
>> 2020-03-04 00:28:12.848 UTC [25] LOG:  starting PostgreSQL 12.1 (Debian
>> 12.1-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6)
>> 8.3.0, 64-bit
>> 2020-03-04 00:28:12.849 UTC [25] LOG:  listening on IPv4 address
>> "127.0.0.1", port 5432
>> 2020-03-04 00:28:12.855 UTC [25] LOG:  listening on Unix socket
>> "/var/run/postgresql/.s.PGSQL.5432"
>> 2020-03-04 00:28:12.928 UTC [36] LOG:  database system was shut down at
>> 2019-12-03 16:35:30 UTC
>> 2020-03-04 00:28:12.931 UTC [37] postgres@postgres FATAL:  the database
>> system is starting up
>> psql: error: could not connect to server: FATAL:  the database system is
>> starting up
>> 2020-03-04 00:28:12.948 UTC [25] LOG:  database system is ready to accept
>> connections
>>   List of databases
>>Name|  Owner   | Encoding | Collate |  Ctype  |   Access privileges
>>
>> ---+--+--+-+-+---
>>  postgres  | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
>>  template0 | postgres | UTF8 | C.UTF-

Re: [Arches] Arches 5.0 Docker setup problems

[Ben O'Steen]

Hi,

I think the key error is on line 63 of the gist you posted. For some
reason, it is interpreting file bind mount as a directory and not as a
file. If I run the default docker compose file here, and start up the db
container, this is the log output I get:

$  docker logs db
Add rule to pg_hba: 0.0.0.0/0
Add rule to pg_hba: replication replicator
Setup master database
2020-03-04 00:28:12.848 UTC [25] LOG:  starting PostgreSQL 12.1 (Debian
12.1-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6)
8.3.0, 64-bit
2020-03-04 00:28:12.849 UTC [25] LOG:  listening on IPv4 address
"127.0.0.1", port 5432
2020-03-04 00:28:12.855 UTC [25] LOG:  listening on Unix socket
"/var/run/postgresql/.s.PGSQL.5432"
2020-03-04 00:28:12.928 UTC [36] LOG:  database system was shut down at
2019-12-03 16:35:30 UTC
2020-03-04 00:28:12.931 UTC [37] postgres@postgres FATAL:  the database
system is starting up
psql: error: could not connect to server: FATAL:  the database system is
starting up
2020-03-04 00:28:12.948 UTC [25] LOG:  database system is ready to accept
connections
  List of databases
   Name|  Owner   | Encoding | Collate |  Ctype  |   Access privileges
---+--+--+-+-+---
 postgres  | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
 template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres
 +
   |  |  | | | postgres=CTc/postgres
 template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres
 +
   |  |  | | | postgres=CTc/postgres
(3 rows)

postgres ready
Setup postgres User:Password
Creating superuser postgres
ALTER ROLE
Creating replication user replicator
CREATE ROLE
postgres db already exists
  List of databases
   Name|  Owner   | Encoding | Collate |  Ctype  |   Access privileges
---+--+--+-+-+---
 postgres  | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
 template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres
 +
   |  |  | | | postgres=CTc/postgres
 template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres
 +
   |  |  | | | postgres=CTc/postgres
(3 rows)

/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/init.sql
CREATE DATABASE
UPDATE 1
You are now connected to database "template_postgis" as user "postgres".
CREATE EXTENSION
GRANT
GRANT
GRANT
CREATE EXTENSION

2020-03-04 00:28:16.764 UTC [25] LOG:  received smart shutdown request
2020-03-04 00:28:16.783 UTC [25] LOG:  background worker "logical
replication launcher" (PID 44) exited with exit code 1
2020-03-04 00:28:16.791 UTC [39] LOG:  shutting down
2020-03-04 00:28:16.866 UTC [25] LOG:  database system is shut down
Postgres initialisation process completed  restarting in foreground


Why it is different for you I cannot tell right now. I'm using
docker-compose version 1.24.1 (docker engine 19.03.5).

Ben

On Tue, 3 Mar 2020 at 15:13, David Osborne 
wrote:

> I need a new Arches installation to develop some stuff for a conference
> presentation, so I thought I'd take the opportunity to not only use the
> latest version but also run it under Docker.
>
> Following the instructions at
> https://github.com/archesproject/arches/blob/master/docker/Readme.md, I
> got as far as Quick Start step 6, docker-compose up, but there were
> problems creating the database for my *archesdemo* project.
>
> I've uploaded my docker-compose.yml file and the results of running 
> docker-compose
> up into a gist at
> https://gist.github.com/daosborne/cd501c56dc117e066b4f118f15af7e57 to
> show the problem. Is it obvious what I've missed or is wrong?
>
> I'm running Docker Desktop 2.2.0.3 on MacOS 10.14.6 Mojave.
>
> Thanks!
> David
>
> --
> -- To post, send email to archesproject@googlegroups.com. To unsubscribe,
> send email to archesproject+unsubscr...@googlegroups.com. For more
> information, visit https://groups.google.com/d/forum/archesproject?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Arches Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to archesproject+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/archesproject/5f42603a-ca04-4b8f-8e90-92af93d20b40%40googlegroups.com
> 
> .
>

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To 

Re: [Arches] Arches v4 Windows Installation with pip

[Ben O'Steen]

Putting aside whether this code should be run or not, the error is a
parsing error concerning the python 2 style of string formatting when run
within python 3 which no longer supports this way of print formating. print
is a function in python 3 and should be in the form print(...) not print
""

Are you sure this is being run within a python 2.7 virtual environment?

Ben

On Tue, 16 Jul 2019 at 14:58, Alexei Peters  wrote:

> Hi Anthony,
> I just did a quick check and something strange seems to be happening with
> your install.  The line of code you're hitting shouldn't even be run as far
> as I can tell.
> Maybe Adam Cox can lend his thoughts to this.
> Cheers,
> Alexei
>
> Director of Web Development - Farallon Geographics, Inc. - 971.227.3173
>
>
> On Tue, Jul 16, 2019 at 12:05 PM Anthony Bronzo 
> wrote:
>
>> Hi,
>> I'm having the same error 2 trying to get Arches 4.4.2 onto my Windows 10
>> machine. It sounds like Arches doesn't use Shapely anymore so I don't think
>> its that, any ideas?
>>
>> This is what happens when I run "pip install arches --no-binary :all:" in
>> my virtual environment:
>>
>> Collecting arches
>>   Using cached
>> https://files.pythonhosted.org/packages/2c/6d/eff4016832a7cf17508b887711d105d36d809a1ac4a0c88e3e9166b22b3a/arches-4.4.2.tar.gz
>> ERROR: Complete output from command python setup.py egg_info:
>> ERROR: Traceback (most recent call last):
>>   File "", line 1, in 
>>   File
>> "C:\Users\funb4\AppData\Local\Temp\pip-install-xsd9_7_d\arches\setup.py",
>> line 15, in 
>> version = __import__('arches').__version__
>>   File
>> "C:\Users\funb4\AppData\Local\Temp\pip-install-xsd9_7_d\arches\arches\__init__.py",
>> line 1, in 
>> from arches.setup import get_version
>>   File
>> "C:\Users\funb4\AppData\Local\Temp\pip-install-xsd9_7_d\arches\arches\setup.py",
>> line 84
>> print "Downloading: %s Bytes: %s" % (file_name, file_size)
>> ^
>> SyntaxError: invalid syntax
>> 
>> ERROR: Command "python setup.py egg_info" failed with error code 1 in
>> C:\Users\funb4\AppData\Local\Temp\pip-install-xsd9_7_d\arches\
>>
>> Thanks,
>> Anthony
>>
>> On Wednesday, April 26, 2017 at 8:26:48 PM UTC-7, Adam Cox wrote:
>>>
>>> Hi Susan, Cyrus is right, error 1 is ok, and it does sound like Shapely
>>> is the culprit here. As noted here
>>> 
>>>  (though
>>> it should be noted in other parts of the documentation as well), you should
>>> install Shapely in after creating your virtual environment, but before pip
>>> installing Arches into it. Try that out and let us know how it goes.
>>>
>>> Adam
>>>
>>>
>>> On Wed, Apr 26, 2017 at 7:13 PM, Cyrus Hiatt  wrote:
>>>
 Hi Susan -

 I think the installation will proceed despite Error 1.

 Regarding Error 2 - Did you install the dependencies using the OSGeo4W
 installer? If so, is 'C:\OSGeo4W64\bin' added to your systems PATH
 environment variable?

 If your PATH is correct, perhaps the issue is with the Shapely
 installation itself?  It looks like you need to install that prior to
 installing Arches on Windows.  If you didn't catch it in the docs, there is
 a short section on installing Shapely for Windows:
 https://github.com/archesproject/arches/wiki/Installing-Dependencies-on-Windows

 - Cyrus




 On Wed, Apr 26, 2017 at 3:56 PM, Susan Penacho 
 wrote:

> Hi,
> After I ran into issues trying to install arches on my windows 10
> machine with the repository, I switched to the pip installation using the
> online documentation (
> https://arches4.readthedocs.io/en/latest/installation/#installing-arches
> ).
>
> Using the pip installation steps on the installation documentation I
> was able to do steps 1 to 3 and create the virtual environment. When I
> tried to install arches in step 4 I got back two errors:
>
> Error 1:
>
>   Requirement already up-to-date: six>=1.6.0 in
> c:\projects\env\lib\site-packages (from setuptools)
>
>   Requirement already up-to-date: pyparsing in
> c:\projects\env\lib\site-packages (from packaging>=16.8->setuptools)
>
>   Could not open requirements file: [Errno 2] No such file or
> directory:
> 'c:\\projects\\env\\Lib\\site-packages\\arches\\install\\requirements.txt'
>
>   error: [Errno 2] No such file or directory:
> 'c:\\projects\\env\\Lib\\site-packages\\arches\\install\\django_overrides\\admin.py'
>
>   
>
>   Failed building wheel for arches
>
>   Running setup.py clean for arches
>
> Failed to build arches
>
> Installing collected packages: arches
>
>   Running setup.py install for arches ... /
>
>
> Error2:

Re: [Arches] Docker-compose image have 404 for all static assets.

[Ben O'Steen]

https://github.com/archesproject/arches/pull/4801
I've put my small tweak as a PR, if you wanted to check it out Vincent and
test yourself!

Ben

On Wed, 24 Apr 2019 at 10:44, Ben O'Steen  wrote:

> Rob G had similar issues running the test suite, but had an extra fail as
> well! Not sure what is going on but I think some tests need looking at
> possibly!
>
> Ben
>
> On Wed, 24 Apr 2019 at 05:13, Vincent Meijer 
> wrote:
>
>> Nice!
>> Sounds like that is a regular assertion error and independent of the
>> environment, or does the test succeed locally?
>>
>> On Tue, Apr 23, 2019 at 6:46 PM Ben O'Steen  wrote:
>>
>>> Good news, Bad news I'm afraid:
>>>
>>> Good news? the environment flag seems to have done the trick and the
>>> Elasticsearch container doesn't quit after running its checks.
>>>
>>> Bad news? There is a failing test:
>>>
>>> FAIL: Test bulk deleting of documents in Elasticsearch
>>> --
>>> Traceback (most recent call last):
>>> File "/web_root/arches/tests/search/search_tests.py", line 66, in
>>> test_bulk_delete
>>> self.assertEqual(se.count(index='test'), 10)
>>> AssertionError: 11 != 10
>>>
>>>
>>> https://cloud.docker.com/repository/registry-1.docker.io/benosteen/arches/builds/88c8a09d-9fa6-44f5-a86a-fc960ba2f327
>>>
>>>
>>>
>>> On Tue, 23 Apr 2019 at 08:23, Ben O'Steen  wrote:
>>>
>>>> That is an odd error, certainly works locally but not when docker hub
>>>> runs the test setup.
>>>>
>>>> One thing it could be is that if docker hub are using default EC2
>>>> instances, they have the vm.max_map_count set too low for Elasticsearch. It
>>>> is high enough that Elasticsearch will boot but not enough to stop it
>>>> erroring out after a very short period. I've been using 'sudo sysctl -w
>>>> vm.max_map_count=262144' on the host to avoid this which is why it is not
>>>> erroring for me during Elasticsearch's bootstrap checks.
>>>>
>>>> We should be able to disable the checks (turn them from hard errors to
>>>> soft warnings in the logs) by passing the ES container
>>>> "discovery.type=single-node" as an environment variable. I'll try this on
>>>> my docker hub account and see if that can fix it.
>>>>
>>>> Ben
>>>>
>>>> On Tue, 23 Apr 2019 at 03:12, Vincent Meijer 
>>>> wrote:
>>>>
>>>>> It seems there are problems with our automated docker builds, which is
>>>>> presumably the cause of the 4.4.1 tag not existing.
>>>>>
>>>>> Latest build of the master branch failed: the Elasticsearch endpoint
>>>>> cannot be reached during the unit tests.
>>>>> Failed to establish a new connection: [Errno -2] Name or service not
>>>>> known)
>>>>>
>>>>> https://cloud.docker.com/u/archesproject/repository/registry-1.docker.io/archesproject/arches/builds/ca27a312-4d02-4e1b-b65c-2f34cb22ef79
>>>>>
>>>>>
>>>>> Not sure why this is, as the Elasticsearch container seems to be up
>>>>> and running during the test.
>>>>>
>>>>> Full command: run_tests
>>>>> Command: run_tests
>>>>> Testing if database server is up...
>>>>> Database server is up
>>>>> Testing if Elasticsearch is up...
>>>>> Elasticsearch is up
>>>>>
>>>>> However, the 4.4.1 build does not seem be have been triggered at all,
>>>>> and I'm at a loss as to why this hasn't happened...
>>>>> Possibly someone with more permissions than me could check the webhook
>>>>> settings in the Github repo? :)
>>>>>
>>>>> Vincent
>>>>>
>>>>> On Fri, Apr 19, 2019 at 12:40 AM Ben O'Steen 
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> - As for Docker 4.4.1 not being put onto Docker Hub, I'm not sure why
>>>>>> that is. I've been making my own images of the codebase in lieu of that.
>>>>>>
>>>>>> - I don't get the System settings errors you mention, though I am
>>>>>> using Arches based on a much more recent codebase (I created my Arches 
>>>>>> base
>>>>>> image last week or so from the master bra

Re: [Arches] Docker-compose image have 404 for all static assets.

[Ben O'Steen]

Rob G had similar issues running the test suite, but had an extra fail as
well! Not sure what is going on but I think some tests need looking at
possibly!

Ben

On Wed, 24 Apr 2019 at 05:13, Vincent Meijer 
wrote:

> Nice!
> Sounds like that is a regular assertion error and independent of the
> environment, or does the test succeed locally?
>
> On Tue, Apr 23, 2019 at 6:46 PM Ben O'Steen  wrote:
>
>> Good news, Bad news I'm afraid:
>>
>> Good news? the environment flag seems to have done the trick and the
>> Elasticsearch container doesn't quit after running its checks.
>>
>> Bad news? There is a failing test:
>>
>> FAIL: Test bulk deleting of documents in Elasticsearch
>> --
>> Traceback (most recent call last):
>> File "/web_root/arches/tests/search/search_tests.py", line 66, in
>> test_bulk_delete
>> self.assertEqual(se.count(index='test'), 10)
>> AssertionError: 11 != 10
>>
>>
>> https://cloud.docker.com/repository/registry-1.docker.io/benosteen/arches/builds/88c8a09d-9fa6-44f5-a86a-fc960ba2f327
>>
>>
>>
>> On Tue, 23 Apr 2019 at 08:23, Ben O'Steen  wrote:
>>
>>> That is an odd error, certainly works locally but not when docker hub
>>> runs the test setup.
>>>
>>> One thing it could be is that if docker hub are using default EC2
>>> instances, they have the vm.max_map_count set too low for Elasticsearch. It
>>> is high enough that Elasticsearch will boot but not enough to stop it
>>> erroring out after a very short period. I've been using 'sudo sysctl -w
>>> vm.max_map_count=262144' on the host to avoid this which is why it is not
>>> erroring for me during Elasticsearch's bootstrap checks.
>>>
>>> We should be able to disable the checks (turn them from hard errors to
>>> soft warnings in the logs) by passing the ES container
>>> "discovery.type=single-node" as an environment variable. I'll try this on
>>> my docker hub account and see if that can fix it.
>>>
>>> Ben
>>>
>>> On Tue, 23 Apr 2019 at 03:12, Vincent Meijer 
>>> wrote:
>>>
>>>> It seems there are problems with our automated docker builds, which is
>>>> presumably the cause of the 4.4.1 tag not existing.
>>>>
>>>> Latest build of the master branch failed: the Elasticsearch endpoint
>>>> cannot be reached during the unit tests.
>>>> Failed to establish a new connection: [Errno -2] Name or service not
>>>> known)
>>>>
>>>> https://cloud.docker.com/u/archesproject/repository/registry-1.docker.io/archesproject/arches/builds/ca27a312-4d02-4e1b-b65c-2f34cb22ef79
>>>>
>>>>
>>>> Not sure why this is, as the Elasticsearch container seems to be up and
>>>> running during the test.
>>>>
>>>> Full command: run_tests
>>>> Command: run_tests
>>>> Testing if database server is up...
>>>> Database server is up
>>>> Testing if Elasticsearch is up...
>>>> Elasticsearch is up
>>>>
>>>> However, the 4.4.1 build does not seem be have been triggered at all,
>>>> and I'm at a loss as to why this hasn't happened...
>>>> Possibly someone with more permissions than me could check the webhook
>>>> settings in the Github repo? :)
>>>>
>>>> Vincent
>>>>
>>>> On Fri, Apr 19, 2019 at 12:40 AM Ben O'Steen  wrote:
>>>>
>>>>>
>>>>> - As for Docker 4.4.1 not being put onto Docker Hub, I'm not sure why
>>>>> that is. I've been making my own images of the codebase in lieu of that.
>>>>>
>>>>> - I don't get the System settings errors you mention, though I am
>>>>> using Arches based on a much more recent codebase (I created my Arches 
>>>>> base
>>>>> image last week or so from the master branch).
>>>>>
>>>>> Ben
>>>>>
>>>>> On Tue, 16 Apr 2019 at 14:39, Matthias Bussonnier <
>>>>> bussonniermatth...@gmail.com> wrote:
>>>>>
>>>>>> Thanks Ben,
>>>>>>
>>>>>> That was useful, I guess I got confused between DEV mode 8000 and
>>>>>> PROD port 80, I got it to work now.
>>>>>>
>>>>>> 2 followup questions; now that I can see the website and login:
>>>>>>   - I see on dockerhub t

Re: [Arches] Docker-compose image have 404 for all static assets.

[Ben O'Steen]

Good news, Bad news I'm afraid:

Good news? the environment flag seems to have done the trick and the
Elasticsearch container doesn't quit after running its checks.

Bad news? There is a failing test:

FAIL: Test bulk deleting of documents in Elasticsearch
--
Traceback (most recent call last):
File "/web_root/arches/tests/search/search_tests.py", line 66, in
test_bulk_delete
self.assertEqual(se.count(index='test'), 10)
AssertionError: 11 != 10

https://cloud.docker.com/repository/registry-1.docker.io/benosteen/arches/builds/88c8a09d-9fa6-44f5-a86a-fc960ba2f327



On Tue, 23 Apr 2019 at 08:23, Ben O'Steen  wrote:

> That is an odd error, certainly works locally but not when docker hub runs
> the test setup.
>
> One thing it could be is that if docker hub are using default EC2
> instances, they have the vm.max_map_count set too low for Elasticsearch. It
> is high enough that Elasticsearch will boot but not enough to stop it
> erroring out after a very short period. I've been using 'sudo sysctl -w
> vm.max_map_count=262144' on the host to avoid this which is why it is not
> erroring for me during Elasticsearch's bootstrap checks.
>
> We should be able to disable the checks (turn them from hard errors to
> soft warnings in the logs) by passing the ES container
> "discovery.type=single-node" as an environment variable. I'll try this on
> my docker hub account and see if that can fix it.
>
> Ben
>
> On Tue, 23 Apr 2019 at 03:12, Vincent Meijer 
> wrote:
>
>> It seems there are problems with our automated docker builds, which is
>> presumably the cause of the 4.4.1 tag not existing.
>>
>> Latest build of the master branch failed: the Elasticsearch endpoint
>> cannot be reached during the unit tests.
>> Failed to establish a new connection: [Errno -2] Name or service not
>> known)
>>
>> https://cloud.docker.com/u/archesproject/repository/registry-1.docker.io/archesproject/arches/builds/ca27a312-4d02-4e1b-b65c-2f34cb22ef79
>>
>>
>> Not sure why this is, as the Elasticsearch container seems to be up and
>> running during the test.
>>
>> Full command: run_tests
>> Command: run_tests
>> Testing if database server is up...
>> Database server is up
>> Testing if Elasticsearch is up...
>> Elasticsearch is up
>>
>> However, the 4.4.1 build does not seem be have been triggered at all, and
>> I'm at a loss as to why this hasn't happened...
>> Possibly someone with more permissions than me could check the webhook
>> settings in the Github repo? :)
>>
>> Vincent
>>
>> On Fri, Apr 19, 2019 at 12:40 AM Ben O'Steen  wrote:
>>
>>>
>>> - As for Docker 4.4.1 not being put onto Docker Hub, I'm not sure why
>>> that is. I've been making my own images of the codebase in lieu of that.
>>>
>>> - I don't get the System settings errors you mention, though I am using
>>> Arches based on a much more recent codebase (I created my Arches base image
>>> last week or so from the master branch).
>>>
>>> Ben
>>>
>>> On Tue, 16 Apr 2019 at 14:39, Matthias Bussonnier <
>>> bussonniermatth...@gmail.com> wrote:
>>>
>>>> Thanks Ben,
>>>>
>>>> That was useful, I guess I got confused between DEV mode 8000 and PROD
>>>> port 80, I got it to work now.
>>>>
>>>> 2 followup questions; now that I can see the website and login:
>>>>   - I see on dockerhub that 4.4.1 has not been published; is that
>>>> expected ?
>>>>   - once logged-in I can't seem to access any of the 3 "System
>>>> settings" pages, they give me a 500,  Is that expected ? (I haven't managed
>>>> to get both my deployment working and django to spit out a traceback,
>>>> otherwise I would give up more info)
>>>>
>>>> I'm happy to start this as another thread if this is better suited.
>>>> --
>>>> Matthias
>>>>
>>>>
>>>>
>>>> On Tue, 16 Apr 2019 at 10:12, Ben O'Steen  wrote:
>>>>
>>>>> I should also make it clear that all of what I mentioned above is the
>>>>> default behaviour and doesn't need you to do anything more to make it 
>>>>> work.
>>>>>
>>>> --
>>> -- To post, send email to archesproject@googlegroups.com. To
>>> unsubscribe, send email to archesproject+unsubscr...@googlegroups.com.
>>> For more information, visit
>>> https://groups.google.com/d/forum/archesprojec

Re: [Arches] Docker-compose image have 404 for all static assets.

[Ben O'Steen]

That is an odd error, certainly works locally but not when docker hub runs
the test setup.

One thing it could be is that if docker hub are using default EC2
instances, they have the vm.max_map_count set too low for Elasticsearch. It
is high enough that Elasticsearch will boot but not enough to stop it
erroring out after a very short period. I've been using 'sudo sysctl -w
vm.max_map_count=262144' on the host to avoid this which is why it is not
erroring for me during Elasticsearch's bootstrap checks.

We should be able to disable the checks (turn them from hard errors to soft
warnings in the logs) by passing the ES container
"discovery.type=single-node" as an environment variable. I'll try this on
my docker hub account and see if that can fix it.

Ben

On Tue, 23 Apr 2019 at 03:12, Vincent Meijer 
wrote:

> It seems there are problems with our automated docker builds, which is
> presumably the cause of the 4.4.1 tag not existing.
>
> Latest build of the master branch failed: the Elasticsearch endpoint
> cannot be reached during the unit tests.
> Failed to establish a new connection: [Errno -2] Name or service not known)
>
>
> https://cloud.docker.com/u/archesproject/repository/registry-1.docker.io/archesproject/arches/builds/ca27a312-4d02-4e1b-b65c-2f34cb22ef79
>
>
> Not sure why this is, as the Elasticsearch container seems to be up and
> running during the test.
>
> Full command: run_tests
> Command: run_tests
> Testing if database server is up...
> Database server is up
> Testing if Elasticsearch is up...
> Elasticsearch is up
>
> However, the 4.4.1 build does not seem be have been triggered at all, and
> I'm at a loss as to why this hasn't happened...
> Possibly someone with more permissions than me could check the webhook
> settings in the Github repo? :)
>
> Vincent
>
> On Fri, Apr 19, 2019 at 12:40 AM Ben O'Steen  wrote:
>
>>
>> - As for Docker 4.4.1 not being put onto Docker Hub, I'm not sure why
>> that is. I've been making my own images of the codebase in lieu of that.
>>
>> - I don't get the System settings errors you mention, though I am using
>> Arches based on a much more recent codebase (I created my Arches base image
>> last week or so from the master branch).
>>
>> Ben
>>
>> On Tue, 16 Apr 2019 at 14:39, Matthias Bussonnier <
>> bussonniermatth...@gmail.com> wrote:
>>
>>> Thanks Ben,
>>>
>>> That was useful, I guess I got confused between DEV mode 8000 and PROD
>>> port 80, I got it to work now.
>>>
>>> 2 followup questions; now that I can see the website and login:
>>>   - I see on dockerhub that 4.4.1 has not been published; is that
>>> expected ?
>>>   - once logged-in I can't seem to access any of the 3 "System settings"
>>> pages, they give me a 500,  Is that expected ? (I haven't managed to get
>>> both my deployment working and django to spit out a traceback, otherwise I
>>> would give up more info)
>>>
>>> I'm happy to start this as another thread if this is better suited.
>>> --
>>> Matthias
>>>
>>>
>>>
>>> On Tue, 16 Apr 2019 at 10:12, Ben O'Steen  wrote:
>>>
>>>> I should also make it clear that all of what I mentioned above is the
>>>> default behaviour and doesn't need you to do anything more to make it work.
>>>>
>>> --
>> -- To post, send email to archesproject@googlegroups.com. To
>> unsubscribe, send email to archesproject+unsubscr...@googlegroups.com.
>> For more information, visit
>> https://groups.google.com/d/forum/archesproject?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Arches Project" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to archesproject+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> -- To post, send email to archesproject@googlegroups.com. To unsubscribe,
> send email to archesproject+unsubscr...@googlegroups.com. For more
> information, visit https://groups.google.com/d/forum/archesproject?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Arches Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to archesproject+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Arches] Docker-compose image have 404 for all static assets.

[Ben O'Steen]

- As for Docker 4.4.1 not being put onto Docker Hub, I'm not sure why that
is. I've been making my own images of the codebase in lieu of that.

- I don't get the System settings errors you mention, though I am using
Arches based on a much more recent codebase (I created my Arches base image
last week or so from the master branch).

Ben

On Tue, 16 Apr 2019 at 14:39, Matthias Bussonnier <
bussonniermatth...@gmail.com> wrote:

> Thanks Ben,
>
> That was useful, I guess I got confused between DEV mode 8000 and PROD
> port 80, I got it to work now.
>
> 2 followup questions; now that I can see the website and login:
>   - I see on dockerhub that 4.4.1 has not been published; is that expected
> ?
>   - once logged-in I can't seem to access any of the 3 "System settings"
> pages, they give me a 500,  Is that expected ? (I haven't managed to get
> both my deployment working and django to spit out a traceback, otherwise I
> would give up more info)
>
> I'm happy to start this as another thread if this is better suited.
> --
> Matthias
>
>
>
> On Tue, 16 Apr 2019 at 10:12, Ben O'Steen  wrote:
>
>> I should also make it clear that all of what I mentioned above is the
>> default behaviour and doesn't need you to do anything more to make it work.
>>
>

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Arches] Docker-compose image have 404 for all static assets.

[Ben O'Steen]

I should also make it clear that all of what I mentioned above is the
default behaviour and doesn't need you to do anything more to make it work.

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Arches] Docker-compose image have 404 for all static assets.

[Ben O'Steen]

How the static files are served depends on a few things. What is the
DJANGO_MODE variable set to in the docker-compose.yml that you are using?
(NB if you also have an .env file in that directory, that may be
overwriting this issue)

If it is PROD, then the command to collect static files should copy them
all to the directory "/static_root" by default. It will look for them in
the directories specified in the STATICFILES_DIRS list in settings.py. This
should be marked as a persisted volume in your docker compose
configuration:   "  - arches-static:/static_root". The nginx container also
has access to this volume, which it mounts as '/www/static' and serves as "
http://host/static/...;. The Arches instance will be available from
http://host:80/ (eg http://localhost ). The Arches instance will be served
from the arches container on port 8000, but you should not be accessing
this directly.

If DJANGO_MODE is DEV, then the django runserver process is used to serve
the contents, and it will be available on port 8000. This will also serve
the static files as well, and you should be able to access the Arches
instance from http://host:8000 (eg http://localhost:8000).

Ben

On Tue, 16 Apr 2019 at 09:24, Matthias Bussonnier <
bussonniermatth...@gmail.com> wrote:

> Hello,
>
> I am trying to deploy Arches for our local institution, I'm trying to go
> through the instruction [1], on how to run in docker.
> Most instruction appear to work so far (a  couple of couchdb and
> elasticsearch warning which I'm going to assume are for first run,
> But when accessing the local install, I get greeted with 404 on any static
> resource situated under /media/.
>
> Arches does seem to properly copy assets :
>
> > arches   | Copying
> '/web_root/arches/arches/app/media/packages/font-awesome/README.md'
> (and alike),
>
> And does this the requests ( so it's likely not a nginx thing):
>
> > arches   | 172.21.0.1 - - [16/Apr/2019:11:18:40 -0500] "GET
> /media/css/unify.css HTTP/1.1" 404 93 "http://localhost:8000/;
>
>
> I'm i'm starting to run out of ideas of what could cause this, and would
> love some guidance – which I would be happy to contribute in the form of
> pull-requests again the main repository.
>
> I'm tried the master  (a20525c1828 as of April 11th) and 4.4.1, making
> sure to update docker-compose to point at my locally built docker image,
> and pruning docker volumes.
>
> Thanks,
> --
> 1:
> https://github.com/archesproject/arches/tree/master/docker#setting-up-your-own-arches-project
>
> --
> -- To post, send email to archesproject@googlegroups.com. To unsubscribe,
> send email to archesproject+unsubscr...@googlegroups.com. For more
> information, visit https://groups.google.com/d/forum/archesproject?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Arches Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to archesproject+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Arches] Falling at first hurdle, then the next, then the next.

[Ben O'Steen]

Sorry to hear that you have had trouble! If you just want to test it out, I
agree with Adam that the demo installation is a good option. The direct
link to that is http://v4demo.archesproject.org/

Ben

On Wed, 10 Apr 2019 at 12:23, Adam Cox  wrote:

> Hello, the error you found won't be especially difficult to overcome, but
> it does sound like perhaps you may like to just try out the demo
> installation that you can get to from the archesproject.org home page?
>
> To handle the error, you'll need to activate your virtual environment, as
> described in step 3 of the installation guide you linked to.
>
> Hope that helps!
> Adam
>
> On Wed, Apr 10, 2019 at 2:01 PM 'Researcher' via Arches Project <
> archesproject@googlegroups.com> wrote:
>
>> Hi all,
>> I've been struggling through trying to install Arches all day today
>> (historian background, not IT professional!) and I thought I was getting
>> somewhere reaching the point where it told me Arches had been installed (in
>> Windows command prompt) but even our IT support are unable to get very far
>> beyond that. Anyone got any advice? Is there a simpler way? I just wanted
>> to try out the software, not get a degree in coding before I had even seen
>> what it looked like.
>> I tried to run step 7 on the install guide:
>> https://arches.readthedocs.io/en/stable/installation/
>> but it says you need elasticsearch running. I tried to install
>> elasticsearch from the "easiest way" (dread to think what the hard way
>> looks like) as per below:
>>
>> The easiest way to install Elasticsearch is to use a command that comes
>> with Arches. Once you have installed Arches (either with pip or from the 
>> Arches
>> repo ), activate your virtual
>> environment, enter your app/project root directory (the one that contains
>> manage.py), run
>>
>> python manage.py es install
>>
>> Elasticsearch will be installed in the root folder. You can specify an
>> alternate destination for the installation by using the -d argument. For
>> example
>>
>> python manage.py es install -d C:\Projects
>>
>> will result in a new directory C:\Projects\elasticsearch-5.2.1 from
>> which you’ll be able to run Elasticsearch.
>>
>> I tried this and got the following:
>> (ENV) C:\Users\FIMNT\my_project>python manage.py es install
>> Traceback (most recent call last):
>>   File "manage.py", line 27, in 
>> from django.core.management import execute_from_command_line
>> ImportError: No module named django.core.management
>>
>> Obviously I or the software have done something wrong but I've spent the
>> entire day working like this with every single step having some kind of
>> problem and our IT support service is also struggling to see why it's
>> having so many issues.
>> Anyone able to guide us through in a simpler way, enlighten me past this
>> step and onto the next error or should I give up on Arches now seeing as
>> it's taken me a day and I can't even install it, yet alone set it up or
>> operate it?
>>
>> Thanks,
>>
>> --
>> -- To post, send email to archesproject@googlegroups.com. To
>> unsubscribe, send email to archesproject+unsubscr...@googlegroups.com.
>> For more information, visit
>> https://groups.google.com/d/forum/archesproject?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Arches Project" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to archesproject+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> -- To post, send email to archesproject@googlegroups.com. To unsubscribe,
> send email to archesproject+unsubscr...@googlegroups.com. For more
> information, visit https://groups.google.com/d/forum/archesproject?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Arches Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to archesproject+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Arches] Arches 4.4.1: Re-Index databases after upgrade to Ubuntu18.4

[Ben O'Steen]

Error from Arches container image built yesterday from master:

(ENV) root@06a7d6571621:/web_root/arches# python  manage.py es
index_database
Indexing concepts
Status: Passed, In Database: 490, Indexed: 490, Took: 0 seconds
Indexing resource type 'Group'
Traceback (most recent call last):
  File "manage.py", line 27, in 
execute_from_command_line(sys.argv)
  File
"/web_root/ENV/local/lib/python2.7/site-packages/django/core/management/__init__.py",
line 364, in execute_from_command_line
utility.execute()
  File
"/web_root/ENV/local/lib/python2.7/site-packages/django/core/management/__init__.py",
line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
  File
"/web_root/ENV/local/lib/python2.7/site-packages/django/core/management/base.py",
line 283, in run_from_argv
self.execute(*args, **cmd_options)
  File
"/web_root/ENV/local/lib/python2.7/site-packages/django/core/management/base.py",
line 330, in execute
output = self.handle(*args, **options)
  File "/web_root/arches/arches/management/commands/es.py", line 76, in
handle
index_database.index_db(clear_index=options['clear_index'],
batch_size=options['batch_size'])
  File "/web_root/arches/arches/app/utils/index_database.py", line 25, in
index_db
index_resources(clear_index=clear_index, batch_size=batch_size)
  File "/web_root/arches/arches/app/utils/index_database.py", line 44, in
index_resources
index_resources_by_type(resource_types, clear_index=clear_index,
batch_size=batch_size)
  File "/web_root/arches/arches/app/utils/index_database.py", line 84, in
index_resources_by_type
result_summary['indexed'] = se.count(index='resources', body=q.dsl)
  File "/web_root/arches/arches/app/search/search.py", line 214, in count
count = self.es.count(**kwargs)
  File
"/web_root/ENV/local/lib/python2.7/site-packages/elasticsearch/client/utils.py",
line 76, in _wrapped
return func(*args, params=params, **kwargs)
  File
"/web_root/ENV/local/lib/python2.7/site-packages/elasticsearch/client/__init__.py",
line 1112, in count
doc_type, '_count'), params=params, body=body)
  File
"/web_root/ENV/local/lib/python2.7/site-packages/elasticsearch/transport.py",
line 318, in perform_request
status, headers_response, data = connection.perform_request(method,
url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File
"/web_root/ENV/local/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py",
line 186, in perform_request
self._raise_error(response.status, raw_data)
  File
"/web_root/ENV/local/lib/python2.7/site-packages/elasticsearch/connection/base.py",
line 125, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code,
error_message, additional_info)
elasticsearch.exceptions.RequestError: RequestError(400,
u'parsing_exception', u'request does not support [_source]')

On Thu, 4 Apr 2019 at 13:22, Ben O'Steen  wrote:

> Ah, that is a good point. Yes I am on master (which still believes itself
> to be 4.4.1 in the logs hence me replying)
>
> On Thu, 4 Apr 2019 at 13:04, Adam Cox  wrote:
>
>> Hey all, isn't 4.4.1 ES 5.x and master branch right now is ES 6.2? That's
>> what has worked for me, and using the wrong ES version broke things...
>> though I'm not in a position to double this right now.
>>
>> Adam
>>
>> On Thu, Apr 4, 2019, 3:50 PM Ben O'Steen  wrote:
>>
>>>
>>> I am running it as docker containers so am just using the default
>>> elasticsearch container.
>>>
>>> On Thu, 4 Apr 2019, 11:34 ,  wrote:
>>>
>>>> Hi Ben,
>>>>
>>>> did you also have problems with Elasticsearch 6.6.2 not running because
>>>> of file/folder permissions?
>>>>
>>>> Jens
>>>>
>>>> On Thursday, 4 April 2019 19:12:12 UTC+1, Ben O'Steen wrote:
>>>>
>>>>> I can confirm I hit the same issue and error running 'python manage.py
>>>>> es index_database'. I hadn't filed an issue about it as I too had just
>>>>> upgraded ES in my environment too and wanted to eliminated operator error
>>>>> first. The command that seems to be the root of the issue was:
>>>>> 'result_summary['indexed'] = se.count(index='resources', body=q.dsl)' for
>>>>> me.
>>>>>
>>>>> Ben
>>>>>
>>>>> On Thu, 4 Apr 2019 at 10:32,  wrote:
>>>>>
>>>>>> Hi folks,
>>>>>>
>>>>>> I upgraded Arches to 4.4.1 which worked fine. Then I became bold and
>>>>>> upgraded my server to the latest ubuntu version 18.04 LTS.
>&

Re: [Arches] Arches 4.4.1: Re-Index databases after upgrade to Ubuntu18.4

[Ben O'Steen]

I am running it as docker containers so am just using the default
elasticsearch container.

On Thu, 4 Apr 2019, 11:34 ,  wrote:

> Hi Ben,
>
> did you also have problems with Elasticsearch 6.6.2 not running because of
> file/folder permissions?
>
> Jens
>
> On Thursday, 4 April 2019 19:12:12 UTC+1, Ben O'Steen wrote:
>
>> I can confirm I hit the same issue and error running 'python manage.py es
>> index_database'. I hadn't filed an issue about it as I too had just
>> upgraded ES in my environment too and wanted to eliminated operator error
>> first. The command that seems to be the root of the issue was:
>> 'result_summary['indexed'] = se.count(index='resources', body=q.dsl)' for
>> me.
>>
>> Ben
>>
>> On Thu, 4 Apr 2019 at 10:32,  wrote:
>>
>>> Hi folks,
>>>
>>> I upgraded Arches to 4.4.1 which worked fine. Then I became bold and
>>> upgraded my server to the latest ubuntu version 18.04 LTS.
>>> After a rough ride I (almost) got it to work.
>>> A major problem was Elasticsearch 6.6.2 not running because of
>>> insufficient rights.
>>> I solved this by giving read-write/execute permission to the entire
>>> folder (# chmod -R 777 /home/ubuntu/Projects/elasticsearch-6.6.2) - I know
>>> it's heavy handed, but it worked!
>>>
>>> However I did not succeed in re-indexing my existing project databases.
>>> python manage.py
>>> migrate - worked
>>> es delete_indexes- worked
>>> es setup_indexes - worked
>>> *es index_database* - FAILED
>>>
>>> The resulting error was:
>>>
>>>  File
>>> "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py",
>>> line 125, in _raise_error
>>> raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code,
>>> error_message, additional_info)
>>> elasticsearch.exceptions.RequestError: RequestError(400,
>>> u'parsing_exception', u'request does not support [_source]')
>>>
>>> Does anyone have an idea how to fix this and re-index existing databases?
>>>
>>> Cheers
>>> Jens
>>>
>>> --
>>> -- To post, send email to arches...@googlegroups.com. To unsubscribe,
>>> send email to arches...@googlegroups.com. For more information, visit
>>> https://groups.google.com/d/forum/archesproject?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Arches Project" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to arches...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> -- To post, send email to archesproject@googlegroups.com. To unsubscribe,
> send email to archesproject+unsubscr...@googlegroups.com. For more
> information, visit https://groups.google.com/d/forum/archesproject?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Arches Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to archesproject+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Arches] Arches 4.4.1: Re-Index databases after upgrade to Ubuntu18.4

[Ben O'Steen]

I can confirm I hit the same issue and error running 'python manage.py es
index_database'. I hadn't filed an issue about it as I too had just
upgraded ES in my environment too and wanted to eliminated operator error
first. The command that seems to be the root of the issue was:
'result_summary['indexed'] = se.count(index='resources', body=q.dsl)' for
me.

Ben

On Thu, 4 Apr 2019 at 10:32,  wrote:

> Hi folks,
>
> I upgraded Arches to 4.4.1 which worked fine. Then I became bold and
> upgraded my server to the latest ubuntu version 18.04 LTS.
> After a rough ride I (almost) got it to work.
> A major problem was Elasticsearch 6.6.2 not running because of
> insufficient rights.
> I solved this by giving read-write/execute permission to the entire folder
> (# chmod -R 777 /home/ubuntu/Projects/elasticsearch-6.6.2) - I know it's
> heavy handed, but it worked!
>
> However I did not succeed in re-indexing my existing project databases.
> python manage.py
> migrate - worked
> es delete_indexes- worked
> es setup_indexes - worked
> *es index_database* - FAILED
>
> The resulting error was:
>
>  File
> "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py",
> line 125, in _raise_error
> raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code,
> error_message, additional_info)
> elasticsearch.exceptions.RequestError: RequestError(400,
> u'parsing_exception', u'request does not support [_source]')
>
> Does anyone have an idea how to fix this and re-index existing databases?
>
> Cheers
> Jens
>
> --
> -- To post, send email to archesproject@googlegroups.com. To unsubscribe,
> send email to archesproject+unsubscr...@googlegroups.com. For more
> information, visit https://groups.google.com/d/forum/archesproject?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Arches Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to archesproject+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Arches] Re: Security testing/audit

[Ben O'Steen]

It is flagging that the string formatting method is being used to construct
a SQL query, rather than a method that might escape or encode parameters
that may have SQL side-effects.

Ben

On Mon, 17 Dec 2018 at 15:35, Adam Cox  wrote:

> Thanks for sharing those results Ben. A number of the issues raised there
> are also a matter of best practices not being followed (like using
> non-typed try/except/pass blocks, or calling os.system) so it's good to see
> them flagged. Also, though it would be really nice to get away from raw SQL
> and rely more on the ORM, looking through those results I think that the
> SQL may be the trickiest to remedy. At any rate, a very good roadmap for an
> initial security update.
>
> Adam
>
> On Mon, Dec 17, 2018 at 5:07 PM Ben O'Steen  wrote:
>
>> I ran the project through the Bandit 'security linter' which flagged up a
>> number of small issues https://pypi.org/project/bandit/ I've attached
>> the result of this in case it is of interest (run on 4.3.1 so obviously run
>> it against whatever you plan to deploy).
>>
>> It is just one tool to use to help gauge the relatively security of an
>> app though. Fuzzing and django-tailored attacks should also be tried.
>>
>> Ben
>>
>>
>> On Mon, 17 Dec 2018 at 14:55, Adam Cox  wrote:
>>
>>> Hi John, I have not had a security audit either, and would be very
>>> interested in the results. I do have a pending deployment for the Bureau of
>>> Land Management which will most likely require an audit, so I'm expecting
>>> to do some prep work on Arches for that at some point.
>>>
>>> On Mon, Dec 17, 2018 at 5:14 AM Vincent Meijer 
>>> wrote:
>>>
>>>> Hi John,
>>>>
>>>> I haven't heard of anyone doing a security audit for Arches, but if so
>>>> I would also be quite interested in hearing about it.
>>>>
>>>> Best,
>>>> Vincent
>>>>
>>>> On Monday, 17 December 2018 10:49:03 UTC+1, John Murphy wrote:
>>>>>
>>>>> Good morning everyone!
>>>>>
>>>>> I was wondering if has anyone has had to commission a security audit
>>>>> of Arches yet and, if so, what have your experiences been? Our corporate 
>>>>> IT
>>>>> are going to require one before they allow me to set this loose on our
>>>>> network and it would be good to know what I need to look out for.
>>>>>
>>>>> Many thanks,
>>>>>
>>>>> John
>>>>>
>>>> --
>>>> -- To post, send email to archesproject@googlegroups.com. To
>>>> unsubscribe, send email to archesproject+unsubscr...@googlegroups.com.
>>>> For more information, visit
>>>> https://groups.google.com/d/forum/archesproject?hl=en
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Arches Project" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to archesproject+unsubscr...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>> --
>>> -- To post, send email to archesproject@googlegroups.com. To
>>> unsubscribe, send email to archesproject+unsubscr...@googlegroups.com.
>>> For more information, visit
>>> https://groups.google.com/d/forum/archesproject?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Arches Project" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to archesproject+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Arches] Re: Security testing/audit

[Ben O'Steen]

I ran the project through the Bandit 'security linter' which flagged up a
number of small issues https://pypi.org/project/bandit/ I've attached the
result of this in case it is of interest (run on 4.3.1 so obviously run it
against whatever you plan to deploy).

It is just one tool to use to help gauge the relatively security of an app
though. Fuzzing and django-tailored attacks should also be tried.

Ben


On Mon, 17 Dec 2018 at 14:55, Adam Cox  wrote:

> Hi John, I have not had a security audit either, and would be very
> interested in the results. I do have a pending deployment for the Bureau of
> Land Management which will most likely require an audit, so I'm expecting
> to do some prep work on Arches for that at some point.
>
> On Mon, Dec 17, 2018 at 5:14 AM Vincent Meijer 
> wrote:
>
>> Hi John,
>>
>> I haven't heard of anyone doing a security audit for Arches, but if so I
>> would also be quite interested in hearing about it.
>>
>> Best,
>> Vincent
>>
>> On Monday, 17 December 2018 10:49:03 UTC+1, John Murphy wrote:
>>>
>>> Good morning everyone!
>>>
>>> I was wondering if has anyone has had to commission a security audit of
>>> Arches yet and, if so, what have your experiences been? Our corporate IT
>>> are going to require one before they allow me to set this loose on our
>>> network and it would be good to know what I need to look out for.
>>>
>>> Many thanks,
>>>
>>> John
>>>
>> --
>> -- To post, send email to archesproject@googlegroups.com. To
>> unsubscribe, send email to archesproject+unsubscr...@googlegroups.com.
>> For more information, visit
>> https://groups.google.com/d/forum/archesproject?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Arches Project" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to archesproject+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> -- To post, send email to archesproject@googlegroups.com. To unsubscribe,
> send email to archesproject+unsubscr...@googlegroups.com. For more
> information, visit https://groups.google.com/d/forum/archesproject?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Arches Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to archesproject+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Run started:2018-10-09 22:00:49.467473

Test results:
>> Issue: [B101:assert_used] Use of assert detected. The enclosed code will be 
>> removed when compiling to optimised byte code.
   Severity: Low   Confidence: High
   Location: arches/arches/app/datatypes/concept_types.py:48
   More Info: 
https://bandit.readthedocs.io/en/latest/plugins/b101_assert_used.html
47  try:
48  assert isinstance(nodevalue, (list, tuple)) #assert 
nodevalue is an array
49  except AssertionError:

--
>> Issue: [B110:try_except_pass] Try, Except, Pass detected.
   Severity: Low   Confidence: High
   Location: arches/arches/app/datatypes/datatypes.py:131
   More Info: 
https://bandit.readthedocs.io/en/latest/plugins/b110_try_except_pass.html
130 tile.data[nodeid] = float(tile.data[nodeid])
131 except:
132 pass

--
>> Issue: [B110:try_except_pass] Try, Except, Pass detected.
   Severity: Low   Confidence: High
   Location: arches/arches/app/datatypes/datatypes.py:213
   More Info: 
https://bandit.readthedocs.io/en/latest/plugins/b110_try_except_pass.html
212 value = str(datetime(value).date())
213 except:
214 pass

--
>> Issue: [B608:hardcoded_sql_expressions] Possible SQL injection vector 
>> through string-based query construction.
   Severity: Medium   Confidence: Low
   Location: arches/arches/app/datatypes/datatypes.py:482
   More Info: 
https://bandit.readthedocs.io/en/latest/plugins/b608_hardcoded_sql_expressions.html
481 WHERE nodeid = '%s'
482 """ % node.pk)
483 

--
>> Issue: [B110:try_except_pass] Try, Except, Pass detected.
   Severity: Low   Confidence: High
   Location: arches/arches/app/datatypes/datatypes.py:982
   

Re: [Arches] Re: Google Analytics

[Ben O'Steen]

Thanks for the heads up.

On Tue, 27 Nov 2018 at 09:41, Adam Cox  wrote:

> Just an FYI to both of you, there was an issue with this a long time ago
> which we fixed and is now closed,
> https://github.com/archesproject/arches/issues/2942, but there's always
> the possibility that something unforeseen has popped up, post bugfix.
>
> Adam
>
> On Tue, Nov 27, 2018 at 10:29 AM Ben O'Steen  wrote:
>
>> Did this fix it? I'm also interested in adding analytics to Arches!
>>
>> Ben
>>
>>
>> On Sun, 25 Nov 2018 at 12:17, Vincent Meijer 
>> wrote:
>>
>>> Found the potential culprit: I need to add www.google-analytics.com to
>>> my Content Security Policy ('script-src') directive.
>>> I'll update here once I am able to test this again.
>>>
>>>
>>> On Sunday, 25 November 2018 11:17:38 UTC+1, Vincent Meijer wrote:
>>>>
>>>> Has anyone used the Google Analytics feature built into Arches?
>>>>
>>>> It doesn't seem to be working for me. I created a GA account and pasted
>>>> my tracking id in the 'Google Analytics Key' field in Arches System
>>>> Settings.
>>>> However, no traffic is being recorded, according to the GA website.
>>>>
>>>> Also, for some reason the analytics.js snippet is only present on about
>>>> every other page refresh (50% of the time), which is odd by itself.
>>>>
>>>>
>>>> [image: ga.png]
>>>>
>>>> Thanks!
>>>> Vincent
>>>>
>>> --
>>> -- To post, send email to archesproject@googlegroups.com. To
>>> unsubscribe, send email to archesproject+unsubscr...@googlegroups.com.
>>> For more information, visit
>>> https://groups.google.com/d/forum/archesproject?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Arches Project" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to archesproject+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> -- To post, send email to archesproject@googlegroups.com. To
>> unsubscribe, send email to archesproject+unsubscr...@googlegroups.com.
>> For more information, visit
>> https://groups.google.com/d/forum/archesproject?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Arches Project" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to archesproject+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Arches] Re: Google Analytics

[Ben O'Steen]

Did this fix it? I'm also interested in adding analytics to Arches!

Ben


On Sun, 25 Nov 2018 at 12:17, Vincent Meijer 
wrote:

> Found the potential culprit: I need to add www.google-analytics.com to
> my Content Security Policy ('script-src') directive.
> I'll update here once I am able to test this again.
>
>
> On Sunday, 25 November 2018 11:17:38 UTC+1, Vincent Meijer wrote:
>>
>> Has anyone used the Google Analytics feature built into Arches?
>>
>> It doesn't seem to be working for me. I created a GA account and pasted
>> my tracking id in the 'Google Analytics Key' field in Arches System
>> Settings.
>> However, no traffic is being recorded, according to the GA website.
>>
>> Also, for some reason the analytics.js snippet is only present on about
>> every other page refresh (50% of the time), which is odd by itself.
>>
>>
>> [image: ga.png]
>>
>> Thanks!
>> Vincent
>>
> --
> -- To post, send email to archesproject@googlegroups.com. To unsubscribe,
> send email to archesproject+unsubscr...@googlegroups.com. For more
> information, visit https://groups.google.com/d/forum/archesproject?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Arches Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to archesproject+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Arches] Re: error initialising

[Ben O'Steen]

(Oh and worth pointing out that RAM allocated is definitely a
consideration! If you are using docker on your local machine, you may be
surprised by how little RAM and CPU gets allocated to the docker host by
default)

On 8 August 2018 at 09:42, Ben O'Steen  wrote:

> I've been hitting something similar. I've noticed that the package.json
> file is not created or copied over when you run "arches-project create
> ...". I'm just learning the project so do not know that this is by design,
> and whether this should be done manually. I fixed my dev site by copying
> over this to my project root (and editing the metadata at the top to
> reflect my project, though if you do that or not it will not affect the
> yarn install AFAIK)
>
> Ben
>
> On 8 August 2018 at 09:07, Adam Cox  wrote:
>
>> Hi David, just wanted to check back on this, a fresh reinstall should be
>> helpful, however is sounds like yarn install did not get run properly
>> during the installation. If you encounter this problem again, enter your
>> project and try running yarn install. This should install all of those
>> .js files that you were missing. This should have happened automatically
>> after you created your project, but perhaps there was a hangup.
>>
>> Adam
>>
>> On Tuesday, July 31, 2018 at 8:51:02 AM UTC-7, David Lopes wrote:
>>>
>>> Hi,
>>>
>>> Is anyone familiar with this error?
>>> The instalation went well but afterwards no access to the site's
>>> management, Could it be not enough RAM?
>>>
>>>
>>>
>>> <https://lh3.googleusercontent.com/-9suz_MIVZd0/W2CE-ZsccMI/Fjk/6pUOteTCJ0c0eJr6_0xWfO2rOGd06ZglACLcBGAs/s1600/2018-07-30-171231_1600x804_scrot.png>
>>>
>>>
>>> Thank you, best regards.
>>>
>> --
>> -- To post, send email to archesproject@googlegroups.com. To
>> unsubscribe, send email to archesproject+unsubscr...@googlegroups.com.
>> For more information, visit https://groups.google.com/d/fo
>> rum/archesproject?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Arches Project" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to archesproject+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Arches] Re: error initialising

[Ben O'Steen]

I've been hitting something similar. I've noticed that the package.json
file is not created or copied over when you run "arches-project create
...". I'm just learning the project so do not know that this is by design,
and whether this should be done manually. I fixed my dev site by copying
over this to my project root (and editing the metadata at the top to
reflect my project, though if you do that or not it will not affect the
yarn install AFAIK)

Ben

On 8 August 2018 at 09:07, Adam Cox  wrote:

> Hi David, just wanted to check back on this, a fresh reinstall should be
> helpful, however is sounds like yarn install did not get run properly
> during the installation. If you encounter this problem again, enter your
> project and try running yarn install. This should install all of those
> .js files that you were missing. This should have happened automatically
> after you created your project, but perhaps there was a hangup.
>
> Adam
>
> On Tuesday, July 31, 2018 at 8:51:02 AM UTC-7, David Lopes wrote:
>>
>> Hi,
>>
>> Is anyone familiar with this error?
>> The instalation went well but afterwards no access to the site's
>> management, Could it be not enough RAM?
>>
>>
>>
>> 
>>
>>
>> Thank you, best regards.
>>
> --
> -- To post, send email to archesproject@googlegroups.com. To unsubscribe,
> send email to archesproject+unsubscr...@googlegroups.com. For more
> information, visit https://groups.google.com/d/forum/archesproject?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Arches Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to archesproject+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
-- To post, send email to archesproject@googlegroups.com. To unsubscribe, send 
email to archesproject+unsubscr...@googlegroups.com. For more information, 
visit https://groups.google.com/d/forum/archesproject?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Arches Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to archesproject+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.