Hi Vincent, Yes. It's on a VM in Azure but not in a container. I did notice performance issues with it, and the node state is yellow.
Next, we do plan to hook it up to an azure single alpine container, then proceed to a cluster like yours but moving away from a VM. Thanks. Regards, Lawrence On Fri, Nov 9, 2018 at 2:40 AM Vincent Meijer <[email protected]> wrote: > Hey Lawrence, > > Sorry for not replying, I was rather swamped in work and things... > Great to hear you got ES (and the complete Arches environment?) up and > running! > > I am currently struggling with Elasticsearch myself. I have a single Azure > VM setup with two docker containers: > version: '2' > > services: > elasticsearch: > container_name: elasticsearch > image: elasticsearch:5.2 > restart: always > volumes: > - elasticsearch-data:/usr/share/elasticsearch/data > environment: > - cluster.name=docker-cluster > - bootstrap.memory_lock=true > - ES_JAVA_OPTS=-Xms2g -Xmx2g > - TZ=EST > ulimits: > memlock: > soft: -1 > hard: -1 > ports: > - "9200:9200" > - "9300:9300" > networks: > - esnet > > elasticsearch2: > container_name: elasticsearch2 > image: elasticsearch:5.2 > restart: always > volumes: > - elasticsearch-data2:/usr/share/elasticsearch/data > environment: > - cluster.name=docker-cluster > - bootstrap.memory_lock=true > - ES_JAVA_OPTS=-Xms2g -Xmx2g > - discovery.zen.ping.unicast.hosts=elasticsearch > - TZ=EST > ulimits: > memlock: > soft: -1 > hard: -1 > networks: > - esnet > > volumes: > elasticsearch-data: > elasticsearch-data2: > > networks: > esnet: > > I just noted, however, that it seems the two nodes don't seem to form a > cluster, potentially causing the performance issues I am investigating... > > How did you set up your ES in Azure? > > Best, > Vincent > > > On Tuesday, 6 November 2018 11:06:16 UTC+1, Lawrence Goh wrote: >> >> Hi Vincent, >> >> Just some updates on my progress. Wonderful news is that I got a VM up >> and running with Elasticsearch stack. Plus, I got the web app to connect >> to it. Found out there's more to it for Elasticsearch, especially on the >> indices. More stuff to read up on. >> >> At this point fingers crossed, my environment is up and running. >> >> Thanks for all the help :) >> >> Regards, >> Lawrence >> >> On Tue, Nov 6, 2018 at 12:34 AM Lawrence Goh <[email protected]> wrote: >> >>> Hi Vincent, >>> >>> I think there's a huge possibility that my azure web app is not talking >>> to elasticsearch. Tracing the code further, it all leads to >>> search/resources and it seems to be calling ES from there. Currently, >>> trying to set up a VM with ES, Logstash and Kibana. Once I figure out on >>> opening the VM's ES up for connection, then will try to point my web app >>> towards it to see it works. It still feels weird that local has no issues >>> but in Azure, it just like being a 3 year old. >>> >>> By the way, upgraded to 4.3.3. >>> >>> Thanks. >>> >>> Regards, >>> Lawrence >>> >>> On Mon, Nov 5, 2018 at 4:15 PM Lawrence Goh <[email protected]> wrote: >>> >>>> Hi Vincent, >>>> >>>> Dug further, that javascript error was not related. I am getting errors >>>> when it is retrieving search results. Search would build up a JSON query >>>> (/search/resources?no_filters=true&page=1), it gave me a "There was an >>>> error retrieving the search results". Found it in the code >>>> (search.py:264). Still thinking on how can I debug the code when it >>>> actually works on my local. Both are pointing to the azure postgresql DB. >>>> >>>> Unsure if there's any missing configuration where it is not allowed >>>> when azure web app - azure db, but all is good with local web app - azure >>>> db. >>>> >>>> Very strange. >>>> >>>> Thanks. >>>> >>>> Regards, >>>> Lawrence >>>> >>>> >>>> >>>> >>>> >>>> On Mon, Nov 5, 2018 at 11:45 AM Lawrence Goh <[email protected]> >>>> wrote: >>>> >>>>> Hi Vincent, >>>>> >>>>> No logs sadly. But I found this weird javascript error when I click >>>>> on Search. >>>>> >>>>> jquery.min.js:2 Uncaught Error: Syntax error, unrecognized expression: >>>>> /search >>>>> at Function.fa.error (jquery.min.js:2) >>>>> at fa.tokenize (jquery.min.js:2) >>>>> at fa.select (jquery.min.js:2) >>>>> at Function.fa [as find] (jquery.min.js:2) >>>>> at n.fn.init.find (jquery.min.js:2) >>>>> at n.fn.init.a.fn.find (jquery-migrate.min.js:2) >>>>> at a.fn.init.n.fn.init (jquery.min.js:2) >>>>> at new a.fn.init (jquery-migrate.min.js:2) >>>>> at n (jquery.min.js:2) >>>>> at HTMLAnchorElement.<anonymous> (one.app.js:111) >>>>> >>>>> About your elasticsearch instance, you created a VM just for it? >>>>> >>>>> Still working things out. Will update you if i had any >>>>> breakthroughs. >>>>> >>>>> Thanks. >>>>> >>>>> Regards, >>>>> Lawrence >>>>> >>>>> On Sat, Nov 3, 2018 at 12:24 AM Vincent Meijer <[email protected]> >>>>> wrote: >>>>> >>>>>> The errors you are getting don't seem familiar, probably because I >>>>>> set up Elasticsearch on a separate virtual machine (I think using the >>>>>> Bitnami Elasticsearch Cluster template, but can't remember), rather than >>>>>> include it in the Web App. >>>>>> The difference in version doesn't sound like the culprit here >>>>>> though... >>>>>> >>>>>> As far as 500 errors go, they can be anything, but mostly they are >>>>>> uncaught exceptions in my experience. Could be related to your >>>>>> Elasticsearch problems. Do you have the logs? >>>>>> >>>>>> >>>>>> On Wednesday, 31 October 2018 10:25:38 UTC+1, Lawrence Goh wrote: >>>>>>> >>>>>>> Hi Vincent, >>>>>>> >>>>>>> Just wondering. When running in Azure's Web App for Containers, did >>>>>>> your search give you an error? I noticed the elasticsearch had an error >>>>>>> where all shards failed. By the way, I noticed I had errors starting up >>>>>>> with v5.2, so I used v5.6. >>>>>>> >>>>>>> Another, for recent edits, when I clicked on view report, it gives a >>>>>>> Server Error 500. >>>>>>> >>>>>>> The oddity is that it works on my local :) >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> Regards, >>>>>>> Lawrence >>>>>>> >>>>>>> On Wed, Oct 31, 2018 at 9:22 AM Lawrence Goh <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Vincent, >>>>>>>> >>>>>>>> Thanks for the reply. Found the issue to be on entrypoint.sh. The >>>>>>>> validations have yet to be updated. Currently, using 4.3.x branch but >>>>>>>> will >>>>>>>> probably move to 4.3.2 since it was just released. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Lawrence >>>>>>>> >>>>>>>> On Tue, Oct 30, 2018 at 10:07 PM Vincent Meijer < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hey Lawrence, sorry for the late response. >>>>>>>>> >>>>>>>>> Regarding settings.py: your project's settings.py take precedence >>>>>>>>> over the default arches settings.py. >>>>>>>>> *However*, in the docker version of Arches, a settings_local.py >>>>>>>>> is also copied to your project folder. >>>>>>>>> This settings_local.py should contain the postgres db settings >>>>>>>>> that you set through the environment variables in your >>>>>>>>> docker-compose.yml. >>>>>>>>> It should already be imported by default in your settings.py. >>>>>>>>> Notice these lines at the bottom of that file: >>>>>>>>> try: >>>>>>>>> from settings_local import * >>>>>>>>> except ImportError: >>>>>>>>> pass >>>>>>>>> >>>>>>>>> When your project gets initialized (at first startup), that >>>>>>>>> settings_local.py is copied to /web_root/<your project>/<your >>>>>>>>> project>/settings_local.py. >>>>>>>>> Here is the source for reference: >>>>>>>>> https://github.com/archesproject/arches/blob/master/docker/settings_local.py >>>>>>>>> >>>>>>>>> >>>>>>>>> With that setup, all you should have to do is set your db >>>>>>>>> credentials in your docker-compose.yml: >>>>>>>>> - PGUSERNAME=username@your-db-server >>>>>>>>> >>>>>>>>> If you still receive above errors, something else is up and we >>>>>>>>> need to investigate :) >>>>>>>>> >>>>>>>>> By the way, which version of Arches are you using right now? >>>>>>>>> >>>>>>>>> Also, if you encounter any errors in the Docker Readme, especially >>>>>>>>> in the 'Setting up your own Arches project' section, please let us >>>>>>>>> know! :) >>>>>>>>> >>>>>>>>> https://github.com/archesproject/arches/tree/master/docker#setting-up-your-own-arches-project >>>>>>>>> >>>>>>>>> >>>>>>>>> On Friday, 26 October 2018 05:43:11 UTC+2, Lawrence Goh wrote: >>>>>>>>>> >>>>>>>>>> Hi Vincent, >>>>>>>>>> >>>>>>>>>> I noticed another odd thing. When it does the import_graphs >>>>>>>>>> after creating the database, it has this error. >>>>>>>>>> >>>>>>>>>> arches | FATAL: Invalid Username specified. Please >>>>>>>>>> check the Username and retry connection. The Username should be in >>>>>>>>>> <username@hostname> format. >>>>>>>>>> arches | FATAL: Invalid Username specified. Please >>>>>>>>>> check the Username and retry connection. The Username should be in >>>>>>>>>> <username@hostname> format. >>>>>>>>>> arches | Running: python manage.py packages -o >>>>>>>>>> import_reference_data -s >>>>>>>>>> "arches/db/schemes/arches_concept_scheme.rdf" >>>>>>>>>> >>>>>>>>>> Any idea where in the code is going nuts? :) >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Lawrence >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Oct 26, 2018 at 4:03 PM Lawrence Goh <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Vincent, >>>>>>>>>>> >>>>>>>>>>> Been digging further into the postgres libs. It seems the dsn >>>>>>>>>>> and kwargs in them are getting values from the <project_name>'s >>>>>>>>>>> settings.py >>>>>>>>>>> which contains the default db credentials (postgres; postgis). >>>>>>>>>>> Normally, I >>>>>>>>>>> assumed it will get it from the arches' settings.py file. >>>>>>>>>>> >>>>>>>>>>> In case I am confusing :) When you browse the container, you >>>>>>>>>>> would be landed into \web_root\arches. Back down, it would be ENV, >>>>>>>>>>> arches, >>>>>>>>>>> <project_name> folders. >>>>>>>>>>> >>>>>>>>>>> Probably I might need a new pull from arches again and work from >>>>>>>>>>> there. I am unsure if my own is corrupted. >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Lawrence >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Oct 25, 2018 at 5:30 AM Vincent Meijer < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Ah I can see the image now (in email). Looks like a regular >>>>>>>>>>>> firewall misconfiguration. >>>>>>>>>>>> Can you show us your current settings? >>>>>>>>>>>> >>>>>>>>>>>> There is also the option 'Enforce SSL Connection' under 'SSL >>>>>>>>>>>> Settings' in your Azure DB. Is this set to Disabled? >>>>>>>>>>>> >>>>>>>>>>>> And can you connect to it from your local machine if you >>>>>>>>>>>> whitelist your ip in that same screen? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tuesday, 23 October 2018 23:45:06 UTC+2, Lawrence Goh wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Vincent, >>>>>>>>>>>>> >>>>>>>>>>>>> Apologies for another set up question. Been going everywhere >>>>>>>>>>>>> and sadly to nowhere. We have set up a Azure Database for >>>>>>>>>>>>> Postgresql. We >>>>>>>>>>>>> could get the docker compose to create the DB. But the arches >>>>>>>>>>>>> container >>>>>>>>>>>>> failed due to some weird network restriction. >>>>>>>>>>>>> >>>>>>>>>>>>> [image: image.png] >>>>>>>>>>>>> >>>>>>>>>>>>> I have added IPs to the firewall. Even having it being from >>>>>>>>>>>>> 0.0.0.0 to 255.255.255.255. The "Allow access to Azure >>>>>>>>>>>>> services" is on. >>>>>>>>>>>>> >>>>>>>>>>>>> Any ideas? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Lawrence >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Oct 19, 2018 at 10:32 PM Vincent Meijer < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Yeah it's a weird choice from Microsoft to do things >>>>>>>>>>>>>> differently and arches wasn't prepared for that @... >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, 19 Oct 2018, 11:23 larrygoh84, <[email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> using 4.3.1. does this mean its a known issue? it doesn't >>>>>>>>>>>>>>> handle the @ well? :) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -------- Original message -------- >>>>>>>>>>>>>>> From: Vincent Meijer <[email protected]> >>>>>>>>>>>>>>> Date: 10/19/18 10:02 PM (GMT+12:00) >>>>>>>>>>>>>>> To: Arches Project <[email protected]> >>>>>>>>>>>>>>> Subject: Re: [Arches] Re: Hosting Arches on Azure as a Web >>>>>>>>>>>>>>> App for Containers >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Which version of Arches are you using? >>>>>>>>>>>>>>> I created a PR (for #4099) for this earlier, but it was >>>>>>>>>>>>>>> after 4.3.1 was released. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Until the next release you could use the master branch >>>>>>>>>>>>>>> (there are docker images built for that in Docker Hub for every >>>>>>>>>>>>>>> new push: >>>>>>>>>>>>>>> https://hub.docker.com/r/archesproject/arches/tags/ ) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> However, if you want something more stable, you are also >>>>>>>>>>>>>>> free to use my own arches base images: >>>>>>>>>>>>>>> https://hub.docker.com/r/globaldigitalheritage/arches/tags/ >>>>>>>>>>>>>>> The current tags in there are all based on Arches 4.3.1 and >>>>>>>>>>>>>>> I push a new version every time I created some additional >>>>>>>>>>>>>>> functionality / >>>>>>>>>>>>>>> fixes for a PR, such as the one with '@' in Azure database >>>>>>>>>>>>>>> names. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Currently the latest I got is >>>>>>>>>>>>>>> globaldigitalheritage/arches:4.3.1.11 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I use these images as a base in my own Dockerfile >>>>>>>>>>>>>>> (as described in step 7 of 'Setting up your own Arches >>>>>>>>>>>>>>> project': >>>>>>>>>>>>>>> https://github.com/archesproject/arches/blob/master/docker/Readme.md#setting-up-your-own-arches-project >>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I recommend to follow these steps to create your own Arches >>>>>>>>>>>>>>> project, rather than editing the core Arches app. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hope this helps! >>>>>>>>>>>>>>> Vincent >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Friday, 19 October 2018 05:14:57 UTC+2, Lawrence Goh >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Vincent, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Had you faced this issue before? We are now trying to >>>>>>>>>>>>>>>> connect to Azure DB for Postgresql. The container log just >>>>>>>>>>>>>>>> freezes at >>>>>>>>>>>>>>>> Testing if database server is up.. Is it senstive to the @? >>>>>>>>>>>>>>>> thanks. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Friday, October 19, 2018 at 3:03:47 PM UTC+13, Lawrence >>>>>>>>>>>>>>>> Goh wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Vincent, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Yups. Was poking around the codes yesterday and had a go >>>>>>>>>>>>>>>>> with removing the staticfiles_storage and all went well :) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>> Lawrence >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Thu, Oct 18, 2018 at 9:00 PM Vincent Meijer < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> If I'm understanding you correctly, you want to serve >>>>>>>>>>>>>>>>>> static files the 'standard' way (directly from the Arches >>>>>>>>>>>>>>>>>> app using >>>>>>>>>>>>>>>>>> Whitenoise) and serve uploaded files from Azure Blob Storage? >>>>>>>>>>>>>>>>>> I do the same at the moment. For that, use: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> DEFAULT_FILE_STORAGE='storages.backends.azure_storage.AzureStorage' >>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>> STATICFILES_STORAGE = >>>>>>>>>>>>>>>>>> 'whitenoise.storage.CompressedStaticFilesStorage' >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> (together with the other appropriate settings for Azure >>>>>>>>>>>>>>>>>> and Whitenoise, that you probably already have). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Does that make sense? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Wednesday, 17 October 2018 23:04:35 UTC+2, Lawrence >>>>>>>>>>>>>>>>>> Goh wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Vincent, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks for that link. Will need to figure to adapt it >>>>>>>>>>>>>>>>>>> for Azure use. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Besides that, I noticed it loads all the static files >>>>>>>>>>>>>>>>>>> into the blob. Was thinking to have the static files be in >>>>>>>>>>>>>>>>>>> the container. >>>>>>>>>>>>>>>>>>> Only push the uploaded 'media files' into the blob. I >>>>>>>>>>>>>>>>>>> guess it doesn't do >>>>>>>>>>>>>>>>>>> that right? :) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>> Lawrence >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Thu, Oct 18, 2018 at 2:12 AM Vincent Meijer < >>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The collectfast package looks promising: >>>>>>>>>>>>>>>>>>>> https://stackoverflow.com/questions/16805232/faster-alternative-to-manage-py-collectstatic-w-s3boto-storage-backend-to-syn >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>> -- To post, send email to [email protected]. >>>>>>>>>>>>>>>>>>>> To unsubscribe, send email to >>>>>>>>>>>>>>>>>>>> [email protected]. For more >>>>>>>>>>>>>>>>>>>> information, visit >>>>>>>>>>>>>>>>>>>> https://groups.google.com/d/forum/archesproject?hl=en >>>>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>>>> You received this message because you are subscribed to >>>>>>>>>>>>>>>>>>>> the Google Groups "Arches Project" group. >>>>>>>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving >>>>>>>>>>>>>>>>>>>> emails from it, send an email to >>>>>>>>>>>>>>>>>>>> [email protected]. >>>>>>>>>>>>>>>>>>>> For more options, visit >>>>>>>>>>>>>>>>>>>> https://groups.google.com/d/optout. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> -- To post, send email to [email protected]. To >>>>>>>>>>>>>>>>>> unsubscribe, send email to >>>>>>>>>>>>>>>>>> [email protected]. For more information, >>>>>>>>>>>>>>>>>> visit >>>>>>>>>>>>>>>>>> https://groups.google.com/d/forum/archesproject?hl=en >>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>> You received this message because you are subscribed to >>>>>>>>>>>>>>>>>> the Google Groups "Arches Project" group. >>>>>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails >>>>>>>>>>>>>>>>>> from it, send an email to >>>>>>>>>>>>>>>>>> [email protected]. >>>>>>>>>>>>>>>>>> For more options, visit >>>>>>>>>>>>>>>>>> https://groups.google.com/d/optout. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> -- To post, send email to [email protected]. To >>>>>>>>>>>>>>> unsubscribe, send email to [email protected]. >>>>>>>>>>>>>>> For more information, visit >>>>>>>>>>>>>>> https://groups.google.com/d/forum/archesproject?hl=en >>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>>> Google Groups "Arches Project" group. >>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails >>>>>>>>>>>>>>> from it, send an email to [email protected]. >>>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>> -- To post, send email to [email protected]. To >>>>>>>>>>>> unsubscribe, send email to [email protected]. >>>>>>>>>>>> For more information, visit >>>>>>>>>>>> https://groups.google.com/d/forum/archesproject?hl=en >>>>>>>>>>>> --- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "Arches Project" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> -- To post, send email to [email protected]. To >>>>>>>>> unsubscribe, send email to [email protected]. For >>>>>>>>> more information, visit >>>>>>>>> https://groups.google.com/d/forum/archesproject?hl=en >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Arches Project" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> -- >>>>>> -- To post, send email to [email protected]. To >>>>>> unsubscribe, send email to [email protected]. For >>>>>> more information, visit >>>>>> https://groups.google.com/d/forum/archesproject?hl=en >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Arches Project" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- > -- To post, send email to [email protected]. To unsubscribe, > send email to [email protected]. For more > information, visit https://groups.google.com/d/forum/archesproject?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Arches Project" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- -- To post, send email to [email protected]. To unsubscribe, send email to [email protected]. For more information, visit https://groups.google.com/d/forum/archesproject?hl=en --- You received this message because you are subscribed to the Google Groups "Arches Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
