[Wikidata-bugs] [Maniphest] T215623: Problem with docker-compose2 (behind proxy)

[Addshore]

Addshore closed this task as "Resolved".
Addshore claimed this task.
Addshore added a comment.
Restricted Application added a project: User-Addshore.


  Great!

TASK DETAIL
  https://phabricator.wikimedia.org/T215623

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Phaebz, Addshore, Aklapper, DD063520, Invadibot, MPhamWMF, maantietaja, 
CBogen, Samantha_Alipio_WMDE, Akuckartz, Jelabra, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Asahiko, despens, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T215623: Problem with docker-compose2 (behind proxy)

[Phaebz]

Phaebz added a comment.


  Following your advice, we use proper FQDN now instead of IP.
  
  Further, following your post here: 
https://addshore.com/2019/11/changing-the-concept-uri-of-an-existing-wikibase-with-data/,
 esp. dumpRdf.php, munge.sh and loadData.sh with the correct conceptUri, we 
managed to re-create the wdqs-updater and the 
`BaseVocabulary$VocabularyVersioningException: null` error is gone. Maybe this 
also helps @DD063520, I consider this as solved.
  
  Thanks for your support!

TASK DETAIL
  https://phabricator.wikimedia.org/T215623

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Phaebz
Cc: Phaebz, Addshore, Aklapper, DD063520, Invadibot, MPhamWMF, maantietaja, 
CBogen, Samantha_Alipio_WMDE, Akuckartz, Jelabra, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Asahiko, despens, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T215623: Problem with docker-compose2 (behind proxy)

[Addshore]

Addshore added a comment.


  > In particular, deleting data/data.jnl as mentioned in T207133#4681278 
 would wipe the blazegraph 
triple store, wouldn't it?
  
  Yes
  
  > This instance was migrated (docker volumes were copied over) from another 
instance that had a FQDN in our network DNS. In the deployment at hand, we do 
not have a FQDN, but only the IP you mentioned looked odd in the concept URI 
query.
  
  So the concept URI can be configured in both the query service updater, as 
the expected concept base, and also in wikibase and the concept base to output.
  
  In your docker-compose for the query service this is `EES_WIKIBASE_FQDN` 
which is currently the IP.
  
  If you look in the docker-compose you'll see the bit `CONFIG - Add your real 
wikibase hostname here, for example wikibase-registry.wmflabs.org`
  So this should indeed be not the IP and it should be what you want to appear 
as the concept base URis.

TASK DETAIL
  https://phabricator.wikimedia.org/T215623

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Phaebz, Addshore, Aklapper, DD063520, Invadibot, MPhamWMF, maantietaja, 
CBogen, Samantha_Alipio_WMDE, Akuckartz, Jelabra, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Asahiko, despens, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T215623: Problem with docker-compose2 (behind proxy)

[Phaebz]

Phaebz added a comment.


  > This does indeed look like your problem.
  
  
  
  > `com.bigdata.rdf.vocab.BaseVocabulary$VocabularyVersioningException: null`
  
  I think T286334  is not my problem 
as I do not see the mentioned:
  
wdqs_1 | /wdqs /wdqs
wdqs_1 | Running Blazegraph from /wdqs on :/bigdata
wdqs_1 | Error: Could not find or load main class #
  
  in there. I want to try the solution in T207133#4681278 
 which seemed to work back 
then. Now I don't fully understand how all pieces fit together, especially the 
part about changes not yet committed to the DB. In particular, deleting 
`data/data.jnl` as mentioned in T207133#4681278 
 would wipe the blazegraph 
triple store, wouldn't it?

TASK DETAIL
  https://phabricator.wikimedia.org/T215623

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Phaebz
Cc: Phaebz, Addshore, Aklapper, DD063520, Invadibot, MPhamWMF, maantietaja, 
CBogen, Samantha_Alipio_WMDE, Akuckartz, Jelabra, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Asahiko, despens, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T215623: Problem with docker-compose2 (behind proxy)

[Phaebz]

Phaebz added a comment.


  Thanks for checking back so quickly.
  
  > This select using an IP address looks wrong, this should be using the 
concept base URI for wikibase, something feels wrong with this configuration.
  
  What is the concept base URI? What config could be causing this? Should this 
query be to the docker host, or directly the docker internal network?
  
  > You are using a very old version of the wdqs docker image and service.
  
  Yes, this is a deployment where we tried to avoid upgrades when we can as we 
had some problems with those in the past.
  
  > However in T286334  you are 
seemingly using wikibase/wdqs:0.3.40-wmde.1?
  
  I just thought the ticket might be related, I am not using 0.3.40. How can I 
debug this further to see where the root cause of 
`com.bigdata.rdf.vocab.BaseVocabulary$VocabularyVersioningException: null` 
might be?

TASK DETAIL
  https://phabricator.wikimedia.org/T215623

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Phaebz
Cc: Phaebz, Addshore, Aklapper, DD063520, Invadibot, MPhamWMF, maantietaja, 
CBogen, Samantha_Alipio_WMDE, Akuckartz, Jelabra, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Asahiko, despens, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T215623: Problem with docker-compose2 (behind proxy)

[Addshore]

Addshore added a comment.


  This select using an IP address looks wrong, this should be using the concept 
base URI for wikibase, something feels wrong with this configuration.
  
SELECT * WHERE {  schema:dateModified ?date }
  
  However that will not lead to the 500 error.
  
  ---
  
  This does indeed look like your problem.
  
com.bigdata.rdf.vocab.BaseVocabulary$VocabularyVersioningException: null
  
  And this will likely be the cause of T286334: `wdqs-frontend` and `wdqs` 
containers are constantly restarting using the wikibase-release-pipeline 
docker-compose example 
  
  However so as not to split discussion lets pick a single task to continue 
with!
  
  
  
wdqs:
  image: wikibase/wdqs:0.3.10
  
  You are using a very old version of the wdqs docker image and service.
  However in T286334  you are 
seemingly using `wikibase/wdqs:0.3.40-wmde.1`?

TASK DETAIL
  https://phabricator.wikimedia.org/T215623

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Phaebz, Addshore, Aklapper, DD063520, Invadibot, MPhamWMF, maantietaja, 
CBogen, Samantha_Alipio_WMDE, Akuckartz, Jelabra, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Asahiko, despens, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T215623: Problem with docker-compose2 (behind proxy)

[Phaebz]

Phaebz reopened this task as "Open".
Restricted Application added a project: wdwb-tech.

TASK DETAIL
  https://phabricator.wikimedia.org/T215623

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Phaebz
Cc: Phaebz, Addshore, Aklapper, DD063520, Invadibot, MPhamWMF, maantietaja, 
CBogen, Samantha_Alipio_WMDE, Akuckartz, Jelabra, Nandana, Namenlos314, Lahi, 
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Asahiko, despens, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T215623: Problem with docker-compose2 (behind proxy)

[Phaebz]

Phaebz added a comment.


  I see this error too with the following error message in the updater
  
18:34:44.410 [main] INFO  org.wikidata.query.rdf.tool.Update - Starting 
Updater 0.3.10-SNAPSHOT (8bba8bfb0bbb12361c0e214fe482fbe15fcaa129)
18:34:45.335 [main] INFO  o.w.q.r.t.change.ChangeSourceContext - Checking 
where we left off
18:34:45.335 [main] INFO  o.w.query.rdf.tool.rdf.RdfRepository - Checking 
for left off time from the updater
18:34:45.517 [main] ERROR org.wikidata.query.rdf.tool.Update - Error during 
initialization.
org.wikidata.query.rdf.tool.exception.ContainedException: Non-200 response 
from triple store:  HttpContentResponse[HTTP/1.1 500 Server Error - 8081 bytes] 
body=
SPARQL-QUERY: queryStr=PREFIX schema: 
SELECT * WHERE {  schema:dateModified ?date }
java.util.concurrent.ExecutionException: java.lang.RuntimeException: off=0, 
len=558::namespace=wdq, timestamp=readOnly(1626719685498), 
readTime=readOnly(1599573101388)
...
  
  this message in the wdqs
  
18:34:45.501 [qtp584634336-16] ERROR c.b.r.sail.webapp.BigdataRDFServlet 
IP:ees-wikibase-docker_wdqs-updater_1.ees-wikibase-docker_default 
UA:Jetty/9.4.z-SNAPSHOT - cause=java.util.concurrent.ExecutionExcept
ion: java.lang.RuntimeException: off=0, len=558::namespace=wdq, 
timestamp=readOnly(1626719685498), readTime=readOnly(1599573101388), 
query=SPARQL-QUERY: queryStr=PREFIX schema: 
SELECT * WHERE {  schema:dateModified ?date }
com.bigdata.rdf.vocab.BaseVocabulary$VocabularyVersioningException: null
at 
com.bigdata.rdf.vocab.BaseVocabulary.readVersion2(BaseVocabulary.java:680)
Wrapped by: java.lang.RuntimeException: off=0, len=558
at 
com.bigdata.io.SerializerUtil.deserialize(SerializerUtil.java:239)
Wrapped by: java.lang.RuntimeException: off=0, len=558::namespace=wdq, 
timestamp=readOnly(1626719685498), readTime=readOnly(1599573101388)
at 
com.bigdata.relation.locator.DefaultResourceLocator.locateResourceOn(DefaultResourceLocator.java:817)
Wrapped by: java.util.concurrent.ExecutionException: 
java.lang.RuntimeException: off=0, len=558::namespace=wdq, 
timestamp=readOnly(1626719685498), readTime=readOnly(1599573101388)
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
  
  And the `docker-compose.yml` (matching .env at the bottom)
  
# Wikibase with Query Service
#
# This docker-compose example can be used to pull the images from docker 
hub.
#
# Examples:
#
# Access Wikibase via "http://localhost:8181";
#   (or "http://$(docker-machine ip):8181" if using docker-machine)
#
# Access Query Service via "http://localhost:8282";
#   (or "http://$(docker-machine ip):8282" if using docker-machine)
version: '3'

services:
  wikibase:
image: wikibase/wikibase:1.34-bundle
links:
  - mysql
ports:
# CONFIG - Change the 8181 here to expose Wikibase & MediaWiki on a 
different port
 - "${EES_WIKIBASE_HOST_PORT}:${EES_WIKIBASE_CONTAINER_PORT}"
volumes:
  - mediawiki-images-data:/var/www/html/images
  - quickstatements-data:/quickstatements/data
  - 
./extensions/WikibaseQualityConstraints:/var/www/html/extensions/WikibaseQualityConstraints:rw
  # Note: enable the following line only after an initial 
docker-compose up -d
  - ./LocalSettings.php:/var/www/html/LocalSettings.php:ro
  - ./logo.png:/var/www/html/logo.png:ro
  # - ./LocalSettings.php.template:/LocalSettings.php.template
depends_on:
- mysql
- elasticsearch
restart: unless-stopped
networks:
  default:
aliases:
 - ${EES_WIKIBASE_FQDN}
 # CONFIG - Add your real wikibase hostname here, for example 
wikibase-registry.wmflabs.org
environment:
  - DB_SERVER=mysql.svc:3306
  - MW_ELASTIC_HOST=elasticsearch.svc
  - MW_ELASTIC_PORT=9200
  # CONFIG - Change the default values below
  - MW_ADMIN_NAME=admin
  - MW_ADMIN_PASS=${EES_WIKIBASE_MW_ADMIN_PASS}
  - MW_ADMIN_EMAIL=redacted
  - MW_WG_SECRET_KEY=${EES_WIKIBASE_MW_WG_SECRET_KEY}
  # CONFIG - Change the default values below (should match mysql values 
in this file)
  - DB_USER=wikiuser
  - DB_PASS=sqlpass
  - DB_NAME=my_wiki
  - QS_PUBLIC_SCHEME_HOST_AND_PORT=http://localhost:9191
  mysql:
image: mariadb:10.3
restart: unless-stopped
volumes:
  - mediawiki-mysql-data:/var/lib/mysql
environment:
  MYSQL_RANDOM_ROOT_PASSWORD: 'yes'
  # CONFIG - Change the default values below (should match values 
passed to wikibase)
  MYSQL_DATABASE: 'my_wiki'
  MYSQL_USER: 'wikiuser'
  MYSQL_PASSWO