[Wikidata-bugs] [Maniphest] T266574: Wikibase docker update strategy

2020-10-27 Thread helmo
helmo created this task.
helmo added a project: Wikibase-Containers.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.

TASK DESCRIPTION
  I'm wondering about an update policy and what to expect.
  
  Pulling the wikibase docker images currently gives me MediaWiki 1.34.1 while 
1.34.4 (and 1.35.0) is the latest according to 
https://www.mediawiki.org/wiki/Download
  
  On Docker Hub <https://hub.docker.com/r/wikibase/wikibase> I noticed `Last 
pushed 15 days ago by wikibasetravis` so I was hoping for 1.34.4.
  
  What do we need to do to stay up to date with security releases? It would be 
nice if the README 
<https://github.com/wmde/wikibase-docker/blob/master/README.md>  or 
README-compose.md 
<https://github.com/wmde/wikibase-docker/blob/master/README-compose.md> would 
give some guidance about this.
  
  There is 
https://addshore.com/2019/01/wikibase-docker-mediawiki-wikibase-update/ but 
that's more about updating between major versions.
  
  I started by commenting on T197220 
<https://phabricator.wikimedia.org/T197220> but that is about mediawiki in 
general. But it did give me the extra question why doesn't wikibase build ontop 
of the regular mediawiki container images?

TASK DETAIL
  https://phabricator.wikimedia.org/T266574

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: helmo
Cc: Hannolans, Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, Jelabra, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Asahiko, despens, Wikidata-bugs, aude, Addshore, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T264431: autocomplete in querybuilder is missing results

2020-10-27 Thread helmo
helmo closed this task as "Resolved".
helmo claimed this task.
helmo added a comment.


  The last question is probably explained by the type of search that is done. 
It's probably a prefix search while we expected a full text search. With large 
datasets it probably makes sense to do it this way.

TASK DETAIL
  https://phabricator.wikimedia.org/T264431

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: helmo
Cc: despens, Addshore, Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, 
Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Asahiko, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T264431: autocomplete in querybuilder is missing results

2020-10-14 Thread helmo
helmo added a subscriber: despens.
helmo added a comment.


  Together with @despens we found that in the search box most letters give a 
result, but not the 'g' which you would expect 'van Gogh' to be found. It's 
still a mystery...

TASK DETAIL
  https://phabricator.wikimedia.org/T264431

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: helmo
Cc: despens, Addshore, Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, 
darthmon_wmde, Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Asahiko, Wikidata-bugs, aude, 
Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T264431: autocomplete in querybuilder is missing results

2020-10-13 Thread helmo
helmo added a comment.


  We're a but further... Via Extension:CirrusSearch 
<https://www.mediawiki.org/wiki/Extension:CirrusSearch> I ended up on the 
CirrusSearch README 
<https://gerrit.wikimedia.org/g/mediawiki/extensions/CirrusSearch/+/HEAD/README>.
 Which mentioned the steps:
  
Now run this script to generate your elasticsearch index:
 php 
$MW_INSTALL_PATH/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php
Now remove $wgDisableSearchUpdate = true from LocalSettings.php.  Updates 
should start heading to Elasticsearch.
Next bootstrap the search index by running:
 php 
$MW_INSTALL_PATH/extensions/CirrusSearch/maintenance/ForceSearchIndex.php 
--skipLinks --indexOnSkip
 php 
$MW_INSTALL_PATH/extensions/CirrusSearch/maintenance/ForceSearchIndex.php 
--skipParse
Note that this can take some time.  For large wikis read "Bootstrapping 
large wikis" below.
  
  After doing those steps I started getting results on the search page 
<https://api.kunstmuseum.nl/w/index.php?search=gogh+=Special:Search=1>
 but still not in the search box URL 
<https://api.kunstmuseum.nl/w/api.php?action=wbsearchentities=gogh=json=en=en=item>
  
  Is there a maintenance script that should have triggered this automatically 
from docker-compsose?

TASK DETAIL
  https://phabricator.wikimedia.org/T264431

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: helmo
Cc: Addshore, Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, darthmon_wmde, 
Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Asahiko, despens, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T264431: autocomplete in querybuilder is missing results

2020-10-13 Thread helmo
helmo added a comment.


  No, here's the log after restarting elasticsearch and attempting a few 
searches:
  
wikibase@wikibase:~/wikibase-docker$ docker-compose  up elasticsearch
Starting wikibase-docker_elasticsearch_1 ... done
Attaching to wikibase-docker_elasticsearch_1
elasticsearch_1| OpenJDK 64-Bit Server VM warning: Option 
UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in 
a future release.
elasticsearch_1| OpenJDK 64-Bit Server VM warning: UseAVX=2 is not 
supported on this CPU, setting it to UseAVX=0
elasticsearch_1| [2020-10-13T12:14:50,181][INFO ][o.e.e.NodeEnvironment 
   ] [7DEERnP] using [1] data paths, mounts [[/ (overlay)]], net usable_space 
[4.8gb], net total_space [14.9gb], types [overlay]
elasticsearch_1| [2020-10-13T12:14:50,186][INFO ][o.e.e.NodeEnvironment 
   ] [7DEERnP] heap size [495.3mb], compressed ordinary object pointers [true]
elasticsearch_1| [2020-10-13T12:14:50,189][INFO ][o.e.n.Node
   ] [7DEERnP] node name derived from node ID [7DEERnPBTzujGriDPZxI8w]; set 
[node.name] to override
elasticsearch_1| [2020-10-13T12:14:50,191][INFO ][o.e.n.Node
   ] [7DEERnP] version[6.5.4], pid[1], 
build[default/tar/d2ef93d/2018-12-17T21:17:40.758843Z], 
OS[Linux/4.19.0-10-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server 
VM/11.0.1/11.0.1+13]
elasticsearch_1| [2020-10-13T12:14:50,192][INFO ][o.e.n.Node
   ] [7DEERnP] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, 
-XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, 
-XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, 
-Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, 
-Dio.netty.noKeySetOptimization=true, 
-Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, 
-Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.T6CKFqYS, 
-XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, 
-XX:ErrorFile=logs/hs_err_pid%p.log, 
-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m,
 -Djava.locale.providers=COMPAT, -XX:UseAVX=2, 
-Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, 
-Des.path.home=/usr/share/elasticsearch, 
-Des.path.conf=/usr/share/elasticsearch/config, 
-Des.distribution.flavor=default, -Des.distribution.type=tar]
elasticsearch_1| [2020-10-13T12:14:52,638][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [aggs-matrix-stats]
elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [analysis-common]
elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [ingest-common]
elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [lang-expression]
elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [lang-mustache]
elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [lang-painless]
elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [mapper-extras]
elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [parent-join]
elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [percolator]
elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [rank-eval]
elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [reindex]
elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [repository-url]
elasticsearch_1| [2020-10-13T12:14:52,640][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [transport-netty4]
elasticsearch_1| [2020-10-13T12:14:52,640][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [tribe]
elasticsearch_1| [2020-10-13T12:14:52,640][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [x-pack-ccr]
elasticsearch_1| [2020-10-13T12:14:52,640][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [x-pack-core]
elasticsearch_1| [2020-10-13T12:14:52,640][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [x-pack-deprecation]
elasticsearch_1| [2020-10-13T12:14:52,640][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [x-pack-graph]
elasticsearch_1| [2020-10-13T12:14:52,640][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [x-pack-logstash]
elasticsearch_1| [2020-10-13T12:14:52,640][INFO ][o.e.p.PluginsService  
   ] [7DEERnP] loaded module [x-pack-ml]
elasticsearch_1| [2020-10-13T12:14:52,640][INFO

[Wikidata-bugs] [Maniphest] T264431: autocomplete in querybuilder is missing results

2020-10-08 Thread helmo
helmo added a comment.


  Removing the `format=json` part of that url shows a warning of type 
'error'... that looks like https://phabricator.wikimedia.org/T260276
  
  The code change there only looks cosmetic in no longer showing this error. 
  I manually patched in the changes 
<https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseCirrusSearch/+/621584/1/src/EntitySearchElastic.php>,
 which indeed hides the error...
  But what could be causing it?
  
  On 
https://api.kunstmuseum.nl/w/index.php?search=figure=Special%3ASearch=1
 It's also showing `An error has occurred while searching: We could not 
complete your search due to a temporary problem. Please try again later. ` But 
there is nothing related in the docker logs unfortunately to point to a cause.
  
  In /var/log/mediawiki/error.log I saw the same error with a stacktrace.
  
2020-10-08 07:43:01 807591ec944f my_wiki: [46cfcdb2d547f4ac352ea8f1] 
/w/api.php?action=wbsearchentities=gogh=json=en=en=item
   ErrorException from line 333 of /var/www/html/includes/debug/MWDebug.php: 
PHP Warning: 
{"type":"error","message":"cirrussearch-backend-error","params":[]} [Called 
from Wikibase\Search\Elastic\EntitySearchElastic::getRankedSearchResults in 
/var/www/html/extensions/WikibaseCirrusSearch/src/EntitySearchElastic.php at 
line 318]
#0 [internal function]: MWExceptionHandler::handleError(integer, string, 
string, integer, array)
#1 /var/www/html/includes/debug/MWDebug.php(333): trigger_error(string, 
integer)
#2 /var/www/html/includes/debug/MWDebug.php(188): 
MWDebug::sendMessage(string, array, string, integer)
#3 /var/www/html/includes/GlobalFunctions.php(1079): 
MWDebug::warning(string, integer, integer, string)
#4 
/var/www/html/extensions/WikibaseCirrusSearch/src/EntitySearchElastic.php(318): 
wfLogWarning(string)
#5 
/var/www/html/extensions/Wikibase/repo/includes/Api/CombinedEntitySearchHelper.php(48):
 Wikibase\Search\Elastic\EntitySearchElastic->getRankedSearchResults(string, 
string, string, integer, boolean)
#6 
/var/www/html/extensions/Wikibase/repo/includes/Api/TypeDispatchingEntitySearchHelper.php(54):
 Wikibase\Repo\Api\CombinedEntitySearchHelper->getRankedSearchResults(string, 
string, string, integer, boolean)
#7 
/var/www/html/extensions/Wikibase/repo/includes/Api/SearchEntities.php(120): 
Wikibase\Repo\Api\TypeDispatchingEntitySearchHelper->getRankedSearchResults(string,
 string, string, integer, boolean)
#8 
/var/www/html/extensions/Wikibase/repo/includes/Api/SearchEntities.php(254): 
Wikibase\Repo\Api\SearchEntities->getSearchEntries(array)
#9 /var/www/html/includes/api/ApiMain.php(1598): 
Wikibase\Repo\Api\SearchEntities->execute()
#10 /var/www/html/includes/api/ApiMain.php(537): ApiMain->executeAction()
#11 /var/www/html/includes/api/ApiMain.php(508): 
ApiMain->executeActionWithErrorHandling()
#12 /var/www/html/api.php(87): ApiMain->execute()
#13 {main}

TASK DETAIL
  https://phabricator.wikimedia.org/T264431

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: helmo
Cc: Addshore, Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, darthmon_wmde, 
Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Asahiko, despens, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T260328: Documentation on setting up https access to the container services

2020-10-01 Thread helmo
helmo added a comment.


  In the DockerWikibaseQueryService I also noticed CORS errors in the Firefox 
developer console,
  
  > Cross-Origin Request Blocked: The Same Origin Policy disallows reading the 
remote resource at 
https://api.kunstmuseum.nl/w/api.php?origin=*=wbsearchentities=json=50=0=en=en=f=item.
 (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).
  
  After some experimentation it seems to help to add the lines below to the 
Apache reverse proxy vhost for the api site.
  
Header set Access-Control-Allow-Origin *
Header set Access-Control-Allow-Methods "POST, GET, OPTIONS, PUT, DELETE"
Header set Access-Control-Allow-Headers "Origin, Content-Type, Accept, 
Authorization, X-Request-With, x-requested-with"
  
  I'm not sure if this opens things up too much though...

TASK DETAIL
  https://phabricator.wikimedia.org/T260328

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: helmo
Cc: Addshore, Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, darthmon_wmde, 
Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Asahiko, despens, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T260328: Documentation on setting up https access to the container services

2020-09-25 Thread helmo
helmo added a comment.


  I've created https://github.com/wmde/wikibase-docker/pull/133 for the network 
alias point above.

TASK DETAIL
  https://phabricator.wikimedia.org/T260328

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: helmo
Cc: Addshore, Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, darthmon_wmde, 
Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Asahiko, despens, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T260328: Documentation on setting up https access to the container services

2020-09-24 Thread helmo
helmo added a comment.


  Thanks again for your help.
  
  One extra thing to note here is the step to reset the wdqs data via
  `docker-compose exec wdqs rm   /wdqs/data/data.jnl` and a restart of the wdqs 
container.

TASK DETAIL
  https://phabricator.wikimedia.org/T260328

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: helmo
Cc: Addshore, Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, darthmon_wmde, 
Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Asahiko, despens, Wikidata-bugs, aude, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs


[Wikidata-bugs] [Maniphest] T260328: Documentation on setting up https access to the container services

2020-08-13 Thread helmo
helmo created this task.
helmo added a project: Wikibase-Containers.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.

TASK DESCRIPTION
  I've not found a single source of truth about securing access to the wikibase 
data via https. I did search quite a bit and read alot on https://addshore.com
  I was expecting a section in the main README file as it's pretty much the 
norm these days.
  
  Setting up a proxy vhost was not a problem, but getting the applications to 
work is. For one the main mediawiki search box would stubbornly try to connect 
to port 80, which my browser refuses because the rest of the page used port 
443. After some digging I found a way to resolve that in LocalSettings.php. An 
issue with the wqds-updater service is however still bugging me and could very 
well be related to this.
  
  I've tried to ask that on 
  stackoverflow.com 
<https://stackoverflow.com/questions/62816363/why-does-the-query-service-in-my-wikibase-installed-through-docker-not-contain-c?noredirect=1#comment112078559_62816363>
 and via Telegram.
  
  Here's a first draft of such a section that might fit in master/README.md 
<https://github.com/wmde/wikibase-docker/blob/master/README.md>. It clearly 
needs work as my setup is not fully functional yet.
  
  SSL
  ---
  
  There are multiple ways to get wikibase behind an SSL enabled address.
  
  Example proxy statements to add to an Apache vhost, (make sure mod_proxy_http 
is enabled)
  

ServerName api.example.com
...
ProxyRequests Off

ProxyPass /.well-known/acme-challenge "!"
ProxyPreserveHost On
ProxyPass / http://127.0.0.1:8181/ max=100
ProxyPassReverse / http://127.0.01:8181/
...


ServerName query.example.com
...
ProxyRequests Off

ProxyPass /.well-known/acme-challenge "!"
ProxyPreserveHost On
ProxyPass / http://127.0.0.1:8182/ max=100
ProxyPassReverse / http://127.0.01:8182/
...
  
  When you setup a proxy in front of it you have to add extra variables to the 
docker-compose.yml file.
  In the quickstatements section:
  
services:
  quickstatements:
environment:
   'WB_PUBLIC_HOST_AND_PORT=https://api.example.com:443
  - WB_PUBLIC_SCHEME_HOST_AND_PORT=https://api.example.com:443
  
  And an extra alias in the wikibase section:
  
services:
  wikibase:
networks:
  default:
aliases:
 - wikibase.svc
 - api.example.com
  
  And you need to update the LocalSettings.php to have:
  
$wgServer = "https://api.example.com;;

# And possibly also 
$_SERVER['HTTPS'] = 'on';

TASK DETAIL
  https://phabricator.wikimedia.org/T260328

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: helmo
Cc: Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, darthmon_wmde, Jelabra, 
Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Asahiko, despens, Wikidata-bugs, aude, Addshore, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs