[Wikidata-bugs] [Maniphest] T266574: Wikibase docker update strategy
helmo created this task. helmo added a project: Wikibase-Containers. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION I'm wondering about an update policy and what to expect. Pulling the wikibase docker images currently gives me MediaWiki 1.34.1 while 1.34.4 (and 1.35.0) is the latest according to https://www.mediawiki.org/wiki/Download On Docker Hub <https://hub.docker.com/r/wikibase/wikibase> I noticed `Last pushed 15 days ago by wikibasetravis` so I was hoping for 1.34.4. What do we need to do to stay up to date with security releases? It would be nice if the README <https://github.com/wmde/wikibase-docker/blob/master/README.md> or README-compose.md <https://github.com/wmde/wikibase-docker/blob/master/README-compose.md> would give some guidance about this. There is https://addshore.com/2019/01/wikibase-docker-mediawiki-wikibase-update/ but that's more about updating between major versions. I started by commenting on T197220 <https://phabricator.wikimedia.org/T197220> but that is about mediawiki in general. But it did give me the extra question why doesn't wikibase build ontop of the regular mediawiki container images? TASK DETAIL https://phabricator.wikimedia.org/T266574 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: helmo Cc: Hannolans, Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Asahiko, despens, Wikidata-bugs, aude, Addshore, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T264431: autocomplete in querybuilder is missing results
helmo closed this task as "Resolved". helmo claimed this task. helmo added a comment. The last question is probably explained by the type of search that is done. It's probably a prefix search while we expected a full text search. With large datasets it probably makes sense to do it this way. TASK DETAIL https://phabricator.wikimedia.org/T264431 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: helmo Cc: despens, Addshore, Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Asahiko, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T264431: autocomplete in querybuilder is missing results
helmo added a subscriber: despens. helmo added a comment. Together with @despens we found that in the search box most letters give a result, but not the 'g' which you would expect 'van Gogh' to be found. It's still a mystery... TASK DETAIL https://phabricator.wikimedia.org/T264431 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: helmo Cc: despens, Addshore, Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, darthmon_wmde, Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Asahiko, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T264431: autocomplete in querybuilder is missing results
helmo added a comment. We're a but further... Via Extension:CirrusSearch <https://www.mediawiki.org/wiki/Extension:CirrusSearch> I ended up on the CirrusSearch README <https://gerrit.wikimedia.org/g/mediawiki/extensions/CirrusSearch/+/HEAD/README>. Which mentioned the steps: Now run this script to generate your elasticsearch index: php $MW_INSTALL_PATH/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php Now remove $wgDisableSearchUpdate = true from LocalSettings.php. Updates should start heading to Elasticsearch. Next bootstrap the search index by running: php $MW_INSTALL_PATH/extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipLinks --indexOnSkip php $MW_INSTALL_PATH/extensions/CirrusSearch/maintenance/ForceSearchIndex.php --skipParse Note that this can take some time. For large wikis read "Bootstrapping large wikis" below. After doing those steps I started getting results on the search page <https://api.kunstmuseum.nl/w/index.php?search=gogh+=Special:Search=1> but still not in the search box URL <https://api.kunstmuseum.nl/w/api.php?action=wbsearchentities=gogh=json=en=en=item> Is there a maintenance script that should have triggered this automatically from docker-compsose? TASK DETAIL https://phabricator.wikimedia.org/T264431 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: helmo Cc: Addshore, Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, darthmon_wmde, Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Asahiko, despens, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T264431: autocomplete in querybuilder is missing results
helmo added a comment. No, here's the log after restarting elasticsearch and attempting a few searches: wikibase@wikibase:~/wikibase-docker$ docker-compose up elasticsearch Starting wikibase-docker_elasticsearch_1 ... done Attaching to wikibase-docker_elasticsearch_1 elasticsearch_1| OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. elasticsearch_1| OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=0 elasticsearch_1| [2020-10-13T12:14:50,181][INFO ][o.e.e.NodeEnvironment ] [7DEERnP] using [1] data paths, mounts [[/ (overlay)]], net usable_space [4.8gb], net total_space [14.9gb], types [overlay] elasticsearch_1| [2020-10-13T12:14:50,186][INFO ][o.e.e.NodeEnvironment ] [7DEERnP] heap size [495.3mb], compressed ordinary object pointers [true] elasticsearch_1| [2020-10-13T12:14:50,189][INFO ][o.e.n.Node ] [7DEERnP] node name derived from node ID [7DEERnPBTzujGriDPZxI8w]; set [node.name] to override elasticsearch_1| [2020-10-13T12:14:50,191][INFO ][o.e.n.Node ] [7DEERnP] version[6.5.4], pid[1], build[default/tar/d2ef93d/2018-12-17T21:17:40.758843Z], OS[Linux/4.19.0-10-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.1/11.0.1+13] elasticsearch_1| [2020-10-13T12:14:50,192][INFO ][o.e.n.Node ] [7DEERnP] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.T6CKFqYS, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar] elasticsearch_1| [2020-10-13T12:14:52,638][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [aggs-matrix-stats] elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [analysis-common] elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [ingest-common] elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [lang-expression] elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [lang-mustache] elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [lang-painless] elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [mapper-extras] elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [parent-join] elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [percolator] elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [rank-eval] elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [reindex] elasticsearch_1| [2020-10-13T12:14:52,639][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [repository-url] elasticsearch_1| [2020-10-13T12:14:52,640][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [transport-netty4] elasticsearch_1| [2020-10-13T12:14:52,640][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [tribe] elasticsearch_1| [2020-10-13T12:14:52,640][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [x-pack-ccr] elasticsearch_1| [2020-10-13T12:14:52,640][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [x-pack-core] elasticsearch_1| [2020-10-13T12:14:52,640][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [x-pack-deprecation] elasticsearch_1| [2020-10-13T12:14:52,640][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [x-pack-graph] elasticsearch_1| [2020-10-13T12:14:52,640][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [x-pack-logstash] elasticsearch_1| [2020-10-13T12:14:52,640][INFO ][o.e.p.PluginsService ] [7DEERnP] loaded module [x-pack-ml] elasticsearch_1| [2020-10-13T12:14:52,640][INFO
[Wikidata-bugs] [Maniphest] T264431: autocomplete in querybuilder is missing results
helmo added a comment. Removing the `format=json` part of that url shows a warning of type 'error'... that looks like https://phabricator.wikimedia.org/T260276 The code change there only looks cosmetic in no longer showing this error. I manually patched in the changes <https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseCirrusSearch/+/621584/1/src/EntitySearchElastic.php>, which indeed hides the error... But what could be causing it? On https://api.kunstmuseum.nl/w/index.php?search=figure=Special%3ASearch=1 It's also showing `An error has occurred while searching: We could not complete your search due to a temporary problem. Please try again later. ` But there is nothing related in the docker logs unfortunately to point to a cause. In /var/log/mediawiki/error.log I saw the same error with a stacktrace. 2020-10-08 07:43:01 807591ec944f my_wiki: [46cfcdb2d547f4ac352ea8f1] /w/api.php?action=wbsearchentities=gogh=json=en=en=item ErrorException from line 333 of /var/www/html/includes/debug/MWDebug.php: PHP Warning: {"type":"error","message":"cirrussearch-backend-error","params":[]} [Called from Wikibase\Search\Elastic\EntitySearchElastic::getRankedSearchResults in /var/www/html/extensions/WikibaseCirrusSearch/src/EntitySearchElastic.php at line 318] #0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array) #1 /var/www/html/includes/debug/MWDebug.php(333): trigger_error(string, integer) #2 /var/www/html/includes/debug/MWDebug.php(188): MWDebug::sendMessage(string, array, string, integer) #3 /var/www/html/includes/GlobalFunctions.php(1079): MWDebug::warning(string, integer, integer, string) #4 /var/www/html/extensions/WikibaseCirrusSearch/src/EntitySearchElastic.php(318): wfLogWarning(string) #5 /var/www/html/extensions/Wikibase/repo/includes/Api/CombinedEntitySearchHelper.php(48): Wikibase\Search\Elastic\EntitySearchElastic->getRankedSearchResults(string, string, string, integer, boolean) #6 /var/www/html/extensions/Wikibase/repo/includes/Api/TypeDispatchingEntitySearchHelper.php(54): Wikibase\Repo\Api\CombinedEntitySearchHelper->getRankedSearchResults(string, string, string, integer, boolean) #7 /var/www/html/extensions/Wikibase/repo/includes/Api/SearchEntities.php(120): Wikibase\Repo\Api\TypeDispatchingEntitySearchHelper->getRankedSearchResults(string, string, string, integer, boolean) #8 /var/www/html/extensions/Wikibase/repo/includes/Api/SearchEntities.php(254): Wikibase\Repo\Api\SearchEntities->getSearchEntries(array) #9 /var/www/html/includes/api/ApiMain.php(1598): Wikibase\Repo\Api\SearchEntities->execute() #10 /var/www/html/includes/api/ApiMain.php(537): ApiMain->executeAction() #11 /var/www/html/includes/api/ApiMain.php(508): ApiMain->executeActionWithErrorHandling() #12 /var/www/html/api.php(87): ApiMain->execute() #13 {main} TASK DETAIL https://phabricator.wikimedia.org/T264431 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: helmo Cc: Addshore, Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, darthmon_wmde, Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Asahiko, despens, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T260328: Documentation on setting up https access to the container services
helmo added a comment. In the DockerWikibaseQueryService I also noticed CORS errors in the Firefox developer console, > Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://api.kunstmuseum.nl/w/api.php?origin=*=wbsearchentities=json=50=0=en=en=f=item. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). After some experimentation it seems to help to add the lines below to the Apache reverse proxy vhost for the api site. Header set Access-Control-Allow-Origin * Header set Access-Control-Allow-Methods "POST, GET, OPTIONS, PUT, DELETE" Header set Access-Control-Allow-Headers "Origin, Content-Type, Accept, Authorization, X-Request-With, x-requested-with" I'm not sure if this opens things up too much though... TASK DETAIL https://phabricator.wikimedia.org/T260328 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: helmo Cc: Addshore, Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, darthmon_wmde, Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Asahiko, despens, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T260328: Documentation on setting up https access to the container services
helmo added a comment. I've created https://github.com/wmde/wikibase-docker/pull/133 for the network alias point above. TASK DETAIL https://phabricator.wikimedia.org/T260328 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: helmo Cc: Addshore, Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, darthmon_wmde, Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Asahiko, despens, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T260328: Documentation on setting up https access to the container services
helmo added a comment. Thanks again for your help. One extra thing to note here is the step to reset the wdqs data via `docker-compose exec wdqs rm /wdqs/data/data.jnl` and a restart of the wdqs container. TASK DETAIL https://phabricator.wikimedia.org/T260328 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: helmo Cc: Addshore, Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, darthmon_wmde, Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Asahiko, despens, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] T260328: Documentation on setting up https access to the container services
helmo created this task. helmo added a project: Wikibase-Containers. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION I've not found a single source of truth about securing access to the wikibase data via https. I did search quite a bit and read alot on https://addshore.com I was expecting a section in the main README file as it's pretty much the norm these days. Setting up a proxy vhost was not a problem, but getting the applications to work is. For one the main mediawiki search box would stubbornly try to connect to port 80, which my browser refuses because the rest of the page used port 443. After some digging I found a way to resolve that in LocalSettings.php. An issue with the wqds-updater service is however still bugging me and could very well be related to this. I've tried to ask that on stackoverflow.com <https://stackoverflow.com/questions/62816363/why-does-the-query-service-in-my-wikibase-installed-through-docker-not-contain-c?noredirect=1#comment112078559_62816363> and via Telegram. Here's a first draft of such a section that might fit in master/README.md <https://github.com/wmde/wikibase-docker/blob/master/README.md>. It clearly needs work as my setup is not fully functional yet. SSL --- There are multiple ways to get wikibase behind an SSL enabled address. Example proxy statements to add to an Apache vhost, (make sure mod_proxy_http is enabled) ServerName api.example.com ... ProxyRequests Off ProxyPass /.well-known/acme-challenge "!" ProxyPreserveHost On ProxyPass / http://127.0.0.1:8181/ max=100 ProxyPassReverse / http://127.0.01:8181/ ... ServerName query.example.com ... ProxyRequests Off ProxyPass /.well-known/acme-challenge "!" ProxyPreserveHost On ProxyPass / http://127.0.0.1:8182/ max=100 ProxyPassReverse / http://127.0.01:8182/ ... When you setup a proxy in front of it you have to add extra variables to the docker-compose.yml file. In the quickstatements section: services: quickstatements: environment: 'WB_PUBLIC_HOST_AND_PORT=https://api.example.com:443 - WB_PUBLIC_SCHEME_HOST_AND_PORT=https://api.example.com:443 And an extra alias in the wikibase section: services: wikibase: networks: default: aliases: - wikibase.svc - api.example.com And you need to update the LocalSettings.php to have: $wgServer = "https://api.example.com;; # And possibly also $_SERVER['HTTPS'] = 'on'; TASK DETAIL https://phabricator.wikimedia.org/T260328 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: helmo Cc: Aklapper, helmo, Samantha_Alipio_WMDE, Akuckartz, darthmon_wmde, Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Asahiko, despens, Wikidata-bugs, aude, Addshore, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs