DD063520 added a comment.
I was able to fix the above and to reload the data in the query service as described in: https://addshore.com/2019/11/changing-the-concept-uri-of-an-existing-wikibase-with-data/#more-4286 Now the problem occurs when creating a new updater instance. I got two type of errors depending if I set or not in the environment variable - WIKIBASE_SCHEME=https If I set it I get: wait-for-it.sh: waiting 300 seconds for linkedopendata.eu:80 wait-for-it.sh: linkedopendata.eu:80 is available after 0 seconds wait-for-it.sh: waiting 300 seconds for wdqs.svc:9999 wait-for-it.sh: wdqs.svc:9999 is available after 0 seconds Updating via http://wdqs.svc:9999/bigdata/namespace/wdq/sparql OpenJDK 64-Bit Server VM warning: Cannot open file /var/log/wdqs/wdqs-updater_jvm_gc.pid7.log due to No such file or directory #logback.classic pattern: %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n 17:13:15.030 [main] INFO org.wikidata.query.rdf.tool.Update - Starting Updater 0.3.10-SNAPSHOT (8bba8bfb0bbb12361c0e214fe482fbe15fcaa129) 17:13:16.018 [main] INFO o.w.q.r.t.change.ChangeSourceContext - Checking where we left off 17:13:16.019 [main] INFO o.w.query.rdf.tool.rdf.RdfRepository - Checking for left off time from the updater 17:13:16.213 [main] INFO o.w.query.rdf.tool.rdf.RdfRepository - Checking for left off time from the dump 17:13:16.255 [main] INFO o.w.q.r.t.change.ChangeSourceContext - Found start time in the RDF store: 2020-01-03T23:12:25Z 17:13:16.317 [main] ERROR org.wikidata.query.rdf.tool.Update - Error during updater run. java.lang.RuntimeException: org.apache.http.conn.HttpHostConnectException: Connect to linkedopendata.eu:443 [linkedopendata.eu/172.29.0.5] failed: Connection refused (Connection refused) at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:317) at org.wikidata.query.rdf.tool.change.RecentChangesPoller.doFetchRecentChanges(RecentChangesPoller.java:325) at org.wikidata.query.rdf.tool.change.RecentChangesPoller.fetchRecentChanges(RecentChangesPoller.java:314) at org.wikidata.query.rdf.tool.change.RecentChangesPoller.batch(RecentChangesPoller.java:338) at org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:162) at org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:38) at org.wikidata.query.rdf.tool.Updater.run(Updater.java:170) at org.wikidata.query.rdf.tool.Update.run(Update.java:174) at org.wikidata.query.rdf.tool.Update.main(Update.java:98) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to linkedopendata.eu:443 [linkedopendata.eu/172.29.0.5] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:84) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.getJson(WikibaseRepository.java:482) at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:314) ... 8 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:338) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134) ... 20 common frames omitted Exception in thread "main" java.lang.RuntimeException: org.apache.http.conn.HttpHostConnectException: Connect to linkedopendata.eu:443 [linkedopendata.eu/172.29.0.5] failed: Connection refused (Connection refused) at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:317) at org.wikidata.query.rdf.tool.change.RecentChangesPoller.doFetchRecentChanges(RecentChangesPoller.java:325) at org.wikidata.query.rdf.tool.change.RecentChangesPoller.fetchRecentChanges(RecentChangesPoller.java:314) at org.wikidata.query.rdf.tool.change.RecentChangesPoller.batch(RecentChangesPoller.java:338) at org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:162) at org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:38) at org.wikidata.query.rdf.tool.Updater.run(Updater.java:170) at org.wikidata.query.rdf.tool.Update.run(Update.java:174) at org.wikidata.query.rdf.tool.Update.main(Update.java:98) Caused by: org.apache.http.conn.HttpHostConnectException: Connect to linkedopendata.eu:443 [linkedopendata.eu/172.29.0.5] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:84) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.getJson(WikibaseRepository.java:482) at org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:314) ... 8 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:338) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134) ... 20 more The website is clearly reachable (https://linkedopendata.eu/). But if the connection goes directly from one container to the other as you described above then the ssl certificate will be missing. Currently it is exposed by an nginx server that is catching all incoming connections of the physical machine. If I do not set the environment variable then I get: wait-for-it.sh: waiting 300 seconds for linkedopendata.eu:80 wait-for-it.sh: linkedopendata.eu:80 is available after 0 seconds wait-for-it.sh: waiting 300 seconds for wdqs.svc:9999 wait-for-it.sh: wdqs.svc:9999 is available after 0 seconds Updating via http://wdqs.svc:9999/bigdata/namespace/wdq/sparql OpenJDK 64-Bit Server VM warning: Cannot open file /var/log/wdqs/wdqs-updater_jvm_gc.pid7.log due to No such file or directory #logback.classic pattern: %d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n 17:14:15.939 [main] INFO org.wikidata.query.rdf.tool.Update - Starting Updater 0.3.10-SNAPSHOT (8bba8bfb0bbb12361c0e214fe482fbe15fcaa129) 17:14:16.905 [main] INFO o.w.q.r.t.change.ChangeSourceContext - Checking where we left off 17:14:16.905 [main] INFO o.w.query.rdf.tool.rdf.RdfRepository - Checking for left off time from the updater 17:14:17.112 [main] INFO o.w.query.rdf.tool.rdf.RdfRepository - Found left off time from the updater 17:14:17.112 [main] INFO o.w.q.r.t.change.ChangeSourceContext - Found start time in the RDF store: 2020-01-04T00:07:25Z 17:14:17.302 [main] INFO o.w.q.r.t.change.RecentChangesPoller - Got 1 changes, from Q9594@10598@20200104000726|10585 to Q9594@10598@20200104000726|10585 17:14:17.507 [update 0] WARN org.wikidata.query.rdf.tool.Updater - Contained error syncing. Giving up on Q9594 org.wikidata.query.rdf.tool.rdf.Munger$BadSubjectException: Unrecognized subjects: [https://linkedopendata.eu/entity/statement/Q9594-1691b013-4544-392d-1c8f-3c2eb0d75a40, https://linkedopendata.eu/entity/Q9594]. Expected only sitelinks and subjects starting with http://linkedopendata.eu/wiki/Special:EntityData/ and [http://linkedopendata.eu/entity/] at org.wikidata.query.rdf.tool.rdf.Munger$MungeOperation.finishCommon(Munger.java:941) at org.wikidata.query.rdf.tool.rdf.Munger$MungeOperation.munge(Munger.java:493) at org.wikidata.query.rdf.tool.rdf.Munger.munge(Munger.java:148) at org.wikidata.query.rdf.tool.rdf.Munger.mungeWithValues(Munger.java:182) at org.wikidata.query.rdf.tool.Updater.handleChange(Updater.java:421) at org.wikidata.query.rdf.tool.Updater.lambda$fetchDataFromWikibaseAndMunge$7(Updater.java:305) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 17:14:17.544 [main] INFO org.wikidata.query.rdf.tool.Updater - Polled up to 2020-01-04T00:07:26Z at (0.0, 0.0, 0.0) updates per second and (0.0, 0.0, 0.0) milliseconds per second 17:14:17.565 [main] INFO o.w.q.r.t.change.RecentChangesPoller - Got no real changes 17:14:17.566 [main] INFO org.wikidata.query.rdf.tool.Updater - Sleeping for 10 secs 17:14:27.593 [main] INFO o.w.q.r.t.change.RecentChangesPoller - Got no real changes 17:14:27.594 [main] INFO org.wikidata.query.rdf.tool.Updater - Sleeping for 10 secs which totally makes sense. The S in httpS is missing. Have you any clue how to get around this? If I set this variable $wgServer with http instead of https then the search bar will not work again since the frontent does not allow not secure connections. TASK DETAIL https://phabricator.wikimedia.org/T240962 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, DD063520 Cc: Aklapper, DD063520, Iflorez, darthmon_wmde, alaa_wmde, Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Asahiko, Wikidata-bugs, aude, Lydia_Pintscher, Addshore, Mbch331
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
