[Wikidata-bugs] [Maniphest] [Commented On] T209034: Incomplete continuation in MWAPI

[Lucas_Werkmeister_WMDE]

Lucas_Werkmeister_WMDE added a comment.

I experimented with this a bit, and applying the following patch is enough to yield me 458 results by GZWDer instead of 35:

diff --git a/blazegraph/src/main/java/org/wikidata/query/rdf/blazegraph/mwapi/MWApiServiceCall.java b/blazegraph/src/main/java/org/wikidata/query/rdf/blazegraph/mwapi/MWApiServiceCall.java
index 2993f8e..396b700 100644
--- a/blazegraph/src/main/java/org/wikidata/query/rdf/blazegraph/mwapi/MWApiServiceCall.java
+++ b/blazegraph/src/main/java/org/wikidata/query/rdf/blazegraph/mwapi/MWApiServiceCall.java
@@ -315,10 +315,10 @@ public class MWApiServiceCall implements MockIVReturningServiceCall, BigdataServ
         // Note though that XPathExpression is not thread-safe. Maybe use ThreadLocal?
         XPathExpression itemsXPath = path.compile(template.getItemsPath());
         NodeList nodes = (NodeList) itemsXPath.evaluate(doc, XPathConstants.NODESET);
-        if (nodes.getLength() == 0) {
-            return null;
-        }
         IBindingSet[] results = new IBindingSet[nodes.getLength()];
+        if (results.length == 0) {
+            return new ResultWithContinue(results, searchContinue);
+        }
         final Map<OutputVariable, XPathExpression> compiledVars = new HashMap<>();
         // TODO: would be nice to convert it to stream _expression_, but xpath
         // throws, and lambdas do not work with checked exceptions.

The problem is that going through the continuations, all the way back to 2013 (the first part of the garvcontinue parameter is clearly a timestamp), actually takes a lot of requests: 148, to be precise. And while they probably won’t take quite as long in production, with WDQS and the API server being in the same datacenter, that’s still a pretty significant slowdown. We might need to put some limit on continuation across empty results after all.

TASK DETAIL

https://phabricator.wikimedia.org/T209034

EMAIL PREFERENCES

https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Aklapper, Smalyshev, Lucas_Werkmeister_WMDE, Nikki, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, D3r1ck01, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331

_______________________________________________
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs