| Smalyshev added a comment. |
Aren't clients making requests to Kartotherian? If they are, then X-Client-IP will be set to the external (to our prod environment) client issuing the request.
Yes. But when Kartotherian calls to WDQS, it calls external Varnish endpoint (now), which might replace X-Client-IP (not 100% sure about it, needs checking).
On the technical side, though, changing the UA for every client is IMHO a bad practice and should not even be considered a solution here.
I am not sure why. What's wrong with changing the UA?
But I am not hard-set on the UA, that'd just make it work automatically, but if it puts it in other place as long as Java can extract it it'd be fine I think.
There is precedent for this: the recommendation api service uses the internal WDQS cluster and issues requests to it upon receiving client requests.
The difference here is whether the queries produced by the service are ours (i.e. written by WMF stuff or trusted volunteers, etc., approved as part of code review process, etc.) or just anybody can run any query there. In the former case, we are reasonably sure we can have certain guarantees. In the latter case, the guarantees are much worse - even with current throttling systems, misbehaving bots launching an avalanche of heavy queries can impact the service.
This is why we split endpoints into external - with less guarantees - and internal, which is supposed to only run known queries that we know who wrote them and why and be much more stable.
The differentiator here is not who initiates the query, but who writes the query itself. As I understand, Kartotherian runs arbitrary user queries, which can be very heavy and unoptimized, and we have little visibility into who produces these queries.
The original idea of the internal WDQS cluster was that internal services requiring it should function regardless of the external client load faced by WDQS directly.
True, but if we allow to run same queries on both services, then essentially we wouldn't be able to guarantee that. See T195559: Consider switching SPARQL endpoint for Karthoterian to WDQS internal cluster for discussion on this.
It wasn't clear enough to me from the task desc that throttling is the main purpose of this.
Sorry, yes, the main reason is (proper) throttling, see parent T195438. The problem is that now Kartotherian is treated as a single client, which means its throttling limits apply on all Kartotherian requests as if they were coming from a single client. With more visibility into Kartotherian clients, they will each have their own throttling limits, so one heavy client would not impact other, more benign, clients.
Cc: mobrovac, MSantos, Gehel, Aklapper, MaxSem, Pnorman, Mholloway, Smalyshev, Amatissart, Lahi, Gq86, Looniverse, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, Orienteerix, merbst, LawExplorer, debt, JGirault, Jonas, phabyogi, Xmlizer, Susannaanas, lxbarth, Eevans, jkroll, Planemad, Hardikj, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Yurik, Jdforrester-WMF, Mbch331, Jay8g
_______________________________________________ Wikidata-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
