[Wikidata-bugs] [Maniphest] T199219: WDQS should use internal endpoint to communicate to Wikidata

[Gehel]

Gehel closed this task as "Resolved".

TASK DETAIL
  https://phabricator.wikimedia.org/T199219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Mstyles, Gehel
Cc: dcausse, Ladsgroup, akosiaris, BBlack, Aklapper, Smalyshev, Gehel, 
Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, 
Jony, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, 
merbst, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T199219: WDQS should use internal endpoint to communicate to Wikidata

[dcausse]

dcausse moved this task from Needs review to To Be Deployed on the 
Discovery-Search (Current work) board.
dcausse assigned this task to Mstyles.

TASK DETAIL
  https://phabricator.wikimedia.org/T199219

WORKBOARD
  https://phabricator.wikimedia.org/project/board/1227/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Mstyles, dcausse
Cc: dcausse, Ladsgroup, akosiaris, BBlack, Aklapper, Smalyshev, Gehel, 
Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, 
Jony, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, 
merbst, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T199219: WDQS should use internal endpoint to communicate to Wikidata

[akosiaris]

akosiaris added a comment.


  In T199219#6855641 , 
@Gehel wrote:
  
  > In terms of implementation in our new updater, the comment from @BBlack is 
the starting point:
  >
  > In T199219#4416896 , 
@BBlack wrote:
  >
  >> For this very particular case, the simplest way would be to do your 
language/platform/library's equivalent of:
  >>
  >>   curl -H 'Host: www.wikidata.org' 
'https://appservers-ro.discovery.wmnet/wiki/Special:EntityData/Q2408871.ttl?nocache=1530836328152&flavor=dump'
  >>
  >> That is, use the internal service endpoint hostname in the URI for TLS 
connection purposes, but then explicitly set the request `Host` header to 
`www.wikidata.org` for use at the HTTP level.
  
  Indeed. But with a minor correction, instead of `appservers-ro` please use 
instead `api-ro` in order to hit the API cluster as the appserver cluster is 
meant to be the //end-user browser serving cluster//.

TASK DETAIL
  https://phabricator.wikimedia.org/T199219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: akosiaris
Cc: dcausse, Ladsgroup, akosiaris, BBlack, Aklapper, Smalyshev, Gehel, 
MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Jony, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T199219: WDQS should use internal endpoint to communicate to Wikidata

[Gehel]

Gehel added a comment.


  In terms of implementation in our new updater, the comment from @BBlack is 
the starting point:
  
  In T199219#4416896 , 
@BBlack wrote:
  
  > For this very particular case, the simplest way would be to do your 
language/platform/library's equivalent of:
  >
  >   curl -H 'Host: www.wikidata.org' 
'https://appservers-ro.discovery.wmnet/wiki/Special:EntityData/Q2408871.ttl?nocache=1530836328152&flavor=dump'
  >
  > That is, use the internal service endpoint hostname in the URI for TLS 
connection purposes, but then explicitly set the request `Host` header to 
`www.wikidata.org` for use at the HTTP level.

TASK DETAIL
  https://phabricator.wikimedia.org/T199219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: dcausse, Ladsgroup, akosiaris, BBlack, Aklapper, Smalyshev, Gehel, 
MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Jony, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T199219: WDQS should use internal endpoint to communicate to Wikidata

[MPhamWMF]

MPhamWMF set the point value for this task to "5".

TASK DETAIL
  https://phabricator.wikimedia.org/T199219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MPhamWMF
Cc: dcausse, Ladsgroup, akosiaris, BBlack, Aklapper, Smalyshev, Gehel, 
MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Jony, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T199219: WDQS should use internal endpoint to communicate to Wikidata

[Gehel]

Gehel added a comment.


  In T199219#6836794 , 
@dcausse wrote:
  
  > The new updater is currently running on the analytics network (working on 
getting k8s deployment reading), we could set it up to appservers-ro but I 
think a hole needs to be open between the two networks (see similar issue in 
T274951 ).
  
  We're moving to production on k8s soon enough, I don't think we should fix 
this on the analytics network.

TASK DETAIL
  https://phabricator.wikimedia.org/T199219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: dcausse, Ladsgroup, akosiaris, BBlack, Aklapper, Smalyshev, Gehel, 
MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Jony, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T199219: WDQS should use internal endpoint to communicate to Wikidata

[MPhamWMF]

MPhamWMF moved this task from Scaling to Current work on the 
Wikidata-Query-Service board.
MPhamWMF added a project: Discovery-Search (Current work).

TASK DETAIL
  https://phabricator.wikimedia.org/T199219

WORKBOARD
  https://phabricator.wikimedia.org/project/board/891/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: MPhamWMF
Cc: dcausse, Ladsgroup, akosiaris, BBlack, Aklapper, Smalyshev, Gehel, 
MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Jony, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T199219: WDQS should use internal endpoint to communicate to Wikidata

[Gehel]

Gehel added a parent task: T244590: [Epic] Rework the WDQS updater as an event 
driven application.

TASK DETAIL
  https://phabricator.wikimedia.org/T199219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: dcausse, Ladsgroup, akosiaris, BBlack, Aklapper, Smalyshev, Gehel, 
MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Jony, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T199219: WDQS should use internal endpoint to communicate to Wikidata

[dcausse]

dcausse added a comment.


  The new updater is currently running on the analytics network (working on 
getting k8s deployment reading), we could set it up to appservers-ro but I 
think a hole needs to be open between the two networks (see similar issue in 
T274951 ).

TASK DETAIL
  https://phabricator.wikimedia.org/T199219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: dcausse, Ladsgroup, akosiaris, BBlack, Aklapper, Smalyshev, Gehel, 
MPhamWMF, CBogen, Akuckartz, Nandana, Namenlos314, Jony, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T199219: WDQS should use internal endpoint to communicate to Wikidata

[Gehel]

Gehel added a comment.


  The new Flink based WDQS updater (T244590 
) will mitigate the potential 
caching issues since it will be running centrally and not duplicate the work 
for each WDQS node. Since this is expected to be completed this quarter, it 
make sense to leave the current updater as-is, but ensure we go through the 
proper channels for the new one.

TASK DETAIL
  https://phabricator.wikimedia.org/T199219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: Ladsgroup, akosiaris, BBlack, Aklapper, Smalyshev, Gehel, MPhamWMF, CBogen, 
Akuckartz, Nandana, Namenlos314, Jony, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Vali.matei, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T199219: WDQS should use internal endpoint to communicate to Wikidata

[akosiaris]

akosiaris added a comment.


  For what is worth, we now have the services proxy (envoy based) with 
persistent connections and doing TLS on its own so any costs from switching to 
TLS connections to the internal LVS services will be largely mitigated. In 
fact, if anything I expect the latencies from that part of the equation to 
decrease since it won't have to go through a proxy and the edge caches. The 
question of whether bypassing the edge caches will hugely increase the load on 
mediawiki still stands, but there have been many changes on the mediawiki 
caching infrastructure too (e.g. we now have onhost memcached) so that might 
very well be largely mitigated as well.
  
  I think we ought to revisit this indeed. Having the updater go through an 
extra 4 (outgoing proxy + 3 layers of edge caches) layers of the 
infrastructure, one of which is in NO WAY deemed critical to have High 
Availability (the outgoing proxy) doesn't help with either easy debugging nor 
ease of operations during maintenance/emergencies.

TASK DETAIL
  https://phabricator.wikimedia.org/T199219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: akosiaris
Cc: Ladsgroup, akosiaris, BBlack, Aklapper, Smalyshev, Gehel, MPhamWMF, CBogen, 
Akuckartz, Nandana, Namenlos314, Jony, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Vali.matei, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T199219: WDQS should use internal endpoint to communicate to Wikidata

[Gehel]

Gehel raised the priority of this task from "Low" to "High".
Gehel moved this task from Operations to All WDQS-related tasks on the 
Wikidata-Query-Service board.
Gehel added a comment.


  Let's reprioritize this given the recent issues we've seen.

TASK DETAIL
  https://phabricator.wikimedia.org/T199219

WORKBOARD
  https://phabricator.wikimedia.org/project/board/891/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: Ladsgroup, akosiaris, BBlack, Aklapper, Smalyshev, Gehel, MPhamWMF, CBogen, 
Akuckartz, Nandana, Namenlos314, Jony, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Vali.matei, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs