[Wikidata-bugs] [Maniphest] T350348: results of query.wikidata are unstable (besides caching issues)

[Herzi.Pinki]

Herzi.Pinki added a comment.


  In T350348#9328058 , 
@Gehel wrote:
  
  > I'm marking this as resolved as we have a working query. Blazegraph being 
unmaintained, reporting the issue upstream is not really helpful.
  
  Houston, we have a maintenance problem!

TASK DETAIL
  https://phabricator.wikimedia.org/T350348

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel, Herzi.Pinki
Cc: Gehel, Aklapper, dcausse, Herzi.Pinki, Danny_Benjafield_WMDE, 
Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, KimKelting, merbst, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, 
Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T350348: results of query.wikidata are unstable (besides caching issues)

[Gehel]

Gehel closed this task as "Resolved".
Gehel claimed this task.
Gehel added a comment.


  I'm marking this as resolved as we have a working query. Blazegraph being 
unmaintained, reporting the issue upstream is not really helpful.

TASK DETAIL
  https://phabricator.wikimedia.org/T350348

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Gehel
Cc: Gehel, Aklapper, dcausse, Herzi.Pinki, Danny_Benjafield_WMDE, 
Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, 
Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, 
GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, 
Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, 
Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T350348: results of query.wikidata are unstable (besides caching issues)

[Herzi.Pinki]

Herzi.Pinki added a comment.


  @dcausse thanks for your investigations. Your query is 8 times faster than 
mine (optimizing is obviously not always the way to go) and it gives 165 
matches instead of my query that still gives 171.
  
  for me as a user of the frontend of wikidata query it is difficult to see 
what fails in the background, even, what is used in the background. Feel free 
to forward the issue to blazegraph. My problem seems to be solved be rewriting 
the query.

TASK DETAIL
  https://phabricator.wikimedia.org/T350348

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Herzi.Pinki
Cc: Aklapper, dcausse, Herzi.Pinki, Danny_Benjafield_WMDE, Astuthiodit_1, 
AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T350348: results of query.wikidata are unstable (besides caching issues)

[dcausse]

dcausse added a comment.


  @Herzi.Pinki sorry to see that this problem is hitting your query again, I 
still believe that this might be a bug in blazegraph possibly related to how it 
optimizes it query plan.
  I think the section to cause much trouble to blazegraph is the named query:
  
SELECT DISTINCT ?item WHERE {
?item wdt:P17 wd:Q40 ;
  wdt:P625 [] ;
  wdt:P31/wdt:P279* wd:Q271669 .
minus {?item wdt:P31/wdt:P279* wd:Q46831 .}
minus {?item wdt:P31/wdt:P279* wd:Q39816 .}
filter not exists { ?item wdt:P131 [] }
}
  
  I seem to obtain better performances by disabling the blazegraph optimizer 
(`hint:Query hint:optimizer "None". `):
  
SELECT DISTINCT ?item WHERE {
hint:Query hint:optimizer "None".  
?item wdt:P17 wd:Q40 ;
  wdt:P625 [] ;
  wdt:P31/wdt:P279* wd:Q271669 .
minus {?item wdt:P31/wdt:P279* wd:Q46831 .}
minus {?item wdt:P31/wdt:P279* wd:Q39816 .}
filter not exists { ?item wdt:P131 [] }
}
  
  But telling blazegraph to disable its optimizer we uncover yet another issue:
  `BIND(IF(EXISTS { ?item p:P18 [] }, 'ff', 'ff') AS ?rgb)  .`
  no longer appears to work appropriately and have to be rewritten as:
  `BIND(IF(BOUND(?image), 'ff', 'ff') AS ?rgb) .`
  reusing the `?image` var which is attached in an optional clause couple lines 
before.
  
  I took the liberty to attempt a rewrite of your query as:
  
#defaultView:Map{"hide":"?rgb"}
SELECT ?item ?itemLabel ?itemDescription (GROUP_CONCAT(DISTINCT 
?whereLabel; SEPARATOR=', ') AS ?whereLabels) (SAMPLE(?image) AS ?image) ?coord 
?rgb ?layer WITH {
  SELECT DISTINCT ?item WHERE {
hint:Query hint:optimizer "None".  
?item wdt:P17 wd:Q40 .
?item wdt:P625 [] .
?item wdt:P31/wdt:P279* wd:Q271669 . #Q35145263 . # Q271669 . #

#?item wdt:P31/wdt:P279* wd:Q35509 .
minus {?item wdt:P31/wdt:P279* wd:Q46831 .}

filter not exists {
  ?item wdt:P131 ?wo
  }
  #minus {?item wdt:P31/wdt:P279* wd:Q27686 .}
  #minus {?item wdt:P31/wdt:P279* wd:Q1444 .}
  minus {?item wdt:P31/wdt:P279* wd:Q39816 .}
  }
} AS %subquery1 WHERE {
  INCLUDE %subquery1 .
  ?item wdt:P31 [] .
  ?item p:P625 ?coordStatement .
  ?coordStatement ps:P625 ?coord .
  #MINUS { ?coordStatement prov:wasDerivedFrom/pr:P143 wd:Q169514 } # 
imported from Wikimedia project: Swedish Wikipedia 
  #MINUS { ?coordStatement prov:wasDerivedFrom/pr:P143 wd:Q837615 } # 
imported from Wikimedia project: Cebuano Wikipedia 
  #MINUS { ?coordStatement prov:wasDerivedFrom/pr:P248 wd:Q1194038 } # 
stated in: GEOnet Names Server
  OPTIONAL {
?item wdt:P131 ?where .
OPTIONAL {
  ?where rdfs:label ?whereLiteral .
  FILTER(LANG(?whereLiteral) = 'de') .
}
  }
  BIND(IF(BOUND(?where), IF(BOUND(?whereLiteral), ?whereLiteral, 
STRAFTER(STR(?where), 'entity/')), 'no P131') AS ?whereLabel) .
  OPTIONAL { ?item wdt:P18 ?image }
  BIND(IF(BOUND(?image), 'ff', 'ff') AS ?rgb) .
  BIND(IF(BOUND(?image), IF(BOUND(?where), 'With Image & P131', 'With Image 
but without P131'), IF(BOUND(?where), 'Without Image but with P131', 'Without 
Image and without P131')) AS ?layer) .
  SERVICE wikibase:label { bd:serviceParam wikibase:language 
'[AUTO_LANGUAGE,de,en]' }
} GROUP BY ?item ?itemLabel ?itemDescription ?whereLabels ?coord ?rgb ?layer
  
  It does seem to be slightly faster, I also added two new layers to select of 
items with a P131  (hoping that it 
might ease detecting when this similar bug happens).
  
  Please let us know if this rewritten query suits your needs and if it helps 
mitigate the issue you're experiencing.

TASK DETAIL
  https://phabricator.wikimedia.org/T350348

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, Herzi.Pinki, Danny_Benjafield_WMDE, Astuthiodit_1, 
AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T350348: results of query.wikidata are unstable (besides caching issues)

[dcausse]

dcausse updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T350348

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: Aklapper, dcausse, Herzi.Pinki, Danny_Benjafield_WMDE, Astuthiodit_1, 
AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T350348: results of query.wikidata are unstable (besides caching issues)

[Herzi.Pinki]

Herzi.Pinki added a comment.


  as things are stochastic:
  Now the difference between landforms and mountains is in Hanauer Spitze 
https://www.wikidata.org/wiki/Q21878328 and Brunnkarspitze 
https://www.wikidata.org/wiki/Q21878293

TASK DETAIL
  https://phabricator.wikimedia.org/T350348

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Herzi.Pinki
Cc: Aklapper, dcausse, Herzi.Pinki, Danny_Benjafield_WMDE, Astuthiodit_1, 
AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T350348: results of query.wikidata are unstable (besides caching issues)

[Nikki]

Nikki updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T350348

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nikki
Cc: Aklapper, dcausse, Herzi.Pinki, Danny_Benjafield_WMDE, Astuthiodit_1, 
AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T350348: results of query.wikidata are unstable (besides caching issues)

[Maintenance_bot]

Maintenance_bot added a project: Wikidata.

TASK DETAIL
  https://phabricator.wikimedia.org/T350348

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: Aklapper, dcausse, Herzi.Pinki, Danny_Benjafield_WMDE, Astuthiodit_1, 
AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, 
Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, 
QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, 
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T350348: results of query.wikidata are unstable (besides caching issues)

[Herzi.Pinki]

Herzi.Pinki created this task.
Herzi.Pinki added a project: Wikidata-Query-Service.
Restricted Application added a subscriber: Aklapper.

TASK DESCRIPTION
  (see also 
https://www.wikidata.org/wiki/Wikidata:Report_a_technical_problem/WDQS_and_Search#query_does_not_reflect_current_state_(not_even_yesterday's_state)
 )
  
  in short:  mountain (Q8502) is a subclass (P279 
) of landform (Q271669). Querying for 
mountains thus should result in a subset of the corresponding query for all 
landforms (as any mountain also is a landform). This is not the case.
  
  caveat: the behaviour is not reproducible over long time spans, as after some 
days things seem to get synchronized again. But the time span is far beyond 
anything that could be explained by replication lags, different caching servers 
with non-sync caches, etc.
  
  **Steps to replicate the issue** (include links if applicable):
  
  - query for landforms: 
https://query.wikidata.org/sparql?query=%23defaultView%3AMap%7B%22hide%22%3A%22%3Frgb%22%7D%0ASELECT%20%3Fitem%20%3FitemLabel%20%3FitemDescription%20(GROUP_CONCAT(DISTINCT%20%3FwhereLabel%3B%20SEPARATOR%3D'%2C%20')%20AS%20%3FwhereLabels)%20(SAMPLE(%3Fimage)%20AS%20%3Fimage)%20%3Fcoord%20%3Frgb%20%3Flayer%20WITH%20%7B%0A%20%20SELECT%20DISTINCT%20%3Fitem%20WHERE%20%7B%0A%20%20%20%20%3Fitem%20wdt%3AP17%20wd%3AQ40%20.%0A%20%20%20%20%3Fitem%20wdt%3AP625%20%5B%5D%20.%0A%20%20%20%20%3Fitem%20wdt%3AP31%2Fwdt%3AP279*%20wd%3AQ271669%20.%20%23Q35145263%20.%20%23%20Q271669%20.%20%23%0A%0A%20%20%20%20%23%3Fitem%20wdt%3AP31%2Fwdt%3AP279*%20wd%3AQ35509%20.%0A%20%20%20%20minus%20%7B%3Fitem%20wdt%3AP31%2Fwdt%3AP279*%20wd%3AQ46831%20.%7D%0A%0A%20%20%20%20filter%20not%20exists%20%7B%0A%20%20%20%20%20%20%3Fitem%20wdt%3AP131%20%3Fwo%0A%20%20%20%20%20%20%7D%0A%20%20%23minus%20%7B%3Fitem%20wdt%3AP31%2Fwdt%3AP279*%20wd%3AQ27686%20.%7D%0A%20%20%23minus%20%7B%3Fitem%20wdt%3AP31%2Fwdt%3AP279*%20wd%3AQ1444%20.%7D%0A%20%20minus%20%7B%3Fitem%20wdt%3AP31%2Fwdt%3AP279*%20wd%3AQ39816%20.%7D%0A%20%20%7D%0A%7D%20AS%20%25subquery1%20WHERE%20%7B%0A%20%20INCLUDE%20%25subquery1%20.%0A%20%20%3Fitem%20wdt%3AP31%20%5B%5D%20.%0A%20%20%3Fitem%20p%3AP625%20%3FcoordStatement%20.%0A%20%20%3FcoordStatement%20ps%3AP625%20%3Fcoord%20.%0A%20%20%23MINUS%20%7B%20%3FcoordStatement%20prov%3AwasDerivedFrom%2Fpr%3AP143%20wd%3AQ169514%20%7D%20%23%20imported%20from%20Wikimedia%20project%3A%20Swedish%20Wikipedia%20%0A%20%20%23MINUS%20%7B%20%3FcoordStatement%20prov%3AwasDerivedFrom%2Fpr%3AP143%20wd%3AQ837615%20%7D%20%23%20imported%20from%20Wikimedia%20project%3A%20Cebuano%20Wikipedia%20%0A%20%20%23MINUS%20%7B%20%3FcoordStatement%20prov%3AwasDerivedFrom%2Fpr%3AP248%20wd%3AQ1194038%20%7D%20%23%20stated%20in%3A%20GEOnet%20Names%20Server%0A%20%20OPTIONAL%20%7B%0A%20%20%20%20%3Fitem%20wdt%3AP131%20%3Fwhere%20.%0A%20%20%20%20OPTIONAL%20%7B%0A%20%20%20%20%20%20%3Fwhere%20rdfs%3Alabel%20%3FwhereLiteral%20.%0A%20%20%20%20%20%20FILTER(LANG(%3FwhereLiteral)%20%3D%20'de')%20.%0A%20%20%20%20%7D%0A%20%20%7D%0A%20%20BIND(IF(BOUND(%3Fwhere)%2C%20IF(BOUND(%3FwhereLiteral)%2C%20%3FwhereLiteral%2C%20STRAFTER(STR(%3Fwhere)%2C%20'entity%2F'))%2C%20'no%20P131')%20AS%20%3FwhereLabel)%20.%0A%20%20OPTIONAL%20%7B%20%3Fitem%20wdt%3AP18%20%3Fimage%20%7D%0A%20%20BIND(IF(EXISTS%20%7B%20%3Fitem%20p%3AP18%20%5B%5D%20%7D%2C%20'ff'%2C%20'ff')%20AS%20%3Frgb)%20.%0A%20%20BIND(IF(EXISTS%20%7B%20%3Fitem%20p%3AP18%20%5B%5D%20%7D%2C%20'With%20image'%2C%20'Without%20image')%20AS%20%3Flayer)%20.%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20'%5BAUTO_LANGUAGE%2Cde%2Cen%5D'%20%7D%0A%7D%20GROUP%20BY%20%3Fitem%20%3FitemLabel%20%3FitemDescription%20%3FwhereLabels%20%3Fcoord%20%3Frgb%20%3Flayer
  
  - query for mountains: