[Wikidata-bugs] [Maniphest] T348877: Lexeme searches prefer forms over lemmas
EBernhardson moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board. EBernhardson added a comment. The example in the ticket looks to work as expected now TASK DETAIL https://phabricator.wikimedia.org/T348877 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, Gehel, Nikki, Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, Mahir256, QZanden, EBjune, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T348877: Lexeme searches prefer forms over lemmas
[Wikidata-bugs] [Maniphest] T348877: Lexeme searches prefer forms over lemmas
EBernhardson claimed this task. EBernhardson moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board. EBernhardson added a comment. The UI for adding statements is using wbsearchentities <https://www.wikidata.org/wiki/Special:ApiSandbox#action=wbsearchentities=json=en=plaintext=asse=en=lexeme=2> (explain <https://www.wikidata.org/w/api.php?action=wbsearchentities=asse=json=plaintext=en=en=lexeme=pretty>). Target results are L1191921 and L1144955. The method of scoring for websearchentities could be sumarized as bucketing results into 3 groups based on how well they match, and then sorting by popularity (statement count and incoming link counts) within those buckets. Of all the docs that make the best possible match (near_match on lemma or near_match on lexeme_forms.representation) the two target documents have the lowest popularity with zero incoming links and a single statement each. Reviewing a few of the documents that were not targeted but ranked higher, they also match lexme_forms.representation. In a more traditional search context using term frequencies the fact that the target lexmes have a single statement each would push them up in the ranking, but because wbsearchentities buckets the results isn't of giving them individual scores that doesn't happen here. One thing we could do is be less strict on the bucketing. In a quick test setting a dismax tie breaker of 0.02 gives these target documents a boost up to the top of the ranking. This is not directly configurable, it was set in the initial commit for WikibaseLexemeCirrusSearch and never changed. This does read from our profile service at least, so it shouldn't be too hard to add a custom profile parameter to control the dismax tie breaker and set this to something that works a bit better. What value is appropriate is hard to say, at 0.01 these docs get a boost up into the top-7, but not all the way to the top. Essentially what ends up pushing these docs to the top of the ranking with the tie breaker is that they match both the lemma and lexeme_forms.representation field, where the other docs only match one of the two fields. TASK DETAIL https://phabricator.wikimedia.org/T348877 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, Gehel, Nikki, Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, Mahir256, QZanden, EBjune, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T349519: Determine if IGUANA and TFT would fit our query analysis needs
EBernhardson set the point value for this task to "8". TASK DETAIL https://phabricator.wikimedia.org/T349519 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: dcausse, Aklapper, Danny_Benjafield_WMDE, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator
EBernhardson changed the point value for this task from "8" to "13". TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, EBernhardson Cc: pfischer, EBernhardson, dcausse, BTullis, Aklapper, bking, Danny_Benjafield_WMDE, Isabelladantes1983, Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator
EBernhardson set the point value for this task to "8". TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, EBernhardson Cc: pfischer, EBernhardson, dcausse, BTullis, Aklapper, bking, Danny_Benjafield_WMDE, Isabelladantes1983, Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T349519: Determine if IGUANA and TFT would fit our query analysis needs
EBernhardson moved this task from Incoming to Current work on the Wikidata-Query-Service board. EBernhardson added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T349519 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: dcausse, Aklapper, AWesterinen, Namenlos314, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T349512: Collect multiple sets of SPARQL queries
EBernhardson moved this task from Incoming to Current work on the Wikidata-Query-Service board. EBernhardson added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T349512 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: dcausse, Aklapper, Danny_Benjafield_WMDE, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator
EBernhardson moved this task from Incoming to Current work on the Wikidata-Query-Service board. EBernhardson added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T349095 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, EBernhardson Cc: pfischer, EBernhardson, dcausse, BTullis, Aklapper, bking, Danny_Benjafield_WMDE, Isabelladantes1983, Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T347333: Tune process_sparql_query_hourly so that it does not get killed by yarn
EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board. EBernhardson added a comment. Reran 2023-09-21T16:00:00, which was previously failing, with memory overhead unconfigured and with the new patch to repartition the input. This has run to completion without failing, should resolve the issue in the future. TASK DETAIL https://phabricator.wikimedia.org/T347333 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, EBernhardson Cc: EBernhardson, bking, Aklapper, dcausse, Danny_Benjafield_WMDE, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T347333: Tune process_sparql_query_hourly so that it does not get killed by yarn
EBernhardson added a comment. 8g was still insufficient, one of the failed jobs passed but the other three still had trouble. Increasing to 12g made it work, but if 8g is already excessive 12g is only more of the same. Returning to the earlier idea of forcing the job to be split up more, patch above adjusts the job to force it to spread the input across 200 partitions which will then spread across more executors and do less work per task. As long as the tasks aren't leaking memory between runs, and our problem isn't singular queries that blow up the whole stack, this will hopefully get things going. TASK DETAIL https://phabricator.wikimedia.org/T347333 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, EBernhardson Cc: EBernhardson, bking, Aklapper, dcausse, Danny_Benjafield_WMDE, Isabelladantes1983, Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T347333: Tune process_sparql_query_hourly so that it does not get killed by yarn
EBernhardson added a comment. Unfortunately the above patch doesn't seem to have worked. Spark turned the input into three tasks. They were all assigned to the same executor, the first two finished and the third caused the container to die after another ~45s due to memory constraints. Spark then spun up a new executor which was only ever assigned that one task, and it failed for same reason. My next guess was to try tuning spark.sql.files.maxPartitionBytes, documentated as `The maximum number of bytes to pack into a single partition when reading files.` Unfortunately while spark did make some extra partitions, 12 instead of 3, all the extra partitions were empty. I glanced over the other spark configuration related to reads and partitioning but I'm not seeing other knobs we can turn in that direction. Next guess is brute force, add some memory overhead until it stops complaining. The actual jvm heap doesn't seem to be overloaded, or at least the GC times prior to getting killed don't look concerning. We should be able to leave the heap at the current size. Tried 4g overhead, still failed. Tried 8g overhead, it still killed a task but with retries managed to finish. I'm not too thrilled to run everything with the 8g overhead, but we could go that way if we have to. TASK DETAIL https://phabricator.wikimedia.org/T347333 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, EBernhardson Cc: EBernhardson, bking, Aklapper, dcausse, Danny_Benjafield_WMDE, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T346456: Improve concurrency limits configuration of the wdqs updater
EBernhardson set the point value for this task to "3". TASK DETAIL https://phabricator.wikimedia.org/T346456 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Aklapper, bking, Clement_Goubert, dcausse, Danny_Benjafield_WMDE, Kappakayala, Astuthiodit_1, AWesterinen, Arnoldokoth, karapayneWMDE, Invadibot, maantietaja, wkandek, JMeybohm, ItamarWMDE, Akuckartz, Nandana, Namenlos314, jijiki, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T346456: Improve concurrency limits configuration of the wdqs updater
EBernhardson added a project: Discovery-Search. TASK DETAIL https://phabricator.wikimedia.org/T346456 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Aklapper, bking, Clement_Goubert, dcausse, Danny_Benjafield_WMDE, Kappakayala, Astuthiodit_1, AWesterinen, Arnoldokoth, karapayneWMDE, Invadibot, maantietaja, wkandek, JMeybohm, ItamarWMDE, Akuckartz, Nandana, Namenlos314, jijiki, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T344284: Rename usages of whitelist to allowlist in query service rdf repo
EBernhardson moved this task from Needs review to To Be Deployed on the Discovery-Search (Current work) board. EBernhardson added a comment. This should be ready for deployment now. The rdf package will need to be built and then deployed with the config updates above iiuc. TASK DETAIL https://phabricator.wikimedia.org/T344284 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, EBernhardson Cc: EBernhardson, Aklapper, bking, Reedy, Gehel, RKemper, Danny_Benjafield_WMDE, Isabelladantes1983, Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, BTullis, Biggs657, karapayneWMDE, Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T342416: Set data permission on new snapshot generation (discovery.wikibase_rdf)
EBernhardson added a comment. New dataset for 20230821 has updated permissions as expected. TASK DETAIL https://phabricator.wikimedia.org/T342416 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: dcausse, BTullis, AndrewTavis_WMDE, Aklapper, JAllemandou, Danny_Benjafield_WMDE, Mohamed-Awnallah, Astuthiodit_1, AWesterinen, lbowmaker, karapayneWMDE, Invadibot, Ywats0ns, maantietaja, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T342416: Set data permission on new snapshot generation (discovery.wikibase_rdf)
EBernhardson added a comment. In T342416#9101474 <https://phabricator.wikimedia.org/T342416#9101474>, @JAllemandou wrote: > In T342416#9091146 <https://phabricator.wikimedia.org/T342416#9091146>, @EBernhardson wrote: > >> Similarly we have other jobs that still run today and emit world readable dumps without explicitly setting the umask, what is causing the difference? >> >> drwxrwxr-x /wmf/data/discovery/cirrus/index/cirrus_replica=codfw/cirrus_group=chi/wiki=enwiki/snapshot=20230716 >> drwxrwxr-x /wmf/data/discovery/cirrus/index/cirrus_replica=codfw/cirrus_group=chi/wiki=enwiki/snapshot=20230723 >> drwxrwxr-x /wmf/data/discovery/cirrus/index/cirrus_replica=codfw/cirrus_group=chi/wiki=enwiki/snapshot=20230730 >> drwxrwxr-x /wmf/data/discovery/cirrus/index/cirrus_replica=codfw/cirrus_group=chi/wiki=enwiki/snapshot=20230806 > > The guess I have about those would be that they are still generated by a Hive job. Hive and spark behave differently in regard to permissions when generating files. Spark uses the configured umask, while hive reproduces the parent-dir patten. I'd be interested to be sure if my guess is correct :) These are both generated by spark. The rdf is being imported by a scala application while the cirrus dump is imported by pyspark, but they should both be using the same underlying implementation. Both applications use `df.write.insertInto(table_name)` to instruct spark to do the actual output. I'm a bit surprised they end up generating different sets of permissions. I suppose it's not super important why the cirrus dump is world readable, it's fine to be readable, it just hints to me that there is something I don't understand about hdfs/spark/permissions happening here. TASK DETAIL https://phabricator.wikimedia.org/T342416 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: dcausse, BTullis, AndrewTavis_WMDE, Aklapper, JAllemandou, Danny_Benjafield_WMDE, Mohamed-Awnallah, Astuthiodit_1, AWesterinen, lbowmaker, karapayneWMDE, Invadibot, Ywats0ns, maantietaja, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T342416: Set data permission on new snapshot generation (discovery.wikibase_rdf)
EBernhardson moved this task from Needs review to To Be Deployed on the Discovery-Search (Current work) board. EBernhardson added a comment. Airflow instance has been updated. I manually changed the permissions of the existing files to 644 and dirs to 755 in `/wmf/data/discovery/wikidata/rdf` so the existing datasets all match the datasets that will be created in the future. Additionally there were three directories for imports from feb 2021 that don't look to have automatically cleaned up, i verified they were not registered as a current hive partition to `discovery.wikibase_rdf` and deleted them. Leaving this in the `To Be Deployed` state to verify the next produced dump has the file permissions we expect before closing. TASK DETAIL https://phabricator.wikimedia.org/T342416 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: dcausse, BTullis, AndrewTavis_WMDE, Aklapper, JAllemandou, Danny_Benjafield_WMDE, Mohamed-Awnallah, Astuthiodit_1, AWesterinen, lbowmaker, karapayneWMDE, Invadibot, Ywats0ns, maantietaja, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T342416: Set data permission on new snapshot generation (discovery.wikibase_rdf)
EBernhardson added a comment. It seems the CodeReviewBot doesn't update the ticket when changing the ticket in a patch on gitlab, the relevant patch is: ebernhardson opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/478 Make wikibase ttl imports world readable TASK DETAIL https://phabricator.wikimedia.org/T342416 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: dcausse, BTullis, AndrewTavis_WMDE, Aklapper, JAllemandou, Danny_Benjafield_WMDE, Mohamed-Awnallah, Astuthiodit_1, AWesterinen, lbowmaker, karapayneWMDE, Invadibot, Ywats0ns, maantietaja, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T342416: Set data permission on new snapshot generation (discovery.wikibase_rdf)
EBernhardson added a comment. I looked into these, the attached patch should fix it but it leaves an open question (@JAllemandou): The `core-site.xml`, along with puppet which writes it out, has the default umask of 027 since at least 2021, which prevents world readability. So why do we have the following permissions for historical dumps: drwxr-xr-x /wmf/data/discovery/wikidata/rdf/date=20230710 drwxr-xr-x /wmf/data/discovery/wikidata/rdf/date=20230716 drwxr-xr-x /wmf/data/discovery/wikidata/rdf/date=20230717 drwxr-x--- /wmf/data/discovery/wikidata/rdf/date=20230723 drwxr-x--- /wmf/data/discovery/wikidata/rdf/date=20230724 drwxr-x--- /wmf/data/discovery/wikidata/rdf/date=20230730 drwxr-x--- /wmf/data/discovery/wikidata/rdf/date=20230731 drwxr-x--- /wmf/data/discovery/wikidata/rdf/date=20230806 Similarly we have other jobs that still run today and emit world readable dumps without explicitly setting the umask, what is causing the difference? drwxrwxr-x /wmf/data/discovery/cirrus/index/cirrus_replica=codfw/cirrus_group=chi/wiki=enwiki/snapshot=20230716 drwxrwxr-x /wmf/data/discovery/cirrus/index/cirrus_replica=codfw/cirrus_group=chi/wiki=enwiki/snapshot=20230723 drwxrwxr-x /wmf/data/discovery/cirrus/index/cirrus_replica=codfw/cirrus_group=chi/wiki=enwiki/snapshot=20230730 drwxrwxr-x /wmf/data/discovery/cirrus/index/cirrus_replica=codfw/cirrus_group=chi/wiki=enwiki/snapshot=20230806 TASK DETAIL https://phabricator.wikimedia.org/T342416 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: dcausse, BTullis, AndrewTavis_WMDE, Aklapper, JAllemandou, Danny_Benjafield_WMDE, Mohamed-Awnallah, Astuthiodit_1, AWesterinen, lbowmaker, karapayneWMDE, Invadibot, Ywats0ns, maantietaja, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T342416: Set data permission on new snapshot generation (discovery.wikibase_rdf)
EBernhardson claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T342416 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: dcausse, BTullis, AndrewTavis_WMDE, Aklapper, JAllemandou, Danny_Benjafield_WMDE, Mohamed-Awnallah, Astuthiodit_1, AWesterinen, lbowmaker, karapayneWMDE, Invadibot, Ywats0ns, maantietaja, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T339347: qlever dblp endpoint for wikidata federated query nomination
EBernhardson added a comment. In T339347#9078729 <https://phabricator.wikimedia.org/T339347#9078729>, @bking wrote: > @WolfgangFahl We've whitelisted the endpoints, but the query you linked above <https://w.wiki/6q2i> still does not work. Can you verify that is it working as expected? My teammate mentioned "it's returning application/sparql-results+xml but we only know how to process application/sparql-results+json, application/qlever-results+json." So maybe if we use a different Accept header? Let us know if we can assist. I had this slightly backwards, after looking closer i think what is happening is: - Blazegraph is submitting (afaict) `Accept: application/sparql-results+xml` to qlever as part of the federated query - qlever is responding that it doesn't know how to respond in that format. - Blazegraph knows how to handle `application/sparql-results+json` for normal api responses, but I'm not sure if it can read that format or how to tell it to use that here TASK DETAIL https://phabricator.wikimedia.org/T339347 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, RKemper, bking, Aklapper, WolfgangFahl, Danny_Benjafield_WMDE, Astuthiodit_1, AWesterinen, BTullis, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T334470: Federated queries to Lingua Libre time out in the Commons query service
EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board. EBernhardson added a comment. These queries look to be running as expected now. TASK DETAIL https://phabricator.wikimedia.org/T334470 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Nikki, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Y.ssk, Muchiri124, CBogen, ItamarWMDE, Akuckartz, Eihel, Nandana, Namenlos314, Poslovitch, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Pamputt, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Ixocactus, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, Base, aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Steinsplitter, Mbch331, Ltrlg ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T335873: Special:Search broken on Beta Wikidata for entity namespaces
EBernhardson moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. EBernhardson added a comment. reindex complete, looks to have resolved the issue as expected. TASK DETAIL https://phabricator.wikimedia.org/T335873 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, RhinosF1, Michael, Aklapper, Lucas_Werkmeister_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, ItamarWMDE, Akuckartz, CptViraj, DannyS712, Nandana, Lahi, Gq86, Bsandipan, GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, TheresNoTime, Scott_WUaS, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T335873: Special:Search broken on Beta Wikidata for entity namespaces
EBernhardson claimed this task. EBernhardson moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board. EBernhardson added a comment. > Search backend error during entity_full_text search for 'test' after 35: Parse error on Cannot search on field [labels.en] since it is not indexed. looks like a reindex that was done in production didn't happen in the beta cluster. Will start a full-cluster reindex there. TASK DETAIL https://phabricator.wikimedia.org/T335873 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, RhinosF1, Michael, Aklapper, Lucas_Werkmeister_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, ItamarWMDE, Akuckartz, CptViraj, DannyS712, Nandana, Lahi, Gq86, Bsandipan, GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, TheresNoTime, Scott_WUaS, Wikidata-bugs, aude, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T334470: Federated queries to Lingua Libre time out in the Commons query service
EBernhardson claimed this task. EBernhardson moved this task from Ready for Dev -- SRE/Ops to Needs review on the Discovery-Search (Current work) board. TASK DETAIL https://phabricator.wikimedia.org/T334470 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Nikki, Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Y.ssk, Juan90264, Muchiri124, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Eihel, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Poslovitch, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Pamputt, Taiwania_Justo, Neuronton, Scott_WUaS, Jonas, Xmlizer, Ixocactus, Wong128hk, jkroll, Wikidata-bugs, Jdouglas, Base, aude, Tobias1984, El_Grafo, Dinoguy1000, Manybubbles, Steinsplitter, Mbch331, Ltrlg ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T334823: Add https://opendata.aragon.es/sparql to the list of federated endpoints for WDQS and WCQS
EBernhardson claimed this task. EBernhardson moved this task from Ready for Dev -- SRE/Ops to Needs review on the Discovery-Search (Current work) board. TASK DETAIL https://phabricator.wikimedia.org/T334823 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: dcausse, Aklapper, Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T332314: Configure new WDQS servers in codfw (wdqs20[13-22])
EBernhardson set the point value for this task to "5". EBernhardson moved this task from Incoming to Ready for Dev -- SRE/Ops on the Discovery-Search (Current work) board. TASK DETAIL https://phabricator.wikimedia.org/T332314 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Gehel, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T332953: Migrate PipelineLib repos to GitLab
EBernhardson moved this task from needs triage to Current work on the Discovery-Search board. EBernhardson edited projects, added Discovery-Search (Current work); removed Discovery-Search. TASK DETAIL https://phabricator.wikimedia.org/T332953 WORKBOARD https://phabricator.wikimedia.org/project/board/1849/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Eevans, Seddon, MSantos, kevinbazira, odimitrijevic, BTullis, Ottomata, calbon, fgiunchedi, WMDE-leszek, leila, fkaelin, ItamarWMDE, elukey, KartikMistry, santhosh, Martaannaj, sbassett, bking, bd808, Ladsgroup, Krinkle, Legoktm, tstarling, Physikerwelt, dcausse, Jdrewniak, taavi, hnowlan, Michaelcochez, cjming, Jdforrester-WMF, dduvall, Aklapper, thcipriani, Bellucii32, Themindcoder, Stevemunene, Adamm71, Jersione, Itsmeduncan, Hellket777, Cleo_Lemoisson, Brielikethecheese, LisafBia6531, JArguello-WMF, Astuthiodit_1, Atieno, 786, EChetty, TheReadOnly, Biggs657, karapayneWMDE, toberto, joanna_borun, Simonmaignan, Invadibot, DAbad, MPhamWMF, Devnull, maantietaja, Juan90264, Muchiri124, Confetti68, Anerka, Alter-paule, Beast1978, CBogen, Un1tY, Nintendofan885, Akuckartz, Otr500, Hook696, WDoranWMF, Ddurigon, MJL, Kent7301, brennen, Mateo1977, EvanProdromou, joker88john, Legado_Shulgin, ReaperDawn, CucyNoiD, Nandana, NebulousIris, Namenlos314, aezell, skpuneethumar, Gaboe420, Zylc, Giuliamocci, Davinaclare77, Abdeaitali, Cpaulf30, 1978Gage2001, Techguru.pc, Lahi, Operator873, Gq86, Af420, Xinbenlv, Vacio, Sharvaniharan, Bsandipan, scblr, Xover, GoranSMilovanovic, SPoore, TBolliger, Chicocvenancio, Hfbn0, QZanden, EBjune, Tbscho, Taquo, LawExplorer, catalandres, Eginhard, Lewizho99, Zppix, JJMC89, Maathavan, TerraCodes, DDJJ, _jensen, rosalieper, Agabi10, PEarleyWMF, Neuronton, RuyP, Liudvikas, Scott_WUaS, Pchelolo, Karthik_sripal, Izno, Wong128hk, Luke081515, Bsadowski1, Niharika, Wikidata-bugs, Jitrixis, aude, Bawolff, Dbrant, Dinoguy1000, Gryllida, Lydia_Pintscher, faidon, Grunny, ssastry, scfc, Alchimista, Arlolra, csteipp, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T328497: Remove unnecessary targets definitions
EBernhardson removed a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T328497 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: KSiebert, WMDE-Fisch, StudiesWorld, Jdforrester-WMF, Aklapper, Krinkle, Catrope, Legoktm, TrevorParscal, ori, Ricordisamoa, Krenair, gerritbot, Florian, brion, Nikerabbit, Tgr, pmiazga, Ciencia_Al_Poder, Tacsipacsi, JohanahoJ, Ltrlg, AntiCompositeNumber, Lens0021, kostajh, Universal_Omega, Michael, alistair3149, Jdlrobson, Mohamed-Awnallah, KLawal-WMF, PMenon-WMF, gonzalez.actor, PWaigi-WMF, Wangombe, Astuthiodit_1, vyuen, Gethan, STH, Sgs, fenpedia, lbowmaker, MaryMunyoki, VPuffetMichel, BTullis, karapayneWMDE, toberto, Simonmaignan, Invadibot, LaMagiaaa, DesignerThan, Func, Zabe, Ywats0ns, H0bby, Asartea, Dentonius, diegodlh, Bebiezaza, HNordeenWMF, Timbaaa, maantietaja, Parlautan, calbon, Wilmanbeno, GhostInTheMachine, Zblace, Pietrasagh, Rost_WMDE, Anerka, CBogen, ItamarWMDE, Nintendofan885, Akuckartz, Soda, Ironie, Demian, apaskulin, Dzaky17, CptViraj, Bouzinac, Erdinc_Ciftci_WMDE, darthmon_wmde, Eihel, Jtneill, abi_, taavi, MJL, Chambersjay, FriedrickMILBarbarossa, Jd3main, Dinadineke, DannyS712, wildly_boy, Nandana, Chief_Mike, Klaas_Z4us_V, Matlin, Tumz24, Urfiner, Jony, lucamauri, Patriccck, CycloneIsaac, tabish.shaikh91, Lahi, Gq86, Xinbenlv, Vacio, Ramsey-WMF, SapphieWillie, dmaza, Daimona, Xover, Lucas_Werkmeister_WMDE, Gboyers, GoranSMilovanovic, Fz-29, TheDragonFire, Chicocvenancio, JakeTheDeveloper, Mahir256, QZanden, cmadeo, Pppery, Viveksr96, Esc3300, merbst, LawExplorer, spatton, RIT_RAJARSHI, Flycatchr, Vali.matei, Samuele2002, Lemondoge, Wugapodes, elukey, Assassas77, Iniquity, YonaB, _jensen, Jseddon, rosalieper, Jason_Quinn, Agabi10, Bodhisattwa, Mkdw, XanonymusX, Taiwania_Justo, shinjiman, gabriel-wmde, Scott_WUaS, mb, Cirdan, Samwilson, DStrine, Shangkuanlc, Volker_E, XenoRyet, Izno, SBisson, Wong128hk, Luke081515, freephile, Unapersona, IKhitron, abian, MusikAnimal, Zache, Hsarrazin, Wikidata-bugs, Snowolf, Base, aude, SPQRobin, AndyRussG, Ebe123, Pcoombe, Dinoguy1000, Amire80, jeblad, jayvdb, Mvolz, RandomDSdevel, Kipod, Shizhao, fbstj, Yurik, Paladox, Arrbee, santhosh, KartikMistry, Isarra, Alchimista, Billinghurst, TheDJ, Ladsgroup, Jackmcbarn, Mbch331, jayantanth, Jay8g, ashley, jeremyb, MPhamWMF, EBjune ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T321170: Wikidata query service does not allow mwapi queries to incubator.wikimedia.org
EBernhardson moved this task from Needs review to Needs Reporting on the
Discovery-Search (Current work) board.
EBernhardson added a comment.
Example query seems to work:
SELECT * WHERE {
SERVICE wikibase:mwapi {
bd:serviceParam wikibase:endpoint "incubator.wikimedia.org";
wikibase:api "Search";
mwapi:srsearch "cheese".
?title wikibase:apiOutput mwapi:title.
}
} LIMIT 20
TASK DETAIL
https://phabricator.wikimedia.org/T321170
WORKBOARD
https://phabricator.wikimedia.org/project/board/1227/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: EBernhardson
Cc: Nikki, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot,
MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314,
Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst,
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll,
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T321170: Wikidata query service does not allow mwapi queries to incubator.wikimedia.org
EBernhardson claimed this task. EBernhardson moved this task from Ready for Dev -- SWE to Needs review on the Discovery-Search (Current work) board. TASK DETAIL https://phabricator.wikimedia.org/T321170 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Nikki, Aklapper, Adamm71, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T317682: Make new Vector search navigate to search result URL when selecting search result using keyboard
EBernhardson added a comment. Poking over the history and the related tests. There are tests in `tests/browser/SearchSatisfactionTests.php` that expect to log a -1 as the position when the user submits their own query and not something provided by the autocomplete. This seems to have been provided as `data.index` to the autocomplete track function. The specific referenced comment looks to be outdated, from the git history that looks to have been added in the first patch that implemented autocomplete handling which was further extended but not TASK DETAIL https://phabricator.wikimedia.org/T317682 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: mpopov, cchen, EBernhardson, ItamarWMDE, dcausse, Gehel, Jdlrobson, Catrope, AnneT, jhsoby, Aklapper, Michael, Lucas_Werkmeister_WMDE, phuedx, hnijhuis, Jersione, Hellket777, NHillard-WMF, LisafBia6531, Astuthiodit_1, STH, 786, Biggs657, Patafisik_WMF, karapayneWMDE, Invadibot, MPhamWMF, Selby, Universal_Omega, maantietaja, Juan90264, Alter-paule, NavinRizwi, Beast1978, CBogen, Un1tY, Akuckartz, Demian, Hook696, Kent7301, joker88john, DannyS712, CucyNoiD, Nandana, Gaboe420, Amorymeltzer, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Xover, GoranSMilovanovic, QZanden, EBjune, LawExplorer, Lewizho99, JJMC89, Maathavan, Iniquity, _jensen, rosalieper, Agabi10, Neuronton, Scott_WUaS, Volker_E, Wikidata-bugs, aude, Dinoguy1000, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps
EBernhardson updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, EBernhardson Cc: bking, EBernhardson, HenkvD, Aklapper, dcausse, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T319136: Allow federated queries with the Eu Knowledge Graph
EBernhardson moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board. EBernhardson added a comment. This has been deployed. If anything isn't working right please ping us here. TASK DETAIL https://phabricator.wikimedia.org/T319136 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: DD063520, Aklapper, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T319136: Allow federated queries with the Eu Knowledge Graph
EBernhardson added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T319136 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: DD063520, Aklapper, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T317681: Make new Vector search navigate to item search results on Wikidata
EBernhardson added a comment. I'm not sure why search results go back into the search engine to be redirected instead of going directly to the page. We return the full link in action=opensearch which is used in other contexts (browser go-bar, etc.). It has simply "always" been that way, at least for the last decade, and never revisited. I wouldn't be surprised if it was done that way as a simplifying factor long ago, or perhaps based on an assumption that search autocomplete might some day complete search queries in addition to page titles. I don't see any particular reason the queries need to route back through the search engine instead of following the provided link directly. TASK DETAIL https://phabricator.wikimedia.org/T317681 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, phuedx, AnneT, Jdlrobson, Michael, Aklapper, jhsoby, Lucas_Werkmeister_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, NavinRizwi, ItamarWMDE, Akuckartz, Dinadineke, DannyS712, Nandana, Amorymeltzer, tabish.shaikh91, Lahi, Gq86, GoranSMilovanovic, Jayprakash12345, JakeTheDeveloper, QZanden, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Dinoguy1000, TheDJ, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps
EBernhardson added a comment. To move this forward one of our SRE's will need to run the following and let it go for a couple days. After that the sre.wdqs.data-transfer cookbook will need to be used. cookbook sre.wdqs.data-reload wcqs2001.codw.wmnet \ --task-id T316236 \ --reason 'reloading data' \ --reuse-downloaded-dump \ --depool \ --reload-data=commons \ --kafka-timestamp=166285440 TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, HenkvD, Aklapper, dcausse, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps
EBernhardson added a comment. The reload that was started on wcqs2001 didn't quite go right. We need to drop the reload scripts from the rdf deploy repo and only use the cookbooks going forward. TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, HenkvD, Aklapper, dcausse, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T317530: MediaInfo does seem to allow entities to share same statement IDs
EBernhardson added a comment. The consumer has been updated to work, but the underlying RDF's should be fixed. Relaxing the consumer means we've disabled sanity checks and in the long term the database will take on inconsistencies. TASK DETAIL https://phabricator.wikimedia.org/T317530 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, WMDE-leszek, bking, Aklapper, dcausse, Astuthiodit_1, AWesterinen, karapayneWMDE, toberto, Invadibot, GFontenelle_WMF, MPhamWMF, maantietaja, Y.ssk, FRomeo_WMF, Muchiri124, CBogen, ItamarWMDE, Nintendofan885, Akuckartz, Nandana, JKSTNK, Namenlos314, Lahi, Gq86, E1presidente, Ramsey-WMF, Cparle, SandraF_WMF, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, Tramullas, Acer, merbst, LawExplorer, Salgo60, Silverfish, _jensen, rosalieper, Taiwania_Justo, Scott_WUaS, Jonas, Xmlizer, Susannaanas, Ixocactus, Wong128hk, Fuzheado, Jane023, jkroll, Wikidata-bugs, Jdouglas, Base, matthiasmullie, aude, Tobias1984, Daniel_Mietchen, El_Grafo, Dinoguy1000, Manybubbles, Ricordisamoa, Wesalius, Lydia_Pintscher, Raymond, Steinsplitter, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps
EBernhardson moved this task from Ready for Development to In Progress on the Discovery-Search (Current work) board. EBernhardson claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T316236 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, HenkvD, Aklapper, dcausse, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps
EBernhardson added a comment. Also stopped wcqs-updater.service on wcqs2001, and disabled puppet so it wont be restarted TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, HenkvD, Aklapper, dcausse, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps
EBernhardson added a comment. Started download/munge on wcqs2001 using the internal dumps.wikimedia.org, we can't use dumps.wikimedia.your.org as it's dumps are two weeks out of date. The dumps are dated 20220911 TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, HenkvD, Aklapper, dcausse, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T316236: Reload WCQS from dumps
EBernhardson added a comment. Started looking into this, first problem is that dumps.wikimedia.your.org has changed their path layouts, a minor change to the data reload script will be necessary to pull from the correct paths and not 404. As long as we are revisiting this script though, it seems worthwhile to reconsider T222349 <https://phabricator.wikimedia.org/T222349>. It looks like we should be able to NFS mount the appropriate data to specific instances and run the data reloads fully within our own network. TASK DETAIL https://phabricator.wikimedia.org/T316236 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, HenkvD, Aklapper, dcausse, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T303831: Productionize Wikidata subgraph analysis
EBernhardson added a comment. data cleanup looks to now have run successfully TASK DETAIL https://phabricator.wikimedia.org/T303831 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, dcausse, Gehel, JAllemandou, Aklapper, AKhatun_WMF, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T307596: User documentation for authentication on WCQS
EBernhardson added a comment. Proposed documentation: P34534 <https://phabricator.wikimedia.org/P34534> I'm intending to update the wiki page after WCQS deployment and re-verifying the updates work as expected. TASK DETAIL https://phabricator.wikimedia.org/T307596 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, GFontenelle_WMF, Dominicbm, Zbyszko, Aklapper, Gehel, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T307596: User documentation for authentication on WCQS
EBernhardson moved this task from Ready for Development to In Progress on the Discovery-Search (Current work) board. EBernhardson claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T307596 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, GFontenelle_WMF, Dominicbm, Zbyszko, Aklapper, Gehel, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors
EBernhardson added a comment. Some quick testing makes this look successful. Using curl to perform a POST no longer 500's: curl 'https://commons-query.wikimedia.org/sparql' \ -XPOST \ -H 'cookie: wcqsOauth=; wcqsSession=' \ -d 'query=prefix%20schema:%20%3Chttp://schema.org/%3E%20SELECT%20*%20WHERE%20%7B%3Chttp://www.wikidata.org%3E%20schema:dateModified%20?y%7D=27701073' Additionally the underlying issue, that the JWT would expire also looks resolved. Tested by opening the UI in a browser tab along with the network inspector and leaving it for many hours. The UI performs a regular request every 10 minutes to ask about update lag, every couple hours those requests return a 307 response that includes a new JWT and the requests continue to work. Looks to be working as expected. If i leave a browser window along with the network inspector open for a few hours can see it getting a 307 every couple hours with a refreshed JWT. Additionally manually POST'ing a request TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: HenkvD, MPhamWMF, DAbad, RKemper, EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, Dominicbm, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T307596: User documentation for authentication on WCQS
EBernhardson added a comment. I still can't see it worthwhile to document the existing workflow. It's so convoluted that I suspect anyone that's willing to follow it would simply monitor the connections in their web browsers development inspector and recreate what they see without any explicit documentation required. Instead in T306899 <https://phabricator.wikimedia.org/T306899> i've reworked the re-authentication flow to use a second cookie that will allow the documentation to be written in a sane manner. TASK DETAIL https://phabricator.wikimedia.org/T307596 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, GFontenelle_WMF, Dominicbm, Zbyszko, Aklapper, Gehel, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T303831: Productionize Wikidata subgraph analysis
EBernhardson added a comment. @JAllemandou The one remaining piece of this ticket is cleaning up the historical data, per T303831#8081172 <https://phabricator.wikimedia.org/T303831#8081172>. Any suggestions on how we should manage droping old data from tables partitioned by a snapshot column? TASK DETAIL https://phabricator.wikimedia.org/T303831 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, dcausse, Gehel, JAllemandou, Aklapper, AKhatun_WMF, Hellket777, LisafBia6531, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors
EBernhardson added a comment. I've tracked down one source of 500 errors, unclear if the original report here is for same thing. Reproduction: curl -XPOST https://commons-query.wikimedia.org/any-url-doesnt-matter -d 'foo=bar' Reason: This request includes a `Content-Length` header which nginx ends up passing along to the /oauth/check_auth endpoint. Jetty (hosting the endpoint) sees the Content-Length header and starts waiting for the content to arrive, which never does. After 30s jetty times out the request. This most likely means all request's with the query in the content, rather than a url query string, receive this 500 error. Resolution: Whitelist the set of headers that will be passed along to the /oauth/* endpoints to only include the Host and Cookies headers. Caveats: While this will fix the timeout, i suspect it will simply fail the request at a different part of the request. At least in my reproduction case the reason the UI is issuing a POST request with the query in the body is that the GET request was rejected due to attempting to re-auth during an XHR and the browser refused to show the response to the javascript. The UI javascript interprets this as the request having never been sent and re-issues the same request over POST. Once this timeout issue is fixed that POST request will have the same CORS problems and it's unlikely we will be able to change mediawiki's Special:OAuth CORS headers for this use case. Possible Solutions: Gergo suggested perhaps we can store an oauth1 related token in the cookies. When the JWT expires after 2 hours and requires a re-auth we might be able to re-validate the previously stored oauth1 token, rather than going through the full redirect-bounce which has CORS issues. Will require more investigation and review of oauth 1 flows to determine if this is viable. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: MPhamWMF, DAbad, RKemper, EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, Dominicbm, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors
EBernhardson added a comment. Leaving the commons-query.wikimedia.org browser tab open for a few hours and re-running queries every 30-60 minutes or so reproduced a 500 after a few hours. Related js console errors. Timestamps are PDT. Unclear if the errors at 13:00 and 13:10 are directly related, but including since they were there) 13:00:42.779 /#%23Depictions%20of%20Douglas%20Adams%0A%23shows%20M-entities%20that%20depict%20Douglas%20Adams%0ASELECT%20%3Ffile%20WHERE%20%7B%0A%20%20%3Ffile%20wdt%3AP180%20wd%3AQ42%20.%0A%7D:1 Access to XMLHttpRequest at 'https://commons.wikimedia.org/wiki/Special:OAuth/authenticate?oauth_token=' (redirected from 'https://commons-query.wikimedia.org/sparql?query=prefix%20schema:%20%3Chttp://schema.org/%3E%20SELECT%20*%20WHERE%20%7B%3Chttp://www.wikidata.org%3E%20schema:dateModified%20?y%7D=27659280') from origin 'https://commons-query.wikimedia.org' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: Redirect is not allowed for a preflight request. 13:00:42.782 commons.wikimedia.org/wiki/Special:OAuth/authenticate?oauth_token=:1 Failed to load resource: net::ERR_FAILED 13:10:42.786 /#%23Depictions%20of%20Douglas%20Adams%0A%23shows%20M-entities%20that%20depict%20Douglas%20Adams%0ASELECT%20%3Ffile%20WHERE%20%7B%0A%20%20%3Ffile%20wdt%3AP180%20wd%3AQ42%20.%0A%7D:1 Access to XMLHttpRequest at 'https://commons.wikimedia.org/wiki/Special:OAuth/authenticate?oauth_token=' (redirected from 'https://commons-query.wikimedia.org/sparql?query=prefix%20schema:%20%3Chttp://schema.org/%3E%20SELECT%20*%20WHERE%20%7B%3Chttp://www.wikidata.org%3E%20schema:dateModified%20?y%7D=27659290') from origin 'https://commons-query.wikimedia.org' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: Redirect is not allowed for a preflight request. 13:10:42.787 commons.wikimedia.org/wiki/Special:OAuth/authenticate?oauth_token=:1 Failed to load resource: net::ERR_FAILED 13:12:36.726 /#%23Depictions%20of%20Douglas%20Adams%0A%23shows%20M-entities%20that%20depict%20Douglas%20Adams%0ASELECT%20%3Ffile%20WHERE%20%7B%0A%20%20%3Ffile%20wdt%3AP180%20wd%3AQ42%20.%0A%7D:1 Access to XMLHttpRequest at 'https://commons.wikimedia.org/wiki/Special:OAuth/authenticate?oauth_token=' (redirected from 'https://commons-query.wikimedia.org/sparql?query=%23Depictions%20of%20Douglas%20Adams%0A%23shows%20M-entities%20that%20depict%20Douglas%20Adams%0ASELECT%20%3Ffile%20WHERE%20%7B%0A%20%20%3Ffile%20wdt%3AP180%20wd%3AQ42%20.%0A%7D') from origin 'https://commons-query.wikimedia.org' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: Redirect is not allowed for a preflight request. 13:12:36.749 commons.wikimedia.org/wiki/Special:OAuth/authenticate?oauth_token=:1 Failed to load resource: net::ERR_FAILED 13:13:06.992 /sparql:1 Failed to load resource: the server responded with a status of 500 () Correlated errors from server logs (13:00 PDT == 20:00 UTC): Aug 3, 2022 @ 20:13:06.938 wcqs1002WARNING /oauth/check_auth java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 3/3 ms TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: MPhamWMF, DAbad, RKemper, EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, Dominicbm, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors
EBernhardson added a comment. In T306899#8128904 <https://phabricator.wikimedia.org/T306899#8128904>, @Dominicbm wrote: > Experienced the same error today again, here is an exact timestamp (of the response): `Wed, 03 Aug 2022 17:15:19 GMT`. This lines up nicely with a message from logging: Aug 3, 2022 @ 17:15:19.203 wcqs1002WARNING /oauth/check_auth java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 3/3 ms TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: MPhamWMF, DAbad, RKemper, EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, Dominicbm, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T307391: Enable CORS support for WCQS SPARQL endpoint access
EBernhardson added a comment. https://commons-query.wikimedia.org/sparql returns CORS headers in the same way that https://query.wikidata.org/sparql does. What doesn't work is CORS during the authentication flow, and I'm not sure this is something we can change. I can setup the appropriate CORS headers to be returned by the query service when redirecting to auth, but that will redirect to https://commons.wikimedia.org/wiki/Special:OAuth/authenticate?oauth_token=... which will then say: Access to XMLHttpRequest at 'https://commons.wikimedia.org/wiki/Special:OAuth/authenticate?oauth_token=...' (redirected from 'https://commons-query.wikimedia.org/sparql') from origin 'https://test.wikipedia.org' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. Changing the CORS headers for Special:OAuth isn't something I can do, that would have to go through the security team. It's hard for me to verify that would be sufficient, testing with a hacked up chrome extension that lets me overwrite request/response headers I can potentially make it work in cases where the user already has a commons-query.wikimedia.org auth token, although right now i'm fighting with nginx to convince it to apply SameSite=none to cookies instead of reubiilding the application jars. TASK DETAIL https://phabricator.wikimedia.org/T307391 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, FRomeo_WMF, GFontenelle_WMF, Aklapper, Dominicbm, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T303831: Productionize Wikidata subgraph analysis
EBernhardson removed a project: Patch-For-Review. EBernhardson added a comment. Double checked all linked patches, no patches remain for review. The work still to be done is to decide how to handle pruning data from the `snapshot=` partitioned tables TASK DETAIL https://phabricator.wikimedia.org/T303831 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AKhatun_WMF, EBernhardson Cc: EBernhardson, dcausse, Gehel, JAllemandou, Aklapper, AKhatun_WMF, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Hellket777, 786, Biggs657, Juan90264, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, Maathavan, Neuronton ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T301336: EntitySchemas API Question
EBernhardson removed a project: ApiFeatureUsage. EBernhardson added a comment. Removing ApiFeatureUsage, that project is specifically about recording information about requests made to api.php in mediawiki TASK DETAIL https://phabricator.wikimedia.org/T301336 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, Lydia_Pintscher, Lucas_Werkmeister_WMDE, Aklapper, Mistermboy, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, SCIdude, Akuckartz, pdehaye, Nandana, Lahi, Gq86, Andrawaag, GoranSMilovanovic, QZanden, YULdigitalpreservation, LawExplorer, Salgo60, _jensen, rosalieper, Scott_WUaS, MisterSynergy, abian, Wikidata-bugs, aude, Mbch331, Amorymeltzer ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T304070: API Endpoint to search for Schemas
EBernhardson removed a project: ApiFeatureUsage. EBernhardson added a comment. Removing ApiFeatureUsage, that project is specifically about usage of api.php in mediawiki TASK DETAIL https://phabricator.wikimedia.org/T304070 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, Lucas_Werkmeister_WMDE, EduardoUT, Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, SCIdude, Akuckartz, pdehaye, Nandana, Lahi, Gq86, Andrawaag, GoranSMilovanovic, QZanden, YULdigitalpreservation, LawExplorer, Salgo60, _jensen, rosalieper, Scott_WUaS, MisterSynergy, abian, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331, Amorymeltzer ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T303831: Productionize Wikidata subgraph analysis
EBernhardson added a comment. There is actually one piece remaining, we typically use `refinery-drop-older-than` to prune our tables. That worked when we used `date=...` as the partitioning scheme, but it doesn't support `snapshot=...`. I t takes minimal work (I already have a working POC) to make it interpret `snapshot` the same as `date`, but I suspect the partitioning changed the name to `snapshot=...` due to an intent to not only use dates for partitioning? If so analytics does have a `refinery-drop-mediawiki-snapshots` script but it's fairly specialized to their use case. I suspect we would need to make a work-alike script that uses the same refinery library methods but provides our own configuration to the script. Or the script could be modified to import it's configuration from somewhere user-defined instead of having the configuration embedded in the script itself. Lots of options, but we have to figure out which is the appropriate way forward. TASK DETAIL https://phabricator.wikimedia.org/T303831 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AKhatun_WMF, EBernhardson Cc: EBernhardson, dcausse, Gehel, JAllemandou, Aklapper, AKhatun_WMF, Hellket777, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T303831: Productionize Wikidata subgraph analysis
EBernhardson added a comment. All dags are now enabled and have completed at least one full execution of each dag. - Increased partition count on map_subgraph_queries to 2048, the largest shuffle is ~600GB and this gets the per-executor work down into the desired 256-512M range. - Increased executor memory on map_subgraph_queries from 8g to 12g. Many executors were red with >10% of time spent in GC. This often leads to intermittent failures that increase when data sizes increase, 12g appears to keep most executors out of the red state. - Seeing intermittent failures in map_subgraph_queries, usually internal spark retries manage to work through it but have seen failures that roll up to the airflow retry level. We might want to increase the timeout waiting on shufle server if it persists. Potentially spark addressed this issue in 3.0 with https://issues.apache.org/jira/browse/SPARK-24355 - Mentioned to analytics team that we have a few new high-resource jobs running. These jobs are all in the `sequential` pool so it shouldn't cause any downstream issues, but seems appropriate to let them know. - Switched SubgraphQueryMapper from coalesce to repartition. Same reasoning as in the weekly dag, the final jobs were giving OOM's and allowing those to compute with the full partition count allows it to complete, at the expense of requiring an additional shuffle. - Removed `wiki=wikidata` from the sparql event partition specification in subgraph_and_query_metrics. There is no wiki column in this table, rather it is limited to wdqs (TODO: is that true? Can wcqs end up in here?) which is implicitly limited to wikidata. TASK DETAIL https://phabricator.wikimedia.org/T303831 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AKhatun_WMF, EBernhardson Cc: EBernhardson, dcausse, Gehel, JAllemandou, Aklapper, AKhatun_WMF, Hellket777, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T303831: Productionize Wikidata subgraph analysis
EBernhardson added a comment. Summary of what was done so far to deploy: - Tuned subgraph_mapping_weekly. Set spark parallelism to 4096, Increased memory to 24G (=6g per task) and reduced total executor count to keep total memory usage around 1TB. Changed `coalesce()` into `repartition()` in SubgraphMapper. Completes without any failed tasks. Might be a bit wasteful of memory, but probably not worth tuning unless there are complaints and we can hope a later upgrade to spark 3 w/ skew-join optimization will improve things. We could manually implement the same skew-join optimization on a per-use case basis, but it's extra work that might not be necessary. - Enabled subgraph_metrics_weekly. Ran without issue. - This patch added a number of new sensors. We've been intending to switch sensors from `mode=poke` to `mode=reschedule`. Adding these new sensors reminded me of why we needed to make that change (all airflow executors used waiting for data to arrive). Deployed a patch to switch everything over. - Enabled subgraph_query_mapping_daily. This started waiting for snapshot=20220613 (last monday) with an execution_date of 20220620 (also a monday). I suspect we should adjust this to target snapshot=20220620, but waiting for confirmation. Turned back off so it doesn't timeout and complain. - Enabled subgraph_query_metrics_daily. This is waiting for `event.wdqs_external_sparql_query/datacenter=eqiad/year=2022/month=6/day=20` (and same for codfw) but it needs to be waiting on the individual hourly partitions. I hadn't thought this fully through when reviewing the patch, we will need to adjust the sensor to use HivePartitionRangeSensor which can generate all the intermediate hourly named partitions. Turned back off as it's also waiting for outputs of subgraph_query_mapping_daily (iiuc) which is turned off currently. TASK DETAIL https://phabricator.wikimedia.org/T303831 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AKhatun_WMF, EBernhardson Cc: EBernhardson, dcausse, Gehel, JAllemandou, Aklapper, AKhatun_WMF, Hellket777, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T303831: Productionize Wikidata subgraph analysis
EBernhardson added a comment. Stats on the final join building `topSubgraphTriples`. this is using 4096 partitions and repartition(). It works for now so probably not worth dealing with the skew, but these stats might be useful to compare against in the future if it starts failing: | Metric | Min | 25th percentile | Median | 75th percentile | Max | | Duration | 15 s | 46 s| 54 s | 1.0 min | 9.2 min | | Scheduler Delay | 2 ms | 3 ms| 3 ms | 4 ms| 0.4 s| | Task Deserialization Time| 1 ms | 2 ms| 2 ms | 3 ms| 0.7 s| | GC Time | 27 ms| 0.1 s | 0.2 s | 0.3 s | 41 s | | Result Serialization Time| 0 ms | 0 ms| 0 ms | 0 ms| 1 ms | | Getting Result Time | 0 ms | 0 ms| 0 ms | 0 ms| 0 ms | | Peak Execution Memory| 2.1 GB | 2.1 GB | 2.1 GB | 2.1 GB | 13.6 GB | | Shuffle Read Blocked Time| 0 ms | 23 s| 32 s | 38 s| 2.1 min | | Shuffle Read Size / Records | 263.2 MB / 3156075 | 269.9 MB / 3235843| 271.6 MB / 3256300 | 273.4 MB / 324| 30.5 GB / 414401248 | | Shuffle Remote Reads | 255.2 MB | 264.1 MB| 266.1 MB | 268.0 MB| 29.7 GB | | Shuffle Write Size / Records | 340.9 MB / 3184514 | 351.8 MB / 3281889| 354.4 MB / 3305742 | 357.0 MB / 3330833| 367.5 MB / 3438583 | | Shuffle spill (memory) | 0.0 B| 0.0 B | 0.0 B | 0.0 B | 98.1 GB | | Shuffle spill (disk) | 0.0 B| 0.0 B | 0.0 B | 0.0 B | 28.2 GB | | TASK DETAIL https://phabricator.wikimedia.org/T303831 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AKhatun_WMF, EBernhardson Cc: EBernhardson, dcausse, Gehel, JAllemandou, Aklapper, AKhatun_WMF, Hellket777, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T303831: Productionize Wikidata subgraph analysis
EBernhardson added a comment.
I tried a run with the three coalesce's in SubgraphMapper converted into
repartitions. In this case instead of having 8 partitions where 7 finish and
the 8th takes forever and then fails, now it has 200 partitions and 199 finish
with the 200th taking forever and then failing. This seems like it could be a
case of skew-join, the dataset is being partitioned based on the join condition
(rather than randomly) and a specific part of the join has significantly more
values to work through than anything else. To get an idea of how significant
the skew is i doubled the ram again (to 24g) in hopes that it will eventually
complete and give some stats. The final stats are as follows, clearly showing a
significant skew:
| Duration | 1 s | 1 s | 2 s
| 2 s | 4.1 min |
| Scheduler Delay | 6 ms | 19 ms| 21 ms
| 26 ms | 34 ms|
| Task Deserialization Time| 37 ms | 61 ms| 77 ms
| 0.1 s | 0.2 s|
| GC Time | 0 ms | 16 ms| 23 ms
| 48 ms | 2.6 min |
| Result Serialization Time| 0 ms | 0 ms | 0 ms
| 0 ms| 1 ms |
| Getting Result Time | 0 ms | 0 ms | 0 ms
| 0 ms| 0 ms |
| Peak Execution Memory| 128.8 MB | 194.3 MB | 196.3
MB | 200.3 MB| 5.6 GB |
| Shuffle Read Blocked Time| 0 ms | 3 ms | 5 ms
| 64 ms | 0.3 s|
| Shuffle Read Size / Records | 1469.5 KB / 35062 | 2.5 MB / 87982 | 3.1
MB / 133528 | 5.0 MB / 258108 | 406.2 MB / 38467392 |
| Shuffle Remote Reads | 1433.7 KB | 2.5 MB | 3.1
MB | 4.9 MB | 398.5 MB |
| Shuffle Write Size / Records | 0.0 B / 0 | 184.5 KB / 18106 | 827.2
KB / 72252 | 2.5 MB / 195511 | 404.2 MB / 38411863 |
|
Resolving skew on the other hand is a harder problem. Spark 3 added a new
skew-join optimization and I've heard that some other teams have spark 3
working in our cluster, but I haven't played around with it at all yet. Will
look into this more and see what solutions can be found. In terms of the exact
code causing this, spark is terrible at telling us exactly where but trying to
infer from the SparkUI output i think it's this join:
def getTopSubgraphItems(topSubgraphs: DataFrame): DataFrame = {
wikidataTriples
.filter(s"predicate='<$p31>'")
.selectExpr("object as subgraph", "subject as item")
.join(topSubgraphs.select("subgraph"), Seq("subgraph"), "right")
I'll probably need to recreate some of this in a jupyterlab notebook to look
at the actual data and see what exactly is in the skewed side of the dataset.
TASK DETAIL
https://phabricator.wikimedia.org/T303831
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: AKhatun_WMF, EBernhardson
Cc: EBernhardson, dcausse, Gehel, JAllemandou, Aklapper, AKhatun_WMF,
Hellket777, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE,
Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen,
ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD,
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420,
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst,
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS,
Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles,
Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T303831: Productionize Wikidata subgraph analysis
EBernhardson added a comment. In T303831#8060472 <https://phabricator.wikimedia.org/T303831#8060472>, @AKhatun_WMF wrote: > In T303831#8058159 <https://phabricator.wikimedia.org/T303831#8058159>, @EBernhardson wrote: > >> the airflow patch is deployed but i only turned on *_init dags and subgraph_mapping_weekly today (ran out of time, will do rest tomorrow). >> >> subgraph_mapping_weekly failed the first time through. I updated executor memory from 8g to 12g but the second execution is still failing. something is quite unbalanced about the topSubgraphItems, of the 8 shards they have inputs varying from 100MB to 450MB giving executions times of ~30s on the small ones and ~8m before the final one fails. >> >> Not specifically related to this patch, but i wonder if we could change up the `SparkUtils.saveTables` method to somehow take parameters in the path to specify coalesce vs repartition and the number of partitions to save by, so we only have to update the airflow invocation and not the jar as well to test variations there. > > Should we have params called `coalesce`, and `repartition`, and have them default to false. And when true, use `num_partitions` to coalesce or repartition accordingly? > > Edit: I realize all arg classes that need to coalesce or repartition will need to have these params set. In this case i was thinking that we could somehow treat the string that is provided over the command line as a specification for how/where to store things and somehow include named parameters in it. So for example right now we provide: --all-subgraphs-table discovery.wikibase_rdf/date=20220620/wiki=wikidata What if instead we could provide (syntax to be bikeshedded): --all-subgraphs-table discovery.wikibase_rdf/date=20220620/wiki=wikidata;repartition=42 This would have the downside that read/write would have different syntaxes and we have to know which to use where, maybe there are better options. Mostly pondering ideas on how to make things we know might have to be modified easier to change. There are probably other ways to magic parameters into various places in the jvm world, this is just a first guess. TASK DETAIL https://phabricator.wikimedia.org/T303831 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AKhatun_WMF, EBernhardson Cc: EBernhardson, dcausse, Gehel, JAllemandou, Aklapper, AKhatun_WMF, Hellket777, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T303831: Productionize Wikidata subgraph analysis
EBernhardson added a comment. the airflow patch is deployed but i only turned on *_init dags and subgraph_mapping_weekly today (ran out of time, will do rest tomorrow). subgraph_mapping_weekly failed the first time through. I updated executor memory from 8g to 12g but the second execution is still failing. something is quite unbalanced about the topSubgraphItems, of the 8 shards they have inputs varying from 100MB to 450MB giving executions times of ~30s on the small ones and ~8m before the final one fails. Not specifically related to this patch, but i wonder if we could change up the `SparkUtils.saveTables` method to somehow take parameters in the path to specify coalesce vs repartition and the number of partitions to save by, so we only have to update the airflow invocation and not the jar as well to test variations there. TASK DETAIL https://phabricator.wikimedia.org/T303831 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: AKhatun_WMF, EBernhardson Cc: EBernhardson, dcausse, Gehel, JAllemandou, Aklapper, AKhatun_WMF, Hellket777, Astuthiodit_1, AWesterinen, 786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T308741: Lexeme search results all have the current timestamp as last changed date
EBernhardson moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board. EBernhardson added a comment. Link in report now correctly shows last edit timestamps. TASK DETAIL https://phabricator.wikimedia.org/T308741 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Bugreporter, Michael, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Wilmanbeno, CBogen, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, Mahir256, QZanden, EBjune, LawExplorer, _jensen, rosalieper, Bodhisattwa, Scott_WUaS, Wikidata-bugs, aude, jayvdb, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors
EBernhardson added a comment. Lacking better ideas on how to align the errors with some request that causes the error I've started up `tcpdump` on all the wcqs instances. They will store up to 100 1GB files per instance before starting to overwrite the initial files. The overall goal here is to match requests from the tcpdump pcap with unexplained error messages like 'Idle timeout expired' tcpdump -ni lo -W 100 -C 1gb -w /srv/T306899/lo.pcap TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: RKemper, EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, Dominicbm, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors
EBernhardson added a comment. Reviewed logs again looking for patterns. Not much, but at least logstash is now aggregating together logs from the various hosts. Can see that the `/oauth/check_auth java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 3/3 ms` errors come in infrequently, but often bunched up a bit. From the last week, on May 22 it came in 5 times starting 11:45 until 15:02. May 26th three times from 14:22 to 14:23, twice on may 27 at 8:07, once on the 28th at 14:20 and once on the 31st at 10:58. Still no strong proof that these are timeouts are the 500's some users are seeing. Additionally still no success in reproducing errors, I run multiple example queries daily for a few weeks now but they always work as expected. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: RKemper, EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, Dominicbm, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T308741: Lexeme search results all have the current timestamp as last changed date
EBernhardson claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T308741 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Bugreporter, Michael, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Wilmanbeno, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Mahir256, QZanden, EBjune, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Bodhisattwa, Neuronton, Scott_WUaS, Wikidata-bugs, aude, jayvdb, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T308741: Lexeme search results all have the current timestamp as last changed date
EBernhardson edited projects, added Discovery-Search (Current work); removed Discovery-Search. TASK DETAIL https://phabricator.wikimedia.org/T308741 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Bugreporter, Michael, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Wilmanbeno, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, Mahir256, QZanden, EBjune, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Bodhisattwa, Neuronton, Scott_WUaS, Wikidata-bugs, aude, jayvdb, Mbch331, jeremyb ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T308786: Track errors in the UI of commons-query.wikimedia.org
EBernhardson created this task.
EBernhardson added a project: Wikidata Query UI.
Restricted Application added a subscriber: Aklapper.
TASK DESCRIPTION
The UI used for the wiki commons query service currently collects no metrics,
even though the UI has metric tracking built in.
This looks to be due to the following function which throws out any attempt
to track on commons query:
SELF.prototype.track = function( metricName, value, valueType ) {
if ( !value ) {
value = 1;
}
if ( !valueType ) {
valueType = 'c';
}
if (
location.hostname !== 'query.wikidata.org' ||
/^1|yes/.test( navigator.doNotTrack || window.doNotTrack )
) {
// skip tracking
return $.when();
}
// https://www.wikidata.org/beacon/statsv?test.statsv.foo2=5c
return this._track( metricName + '=' + value + valueType );
};
AC: Metrics are collected from the UI for commons-query.wikimedia.org
TASK DETAIL
https://phabricator.wikimedia.org/T308786
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: EBernhardson
Cc: MPhamWMF, EBernhardson, Aklapper, AWesterinen, CBogen, Namenlos314, Gq86,
Lucas_Werkmeister_WMDE, Mahir256, EBjune, merbst, Salgo60, Jonas, Xmlizer,
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Lydia_Pintscher
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors
EBernhardson claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, Dominicbm, Fernandobacasegua34, Astuthiodit_1, AWesterinen, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306644: re-run wbsearchentities optimization process
EBernhardson moved this task from Waiting to Needs review on the Discovery-Search (Current work) board. EBernhardson added a comment. Reports found in https://people.wikimedia.org/~ebernhardson/T306644/ Summary is that the tuning is either the same or slightly worse almost everywhere. Unclear currently where things went wrong. It's not significantly worse so the process is still coming up with reasonable values, but those reasonable values aren't resulting in better ranking than the tuning from a few years ago. TASK DETAIL https://phabricator.wikimedia.org/T306644 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Aklapper, Smalyshev, dcausse, Liuxinyu970226, EJoseph, EBernhardson, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T209859: Wikidata autocomplete (wbsearchentities) results with score <= 0
EBernhardson added a comment. In T209859#7903772 <https://phabricator.wikimedia.org/T209859#7903772>, @Lucas_Werkmeister_WMDE wrote: > In T209859#7881777 <https://phabricator.wikimedia.org/T209859#7881777>, @gerritbot wrote: > >> Change 786267 **merged** by jenkins-bot: >> >> [mediawiki/extensions/CirrusSearch@es68] Prevent negative weights on BoostedQueriesFunction >> >> https://gerrit.wikimedia.org/r/786267 > > Do you think there’s any chance that this change (which ended up in wmf.10) caused T307586: wbsearchentities produces no results on 1.39.0-wmf.10 <https://phabricator.wikimedia.org/T307586>? > > (Edit: I quoted the wrong version of the change – the commit on master, rECIRd5cf710f34ee: Prevent negative weights on BoostedQueriesFunction <https://phabricator.wikimedia.org/rECIRd5cf710f34ee99251dfe9306a02d225a68fea24b>, is the one that ended up in wmf.10. I think.) Nope, this would have been caused by c9c499fe19ec14e939f755e50b9f1c66805c79f4 <https://phabricator.wikimedia.org/rECIRc9c499fe19ec14e939f755e50b9f1c66805c79f4>, or more generally by the in progress upgrade to elasticsearch 7.10. TASK DETAIL https://phabricator.wikimedia.org/T209859 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EJoseph, EBernhardson Cc: Lucas_Werkmeister_WMDE, EJoseph, Liuxinyu970226, dcausse, Smalyshev, EBernhardson, Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors
EBernhardson added a comment. Following the thread of something related to auth, I've found that the application server (jetty) which hosts the app has never properly had it's logging setup. Logs only come from the embedded applications, the application server itself ends up with bare minimum logging into the host local journald where it's mostly forgotten about. Currently working out how jetty should be configured for logging to work as expected. This likely means there are wdqs errors that go unnoticed as well. Hoping that with proper logging in place we start to get more details about whatever is causing these 500's. The default logging is quite minimal, looking through the logs turns up a few unexplained errors that could be related. Not clear any of these are the symptoms of the same problem, but lacking more information best bet seems to be to look into these. 0-3 per day. These don't seem to be new, logs go back to mar 23, and this shows up on mar 24. Frequency is quite low. May 10 15:34:21 wcqs1001 wcqs-blazegraph[29631]: 2022-05-10 15:34:21.036:WARN:oejs.HttpChannel:qtp968514068-231008: /oauth/check_auth java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 3/3 ms Occured on multiple days over the last month, but not with any regularity. The value inside quotes is sometimes an html error page, sometimes the included value. suggests error messages are being interpreted as a valid response (but then not validating and failing later): May 05 05:47:54 wcqs1002 wcqs-blazegraph[24508]: javax.servlet.ServletException: javax.servlet.ServletException: com.github.scribejava.core.exceptions.OAuthException: Response body is incorrect. Can't extract token and secret from this: 'upstream connect error or disconnect/reset before headers. reset reason: overflow' TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, Dominicbm, Astuthiodit_1, AWesterinen, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors
EBernhardson added a comment. Not finding anything decisive yet, but will continue looking. It occured to me that if it's happening consistently for an individual user but not in general that it could somehow be related to their authentication cookie. If seems plausible clearing the auth cookie could fix things, if the problem is related to auth. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, Dominicbm, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors
EBernhardson added a comment.
Usually the first stop for this kind of error would be reviewing the `ATS
Backends <-> Origin Servers Overview` which suggest a low rate of 5xxs,
typically 1-5% of requests fail. In a quick review of the last few 500 requests
on one of the servers they were all malformed queries. We may need to look into
more specific timespans rather than the generic 500 errors. Modifying one of
the dashboard queries[1] to return success rate per 15 minutes and running it
against thanos to get all DC's, looking for time periods of low success, the
following time periods should be reviewed:
2022-04-16T17:30-18:10
2022-04-17T08:30-10:00
2022-04-22T16:26-17:12
2022-04-22T19:09-19:36
2022-04-26T16:20-17:37
2022-05-04T19:50-21:42
If this turns up the problem we could consider how it could be turned into an
alert.
[1]
sum(increase(trafficserver_backend_requests_seconds_count{status=~"2[0-9][0-9]",
cluster=~"cache_text", backend=~"wcqs\\.discovery\\.wmnet"}[15m])) by (backend)
/
sum(increase(trafficserver_backend_requests_seconds_count{status=~"[25][0-9][0-9]",
cluster=~"cache_text", backend=~"wcqs\\.discovery\\.wmnet"}[15m])) by (backend)
TASK DETAIL
https://phabricator.wikimedia.org/T306899
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: EBernhardson
Cc: EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper,
Dominicbm, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja,
CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86,
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst,
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll,
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T307586: wbsearchentities produces no results on 1.39.0-wmf.10
EBernhardson added a comment. Patch should resolve the issue. In terms of testing I would estimate that only integration testing would reliably catch this type of problem. We have some of that in CirrusSearch itself but nothing I'm aware of for the specialized wikidata extension. TASK DETAIL https://phabricator.wikimedia.org/T307586 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, Zabe, dcausse, brennen, hashar, jcrespo, Raymond, Moebeus, Lucas_Werkmeister_WMDE, Aklapper, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, R4356th, Bebiezaza, EhsanKhandowa, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, darthmon_wmde, Rosalie_WMDE, PatsagornY, Kent7301, joker88john, Viztor, CucyNoiD, Nandana, Gaboe420, Amorymeltzer, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, QZanden, EBjune, LawExplorer, Lewizho99, JJMC89, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Johan, Luke081515, Verdy_p, Wikidata-bugs, aude, TheDJ, Jdforrester-WMF, Addshore, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T307586: wbsearchentities produces no results on 1.39.0-wmf.10
EBernhardson added a comment. There is a variety of churn in Cirrus right now related to a version upgrade which likely caused this. Will look what is causing the breakage today. TASK DETAIL https://phabricator.wikimedia.org/T307586 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, Zabe, dcausse, brennen, hashar, jcrespo, Raymond, Moebeus, Lucas_Werkmeister_WMDE, Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, R4356th, Bebiezaza, EhsanKhandowa, maantietaja, CBogen, ItamarWMDE, Akuckartz, darthmon_wmde, Rosalie_WMDE, PatsagornY, Viztor, Nandana, Amorymeltzer, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, LawExplorer, JJMC89, _jensen, rosalieper, Scott_WUaS, Johan, Luke081515, Verdy_p, Wikidata-bugs, aude, TheDJ, Jdforrester-WMF, Addshore, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T305952: Update WDQS update lag SLO grafana page to new 95% SLO
EBernhardson moved this task from Ready for Development to Needs Reporting on the Discovery-Search (Current work) board. EBernhardson added a comment. Updated graph on wdqs-wcqs-lag-slo dashboard to use 95 instead of 99 for the threshold value. TASK DETAIL https://phabricator.wikimedia.org/T305952 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, MPhamWMF, Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306644: re-run wbsearchentities optimization process
EBernhardson added a comment. Ran the previous AB testing report to get a preliminary look at the data and ensure it's collecting as expected. Everything seems reasonable, the new tuning isn't clearly better but not clearly worse either and we only have a few hundred events. As stated previously intending to run for two weeks, ending data collection on May 11. TASK DETAIL https://phabricator.wikimedia.org/T306644 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Aklapper, Smalyshev, dcausse, Liuxinyu970226, EJoseph, EBernhardson, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306644: re-run wbsearchentities optimization process
EBernhardson added a comment. Profiles are deployed, they can be enabled for testing in a single page with a magic query string like wikidataCompletionSearchClicksBucket=T306644-fr <https://www.wikidata.org/wiki/Q2?wikidataCompletionSearchClicksBucket=T306644-fr>. Next steps would be to turn the test on, and set the turn-off date. Previously we did two weeks, I don't remember what went into that decision but running this for two weeks seems plausible as well. Should we inform anyone at wikidata that we will be turning on the test? Who? TASK DETAIL https://phabricator.wikimedia.org/T306644 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Aklapper, Smalyshev, dcausse, Liuxinyu970226, EJoseph, EBernhardson, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306644: re-run wbsearchentities optimization process
EBernhardson claimed this task. EBernhardson added a comment. Few ideas for future exploration: - Lots of the weights in the tuning report claim to have minimal influence on the final output, look into why. Do we need to collect more negative samples in the training set? Are the features useless? - Could be interesting to generate the sensitivity portion of the report against current production deployed values. - The improvement levels are surprisingly similar to before, perhaps suspisously so. Would also be interesting to re-run the optimization process after deploying the new values. If training with the optimized values as the comparison we should see little if any improvement. If it still shows significant improvements there could be errors in the reporting. TASK DETAIL https://phabricator.wikimedia.org/T306644 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Aklapper, Smalyshev, dcausse, Liuxinyu970226, EJoseph, EBernhardson, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306644: re-run wbsearchentities optimization process
EBernhardson added a comment. Reports generated and published: https://people.wikimedia.org/~ebernhardson/wbsearchentities_202203 TASK DETAIL https://phabricator.wikimedia.org/T306644 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EJoseph, EBernhardson Cc: Aklapper, Smalyshev, dcausse, Liuxinyu970226, EJoseph, EBernhardson, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306054: Upgrade deployment-wdqs01 host to Buster
EBernhardson moved this task from Incoming to In Progress on the Discovery-Search (Current work) board. EBernhardson set the point value for this task to "1". TASK DETAIL https://phabricator.wikimedia.org/T306054 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, EBernhardson Cc: dcausse, Lucas_Werkmeister_WMDE, Mathew.onipe, Aklapper, Majavah, Peachey88, Jdforrester-WMF, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, CptViraj, DannyS712, Nandana, Namenlos314, Lahi, Gq86, Bsandipan, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Addshore, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T305952: Update WDQS update lag SLO grafana page to new 95% SLO
EBernhardson moved this task from Incoming to Ready for Development on the Discovery-Search (Current work) board. EBernhardson set the point value for this task to "1". TASK DETAIL https://phabricator.wikimedia.org/T305952 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: MPhamWMF, Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306156: New upstream release for jvmquake
EBernhardson closed this task as "Resolved". EBernhardson claimed this task. EBernhardson added a comment. This is the already deployed version, pinged on first run of libup-bot for jvmquake TASK DETAIL https://phabricator.wikimedia.org/T306156 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, LibUp-bot, Aklapper, MPhamWMF, CBogen, Namenlos314, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T306644: re-run wbsearchentities optimization process
EBernhardson created this task. EBernhardson added projects: Wikidata, Discovery-Search (Current work). TASK DESCRIPTION To support elasticsearch 7 the scoring equation for wbsearchentities needs some small shape changes. The weights we use in this search came from relforge_wbsearchentities. The process was last used on elasticserach 5.5, likely some changes will be necessary to get it up and running against 6.8. These reports can be run against the current equation and not the updated one, the goal of having tuning reports is to know that the full process is working and runnable again. AC: Tuning reports, including weights to deploy to prod, for all languages that have custom weights already deployed TASK DETAIL https://phabricator.wikimedia.org/T306644 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EJoseph, EBernhardson Cc: Aklapper, Smalyshev, dcausse, Liuxinyu970226, EJoseph, EBernhardson, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T304437: Allow federated queries with cellar endpoint of the Publication Office and European Commission
EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board. EBernhardson added a comment. This should now be enabled TASK DETAIL https://phabricator.wikimedia.org/T304437 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: MPhamWMF, DD063520, Aklapper, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, CBogen, ItamarWMDE, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T304437: Allow federated queries with cellar endpoint of the Publication Office and European Commission
EBernhardson claimed this task. EBernhardson moved this task from Ready for Development to Needs review on the Discovery-Search (Current work) board. TASK DETAIL https://phabricator.wikimedia.org/T304437 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: MPhamWMF, DD063520, Aklapper, Fernandobacasegua34, Astuthiodit_1, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T301650: WCQS "Application Connection Error" E009
EBernhardson added a comment. I'm not convinced the patch here will fix anything, but the symptom reported has to do with re-using an old cached response. This is a simple enough change and semantically correct regardless of if it fixes this issue so will deploy it sometime this week. TASK DETAIL https://phabricator.wikimedia.org/T301650 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: MPhamWMF, EBernhardson, Zbyszko, Aklapper, Dominicbm, Fernandobacasegua34, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, maantietaja, FRomeo_WMF, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Nintendofan885, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, JKSTNK, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T280487: Redirect requests from wcqs-beta.wmflabs.org to the final URL for WCQS
EBernhardson added a subtask: T303202: Redirect wcqs-beta.wmflabs.org to commons-query.wikimedia.org. TASK DETAIL https://phabricator.wikimedia.org/T280487 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, WikiLucas00, Gehel, Aklapper, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T299062: Save stats from wcqs-beta
EBernhardson moved this task from Waiting to Needs review on the Discovery-Search (Current work) board. EBernhardson added a comment. With wcqs-beta 1 shut down and redirected to beta 2 i suspect this is complete? Moving to needs review if someone knows what steps are still necessary. TASK DETAIL https://phabricator.wikimedia.org/T299062 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko, EBernhardson Cc: EBernhardson, Aklapper, Gehel, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T301650: WCQS "Application Connection Error" E009
EBernhardson added a comment. After reviewing mdn's CORS docs and stack overflow posts about redirect based auth combined with xmlhttprequest, I'm not finding a simple way to do this that avoids changing the application. I suspect we will need some sort of hook or support within the javascript application for this use case. In particular one way forward is: - Adjust the backend to return errors to XMLHttpRequest instead of doing the redirect bounce. The standard way would be returning 401 Not Authorized. Some online solutions always return a 2xx and embed this into the json, but i would prefer to avoid changing the responses as much as possible. - Adjust the frontend to recognize the failed auth and refresh the page. As long as the auth is non-interactive (mediawiki doesn't ask them to login) it should preserve the users previous query. If mediawiki does ask them to login the query (stored in the url fragment) will likely be lost. - This might be doable through `jQuery.ajaxSetup` by having it perform a pre-check but that would introduce additional round-trip latency. - Integrating more directly with the code that handles the response in the UI would allow for more direct handling, but would need to involve WMDE approving, or possibly even writing, the changes TASK DETAIL https://phabricator.wikimedia.org/T301650 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, Zbyszko, Aklapper, Dominicbm, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, FRomeo_WMF, CBogen, Nintendofan885, Akuckartz, Nandana, JKSTNK, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T301650: WCQS "Application Connection Error" E009
EBernhardson added a comment. While trying a few different things I found one way to cause this to fail, although it's going the opposite way of this ticket, so not certain it's related. In particular 1. Open commons-query and run an example query 2. Open browser settings and delete the wcqsSession cookie 3. Attempt to execute a query This fails with a CORS error, particularly: Access to XMLHttpRequest at 'https://commons.wikimedia.org/wiki/Special:OAuth/authenticate?oauth_token=redacted' (redirected from 'https://commons-query.wikimedia.org/sparql?query=prefix%20schema:%20%3Chttp://schema.org/%3E%20SELECT%20*%20WHERE%20%7B%3Chttp://www.wikidata.org%3E%20schema:dateModified%20?y%7D=27434725') from origin 'https://commons-query.wikimedia.org' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: Redirect is not allowed for a preflight request. I'm not sure what the appropriate action is here, it might be the intent that this isn't supposed to be able to authenticate in the background, or it might be an unintended limitation. While this ticket is likely about the contained token expiring, rather than the cookie expiring, i suspect the result will be similar with respect to it attempting to re-auth in the background. TASK DETAIL https://phabricator.wikimedia.org/T301650 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, Zbyszko, Aklapper, Dominicbm, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, FRomeo_WMF, CBogen, Nintendofan885, Akuckartz, Nandana, JKSTNK, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T293462: Add user blocking in WCQS
EBernhardson added a comment. I manually applied the fixes in the latest patch, to pass cookies on to blazegraph, and my username came through into the request logs. Hoping this will be resovled once the above is merged. TASK DETAIL https://phabricator.wikimedia.org/T293462 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Aklapper, Zbyszko, Fernandobacasegua34, 786, Suran38, Biggs657, karapayneWMDE, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T299222: Properly configure logback for W[CD]QS streaming updater
EBernhardson removed a project: Patch-For-Review. EBernhardson added a comment. doesn't look like there are any more patches here, removing patch-for-review TASK DETAIL https://phabricator.wikimedia.org/T299222 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Gehel, Aklapper, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, Fernandobacasegua34, 786, Suran38, Biggs657, Lalamarie69, Juan90264, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, Maathavan, Neuronton ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T282117: WCQS needs to be exposed through a wikimedia.org domain
EBernhardson removed a project: Patch-For-Review. EBernhardson moved this task from Waiting to Needs Reporting on the Discovery-Search (Current work) board. TASK DETAIL https://phabricator.wikimedia.org/T282117 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: RKemper, So9q, Aklapper, Gehel, CBogen, ttaylor, Zbyszko, Invadibot, MPhamWMF, maantietaja, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331, 786, Suran38, Biggs657, Lalamarie69, Juan90264, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, Maathavan, Neuronton ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T279541: Add a reconciliation strategy to the wdqs streaming updater
EBernhardson added a comment. Airflow DAG has been deployed. I have left it turned off for now, when ready someone will need to enable it (and potentially update the start_date). TASK DETAIL https://phabricator.wikimedia.org/T279541 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, EBernhardson Cc: EBernhardson, RShigapov, dcausse, Aklapper, 786, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T299222: Properly configure logback for W[CD]QS streaming updater
EBernhardson added a comment. Logs themselves have been flowing for a while now, since the patch merge on Jan 26. I put up one more cleanup pa tch, after that i believe this should be complete. We don't need to do a deploy for this patch, it can run with whatever the next deployment is. TASK DETAIL https://phabricator.wikimedia.org/T299222 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Gehel, Aklapper, 786, Suran38, Biggs657, Invadibot, Lalamarie69, MPhamWMF, maantietaja, Juan90264, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD, Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Neuronton, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T293862: Investigate using jvmquake to limit the time a JVM is unusable due to GC overhead
EBernhardson updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T293862 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana, Namenlos314, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
