[Wikidata-bugs] [Maniphest] T263265: wdqs-updater fails to work due to badtimestamp_rcstart

[Nikerabbit]

Nikerabbit closed this task as "Resolved".
Nikerabbit claimed this task.

TASK DETAIL
  https://phabricator.wikimedia.org/T263265

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nikerabbit
Cc: CBogen, Addshore, Aklapper, Nikerabbit, Samantha_Alipio_WMDE, Akuckartz, 
darthmon_wmde, Jelabra, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Asahiko, despens, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Nemo_bis, 
Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263265: wdqs-updater fails to work due to badtimestamp_rcstart

[CBogen]

CBogen added a comment.


  @Nikerabbit, if the issue in this ticket is resolved, can you close and open 
a new one for the issue you are experiencing now? Thanks!

TASK DETAIL
  https://phabricator.wikimedia.org/T263265

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: CBogen
Cc: CBogen, Addshore, Aklapper, Nikerabbit, Samantha_Alipio_WMDE, Akuckartz, 
darthmon_wmde, Jelabra, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Asahiko, despens, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Nemo_bis, 
Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263265: wdqs-updater fails to work due to badtimestamp_rcstart

[Nikerabbit]

Nikerabbit added a comment.


  I am not sure which of changes done for ticket T263427: Unable to process a 
particular wikibase dump using munge.sh (localised namespace name) 
 made a difference, but I no longer 
see this. I suspect it may be due to `WIKIBASE_SCHEME=https` or just having 
data in `wdqs`.
  
  Oh and this also happened with 0.3.40.
  
  Anyway, it is now failing due to the localised namespace name like in the 
above task. And I cannot use `sed` to fix this for the updater.

TASK DETAIL
  https://phabricator.wikimedia.org/T263265

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nikerabbit
Cc: Addshore, Aklapper, Nikerabbit, CBogen, Samantha_Alipio_WMDE, Akuckartz, 
darthmon_wmde, Jelabra, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Asahiko, despens, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Nemo_bis, 
Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263265: wdqs-updater fails to work due to badtimestamp_rcstart

[Addshore]

Addshore added a project: Wikidata-Query-Service.
Addshore added a comment.


  Not seen this before.
  This could already be fixed in the latest wdqs version, have you tried?

TASK DETAIL
  https://phabricator.wikimedia.org/T263265

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Addshore, Aklapper, Nikerabbit, CBogen, Samantha_Alipio_WMDE, Akuckartz, 
darthmon_wmde, Jelabra, Nandana, Namenlos314, Lahi, Gq86, 
Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, 
LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Xmlizer, Asahiko, despens, 
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Nemo_bis, 
Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T263265: wdqs-updater fails to work due to badtimestamp_rcstart

[Nikerabbit]

Nikerabbit created this task.
Nikerabbit added projects: Wikibase-Containers, User-Nikerabbit.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.

TASK DESCRIPTION
  I'm using the wdqs-updater image with the following configuration:
  
wdqs-updater:
  image: wikibase/wdqs:0.3.10
  restart: unless-stopped
  command: /runUpdate.sh
  depends_on:
  - wdqs
  networks:
default:
  aliases:
   - wdqs-updater.svc
  environment:
   - WIKIBASE_HOST=nimiarkisto.fi
   - WDQS_HOST=wdqs.svc
   - WDQS_PORT=
  
  When inspecting it's status with `docker logs` I see the following error 
message repeated many times per second:
  
14:47:18.037 [main] WARN  org.wikidata.query.rdf.tool.Updater - Retryable 
error fetching first batch.  Retrying.
org.wikidata.query.rdf.tool.exception.RetryableException: Error result from 
Mediawiki:  {code=badtimestamp_rcstart, info=Invalid value 
"2020-06-20T11%3A23%3A36.954Z" for timestamp parameter "rcstart"., *=See 
https://nimiarkisto.fi/w/api.php for API usage. Subscribe to the 
mediawiki-api-announce mailing list at 
; for 
notice of API deprecations and breaking changes.}
at 
org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.checkApi(WikibaseRepository.java:498)
at 
org.wikidata.query.rdf.tool.wikibase.WikibaseRepository.fetchRecentChanges(WikibaseRepository.java:314)
at 
org.wikidata.query.rdf.tool.change.RecentChangesPoller.doFetchRecentChanges(RecentChangesPoller.java:325)
at 
org.wikidata.query.rdf.tool.change.RecentChangesPoller.fetchRecentChanges(RecentChangesPoller.java:314)
at 
org.wikidata.query.rdf.tool.change.RecentChangesPoller.batch(RecentChangesPoller.java:338)
at 
org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:162)
at 
org.wikidata.query.rdf.tool.change.RecentChangesPoller.firstBatch(RecentChangesPoller.java:38)
at org.wikidata.query.rdf.tool.Updater.run(Updater.java:170)
at org.wikidata.query.rdf.tool.Update.run(Update.java:174)
at org.wikidata.query.rdf.tool.Update.main(Update.java:98)
  
  The timestamp is double URL encoded. Confirmed by inspecting the wikibase 
(not from docker image, but a standalone MediaWiki install) apache logs:
  
x.y.z.q - - [18/Sep/2020:17:47:42 +0300] "GET 
/w/api.php?format=json&action=query&list=recentchanges&rcdir=newer&rcprop=title%257Cids%257Ctimestamp&rcnamespace=120%257C122&rclimit=100&continue=&rcstart=2020
-06-20T11%253A23%253A36.954Z HTTP/1.1" 200 276 "-" "Wikidata Query Service 
Updater"
  
  Also, since it keeps retrying forever, this causes a minor denial of service 
attack on the target server.

TASK DETAIL
  https://phabricator.wikimedia.org/T263265

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nikerabbit
Cc: Aklapper, Nikerabbit, Samantha_Alipio_WMDE, Akuckartz, darthmon_wmde, 
Jelabra, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Asahiko, despens, Wikidata-bugs, aude, Nemo_bis, 
Addshore, Mbch331
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs