I am looking into the issue... The errors appear to have stopped. We didn't deploy new wikidata code this week so don't think it's related to deployment
My guess is it's a bot passing bad data to the api, and we are missing some check of the user input in the code thus causing this to be a fatal error. (We can probably improve handling for this in the code) I think it's ok to proceed with deployment on monday. Cheers, Katie On Jun 22, 2017 20:28, "Mukunda Modell" <[email protected]> wrote: > Correction: the train will resume on Monday since there are no deployments > on Fridays. > > On Thu, Jun 22, 2017 at 6:49 PM, Mukunda Modell <[email protected]> > wrote: > >> Starting today, immediately after SWAT, thcipriani noticed a new error in >> fatalmonitor which appeared to coincide with running scap. The curious >> thing is that scap only affected localization, no code was changed. >> >> At that point I took over for thcipriani and filed an UBN bug: >> https://phabricator.wikimedia.org/T168681 >> >> The error message: >> >> Argument 1 passed to DataValues\UnboundedQuantityValue::newFromArray() >> must be an instance of array, string given in extensions/Wikidata/vendor/dat >> a-values/serialization/src/Deserializers/DataValueDeserializer.php on >> line 141 >> >> After monitoring the errors in kibana for a while I noticed a pattern: >> the error rate is changing in a very consistent way. The graph of errors >> makes an almost perfect sine wave, peaking at about 150 per minute and then >> bottoming out at zero. This pattern has repeated all afternoon. It seems >> like such a consistent pattern could only be caused by either a) a queued >> job or b) an api client. Since the bug is reproducible via the API, I >> suspect that it's b. >> >> As of now it I believe this error to be unrelated to any recent code >> deployments. The timing might have just been coincidence. If that is the >> case then it should not block the train any further, however, Greg made the >> call to postpone further deployments until tomorrow. >> >> If you have any input on this error, please comment on the phabricator >> task. You can also ping me on IRC, I will be around for much of the evening. >> >> Thanks, that is all I've got for now. >> - Mukunda >> > > > _______________________________________________ > Ops mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/ops > > _______________________________________________ Wikitech-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikitech-l
