[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[zhuyifei1999]

zhuyifei1999 added a comment.


  (trying to reproduce it again)

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: zhuyifei1999
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, Jyoo1011, Y.ssk, JohnsonLee01, SHEKH, Dijkstra, 
Muchiri124, CBogen, Khutuck, Zkhalido, Viztor, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Krenair
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[zhuyifei1999]

zhuyifei1999 added a comment.


  Sorry, I think I was working on it last year and then forgot about this
  ticket. I'll check what I was doing back then.
  
  Is this bug still reproducible?

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: zhuyifei1999
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, Jyoo1011, Y.ssk, JohnsonLee01, SHEKH, Dijkstra, 
Muchiri124, CBogen, Khutuck, Zkhalido, Viztor, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Krenair
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  @zhuyifei1999 any word?

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: zhuyifei1999, TheSandDoctor
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, Jyoo1011, Y.ssk, JohnsonLee01, SHEKH, Dijkstra, 
Muchiri124, CBogen, Khutuck, Zkhalido, Viztor, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Krenair
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  Thanks @zhuyifei1999 ! Have you been able to work on this any?

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: zhuyifei1999, TheSandDoctor
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, JohnsonLee01, SHEKH, Dijkstra, CBogen, Khutuck, 
Zkhalido, Viztor, Wenyi, Ramsey-WMF, Chicocvenancio, Tbscho, MayS, Mdupont, 
JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, Ixocactus, 
Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, Alchimista, 
Steinsplitter, Rxy, Krenair, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[Dvorapa]

Dvorapa added a comment.


  Yeah, it seems like PoolCounter  
response

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, 
Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, Poyekhali, Altostratus, 
Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, El_Grafo, 
Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Krenair, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[Count_Count]

Count_Count added a comment.


  In T181443#5931256 , 
@TheSandDoctor wrote:
  
  > I just discovered that rcwatcher.py crashed at some point within the past 
couple of days. Interesting.
  > [...]
  > requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: 
https://stream.wikimedia.org/v2/stream/recentchange
  >
  >   
  
  This is not the fault of the sseclient library. It happens if more than a few 
(don't know the exact limit) connections are made to wikimedia event streaming 
servers.

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, Count_Count
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, 
Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, Poyekhali, Altostratus, 
Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, El_Grafo, 
Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Krenair, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  I just discovered that rcwatcher.py crashed at some point within the past 
couple of days. Interesting.
  
Traceback (most recent call last):
  File "rcwatcher.py", line 65, in 
main()
  File "rcwatcher.py", line 57, in main
run_watcher()
  File "rcwatcher.py", line 41, in run_watcher
for change in rc:
  File 
"/usr/local/lib/python3.8/site-packages/pywikibot/comms/eventstreams.py", line 
291, in __iter__
self.source = EventSource(**self.sse_kwargs)
  File "/home/thesanddoctor/sseclient/sseclient.py", line 48, in __init__
self._connect()
  File "/home/thesanddoctor/sseclient/sseclient.py", line 63, in _connect
self.resp.raise_for_status()
  File "/home/ccc/.local/lib/python3.8/site-packages/requests/models.py", 
line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: 
https://stream.wikimedia.org/v2/stream/recentchange

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, 
Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, Poyekhali, Altostratus, 
Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, El_Grafo, 
Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Krenair, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  In T181443#5927865 , 
@zhuyifei1999 wrote:
  
  > Whichever is simplest.
  
  That would be test_rc.py 
.

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, 
Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, Poyekhali, Altostratus, 
Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, El_Grafo, 
Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Krenair, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[zhuyifei1999]

zhuyifei1999 added a comment.


  Whichever is simplest.

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, zhuyifei1999
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, 
Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, Poyekhali, Altostratus, 
Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, El_Grafo, 
Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Krenair, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  In T181443#5927759 , 
@zhuyifei1999 wrote:
  
  > I will be running the script with pdb + save all sseclient trace over the 
weekend.
  
  Running test_rc.py, right?

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, 
Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, Poyekhali, Altostratus, 
Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, El_Grafo, 
Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Krenair, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[zhuyifei1999]

zhuyifei1999 added a comment.


  I will be running the script with pdb + save all sseclient trace over the 
weekend.

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, zhuyifei1999
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, 
Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, Poyekhali, Altostratus, 
Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, El_Grafo, 
Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  @Dvorapa I saw that myself & just updated my above comment prior to seeing 
your response. It appears to have slipped through, will have to add a catch for 
that. Others still relevant though.
  
  In T181443#5927703 , 
@Dvorapa wrote:
  
  >> pywikibot.exceptions.Error: Title contains illegal char (\uFFFD 
'REPLACEMENT CHARACTER')
  >
  > This should not happen after the patch anymore?
  
  No, it shouldn't. sseclient's master just had a patch merged prior to my 
updating that should've fixed it (it was the one that @zhuyifei1999 linked).

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, 
Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, Poyekhali, Altostratus, 
Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, El_Grafo, 
Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[Dvorapa]

Dvorapa added a comment.


  > pywikibot.exceptions.Error: Title contains illegal char (\uFFFD 
'REPLACEMENT CHARACTER')
  
  This should not happen after the patch anymore?

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, 
Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, Poyekhali, Altostratus, 
Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, El_Grafo, 
Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[Dvorapa]

Dvorapa added a comment.


  I tried this file and a random file to compare and Pywikibot is correct. The 
file has no imageinfo as it is a redirect. See the file page: 
https://commons.wikimedia.org/wiki/File:Vanessa_Mai_at_Gruenspan_2019_(3).png

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, 
Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, Poyekhali, Altostratus, 
Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, El_Grafo, 
Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  Since updating to the latest master version of sseclient (post-fix merges) 
more of the workers crashed than usual (4 of the 5). 3 of the 4 crashes were 
due to the same issue and the 4th was:
  
2020-02-27 23:07:01,324 __main__: DEBUG None
2020-02-27 23:07:01,324 __main__: INFO File:Vanessa Mai at Gruenspan 
2019 (3).png
Traceback (most recent call last):
  File "rcworker.py", line 221, in 
main()
  File "rcworker.py", line 214, in main
run_worker()
  File "rcworker.py", line 110, in run_worker
revision = change.getRevision(file_page)
  File "/home/ccc/Commons-image-corruption-detector/Image.py", line 29, in 
getRevision
revision = 
file_page.get_file_history()[pywikibot.Timestamp.fromtimestampformat(self.log_timestamp)]
  File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 
2533, in get_file_history
self.site.loadimageinfo(self, history=True)
  File "/usr/local/lib/python3.8/site-packages/pywikibot/site.py", line 
3104, in loadimageinfo
raise PageRelatedError(
pywikibot.exceptions.PageRelatedError: loadimageinfo: Query on 
[[commons:File:Vanessa Mai at Gruenspan 2019 (3).png]] returned no imageinfo

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, 
Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, Poyekhali, Altostratus, 
Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, El_Grafo, 
Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[Dvorapa]

Dvorapa added a comment.


  Hopefully this helps. There were more issues with sseclient 0.0.23 and 0.0.24 
(https://github.com/wikimedia/pywikibot/commit/6cf25c0fe51991a892486a32211a434c695357b6),
 hopefully 0.0.25 will solve them all

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, 
Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, Poyekhali, Altostratus, 
Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, El_Grafo, 
Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  @zhuyifei1999 requests has been updated & the workers/feeder all restarted. I 
have re-started test_rc.py and will post back here if anything crashes. If it 
is good in a few days/week or something like that I think we could consider 
this resolved. Thanks for your help so far!

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, 
Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, Poyekhali, Altostratus, 
Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, El_Grafo, 
Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[zhuyifei1999]

zhuyifei1999 added a comment.


  Yes, what I was saying was, the first and the third and two separate 
consumers, so events on first should also be received on the third. If there 
were something fundamentally wrong with the event data, then both would crash. 
Since this is not the case, there is nothing fundamentally wrong with the event 
data and therefore the error must be in other places, such as transmission / 
decoding, which leads to the linked bug report / PR.
  
  In any case, could you see if that fixes the issue?

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, zhuyifei1999
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, 
Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, Poyekhali, Altostratus, 
Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, El_Grafo, 
Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  @zhuyifei1999 Yes, first and third are two separate customers. The first and 
second are working with the same customer. The third (code 
)
 is just a plain/direct printing of the file name direct from recent changes 
listener and trying to turn it into a file page (until it fails).

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, 
Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, Poyekhali, Altostratus, 
Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, El_Grafo, 
Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[zhuyifei1999]

zhuyifei1999 added a comment.


  In T181443#5916753 , 
@TheSandDoctor wrote:
  
  > @zhuyifei1999 the first and second traceback are from "production" worker 
instances and pop items off of the same redis queue (all fed by a single 
instance of rcwatcher.py), thus they wouldn't get the same image. So it isn't 
feasible that they would crash all at once. They basically get images 
first-come, first-serve from recent changes.
  
  The first and third are two separate consumers of rcstreams right?

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, zhuyifei1999
Cc: Count_Count, Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, 
pywikibot-bugs-list, Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, 
Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, Poyekhali, Altostratus, 
Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, El_Grafo, 
Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  @zhuyifei1999 the first and second traceback are from "production" worker 
instances and pop items off of the same redis queue (all fed by a single 
instance of rcwatcher.py), thus they wouldn't get the same image. So it isn't 
feasible that they would crash all at once.

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[zhuyifei1999]

zhuyifei1999 added a comment.


  In T181443#5916068 , 
@TheSandDoctor wrote:
  
  > The first traceback above
  > 2020-02-22 20:02:09 <- start
  > 2020-02-24 08:42:16 <- crash
  > The second traceback I posted above
  > 2020-02-22 20:02:20 <-start
  > 2020-02-23 19:44:35 <-crash
  > 2020-02-23 07:46:35 <- test_rc.py launched
  > 2020-02-24 19:52:12 <- crash
  
  Hmm so the first one crashed during the third's run? I would assume that if 
events were global then all readers would crash at once.

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, zhuyifei1999
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  2020-02-22 20:02:20

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  In T181443#5915796 , 
@Dvorapa wrote:
  
  > Also, what is the output of `echo $LC_CTYPE` and `echo $LANG`? Weird is 
this happens only for some letters. I can not reproduce it on my machine. If 
you use Pywikibot packaged with scripts, could you share the output of `python 
pwb.py version`? And what is your version of requests and sseclient?
  
  @Dvorapa `echo $LC_CTYPE` is just a blank return (null?). `echo $LANG` 
returns `en_US.UTF-8`

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[Dvorapa]

Dvorapa added a comment.


  Also, what is the output of `echo $LC_CTYPE` and `echo $LANG`? Weird is this 
happens only for some letters. I can not reproduce it on my machine. If you use 
Pywikibot packaged with scripts, could you share the output of `python pwb.py 
version`?

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  @Dvorapa python 3

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[Dvorapa]

Dvorapa added a comment.


  In T181443#5915665 , 
@TheSandDoctor wrote:
  
  > @Dvorapa all encoding is set to UTF-8.
  > @zhuyifei1999 Oops, thought I had. see here 

  
  Python 2 or 3?

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  @zhuyifei1999 Oops, thought I had. see here 


TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[zhuyifei1999]

zhuyifei1999 added a comment.


  Would you mind posting the code of the 'minimal test case' somewhere?

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, zhuyifei1999
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[Dvorapa]

Dvorapa added a comment.


  This seems like an encoding issue

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  @zhuyifei1999 For both of these worth noting that I have //not// updated the 
the latest version with the change in behaviour that this task merged.
  
  From a normal worker:
  
2020-02-24 08:42:16,590 __main__: INFO File:�রপক্ষ মন্দির থেকে ভক্তদের 
বের হওয়াjpgশ্য..
Traceback (most recent call last):
  File "rcworker.py", line 221, in 
main()
  File "rcworker.py", line 214, in main
run_worker()
  File "rcworker.py", line 63, in run_worker
file_page = pywikibot.FilePage(site, change.title)
  File 
"/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", line 
1744, in wrapper
return obj(*__args, **__kw)
  File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 
2478, in __init__
super(FilePage, self).__init__(source, title, 6)
  File 
"/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", line 
1744, in wrapper
return obj(*__args, **__kw)
File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 2478, 
in __init__
super(FilePage, self).__init__(source, title, 6)
  File 
"/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", line 
1744, in wrapper
return obj(*__args, **__kw)
  File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 
2327, in __init__
super(Page, self).__init__(source, title, ns)
  File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 
200, in __init__
self._link = Link(title, source=source, default_namespace=ns)
  File 
"/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", line 
1744, in wrapper
return obj(*__args, **__kw)
  File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 
6029, in __init__
raise pywikibot.Error(
pywikibot.exceptions.Error: Title contains illegal char (\uFFFD 
'REPLACEMENT CHARACTER')
  
  From minimal reproducible test case:
  
2020-02-24 19:52:12,881 __main__: DEBUG File:Мавзолей Д�тэ Мицумуне в 
храме Энцуин.jpg
Traceback (most recent call last):
  File "test_rc.py", line 68, in 
main()
  File "test_rc.py", line 60, in main
run_watcher()
  File "test_rc.py", line 51, in run_watcher
file_page = pywikibot.FilePage(site, change['title'])
  File 
"/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", line 
1744, in wrapper
return obj(*__args, **__kw)
  File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 
2478, in __init__
super(FilePage, self).__init__(source, title, 6)
  File 
"/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", line 
1744, in wrapper
return obj(*__args, **__kw)
  File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 
2327, in __init__
super(Page, self).__init__(source, title, ns)
  File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 
200, in __init__
self._link = Link(title, source=source, default_namespace=ns)
  File 
"/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", line 
1744, in wrapper
return obj(*__args, **__kw)
  File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 
6029, in __init__
raise pywikibot.Error(
pywikibot.exceptions.Error: Title contains illegal char (\uFFFD 
'REPLACEMENT CHARACTER')

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  @zhuyifei1999 Unknown at this point. Implemented and running alongside it 
now. If either crashes will report back here.

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[zhuyifei1999]

zhuyifei1999 added a comment.


  > rcworker.py would just be reduced to a few lines
  
  Is redis queue critical to reproducing the issue? If not, that is an extra 
layer of complexity and a minimal reproducible test case does not need that.

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, zhuyifei1999
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  @zhuyifei1999 The only log currently available is as follows (and linked 
above):
  
2020-02-18 18:45:16,662 __main__: INFO File:PICT0430 - 301032 - 
onroerenderfgoed.jpg :Not corrupt. Stored
Traceback (most recent call last):
  File "rcworker.py", line 221, in 
main()
  File "rcworker.py", line 214, in main
run_worker()
  File "rcworker.py", line 61, in run_worker
file_page = pywikibot.FilePage(site, change.title)
  File 
"/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", line 
1744, in wrapper
 return obj(*__args, **__kw)
  File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 
2478, in __init__
super(FilePage, self).__init__(source, title, 6)
  File 
"/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", line 
1744, in wrapper
return obj(*__args, **__kw)
  File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 
2327, in __init__
super(Page, self).__init__(source, title, ns)
  File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 
200, in __init__
self._link = Link(title, source=source, default_namespace=ns)
File "/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", 
line 1744, in wrapper
return obj(*__args, **__kw)
  File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 
6029, in __init__
raise pywikibot.Error(
pywikibot.exceptions.Error: Title contains illegal char (\uFFFD 
'REPLACEMENT CHARACTER')
  
  I have improved the logging so that prior to attempting to generate a 
FilePage object it logs the name, instead of doing so afterwords. The current 
code setup is basically a minimal test case in the sense that rcwatcher.py 
would not change and rcworker.py would just be reduced to a few lines. The best 
way to go forward with this would be to just wait for it to crash again as that 
way it is still doing work in the meantime. I will post here when the next log 
is available.

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[zhuyifei1999]

zhuyifei1999 added a comment.


  What is the traceback?
  
  Any minimal reproducible test case (even if it takes a long time to reproduce 
that's still something)?

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, zhuyifei1999
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[Dvorapa]

Dvorapa added a comment.


  Now we need to know, why is this happening, because I don't feel like there 
are so many new files in Commons with this character

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  @Dvorapa Commons. The issue appears to happen at random. I have improved the 
ordering of my logs so next time it happens it should //hopefully// actually 
tell me the issue. Given that the files are only run from recent changes if 
they are new uploads, this isn't something easily repeatable and does appear to 
happen at random. I will update here when I have more logs. Thank you for your 
patch to make the exception catchable.

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[Dvorapa]

Dvorapa added a comment.


  I don't understand it either. Could you find out, which file makes this (or 
is it for random files?) and also tell me, which wiki do you check? (Commons?)

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  @Dvorapa grabs the file from recent changes using site_rc_listener 

 and then sends it to the other script for processing. site_rc_listener is what 
must be giving it the invalid images.

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[Dvorapa]

Dvorapa added a comment.


  I'm sorry, I don't understand, what your script does and how. There should be 
no title in commons with that character and if there are links to a page with 
that character, the freshly called InvalidTitle error should be always checked 
in scripts.
  
  So what else do you think we should fix in Pywikibot?

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  @Dvorapa Just trying to create a file page reference given a name of a file 
from recent changes. Here is the error log 
.
  
  The line that causes the problem is `file_page = pywikibot.FilePage(site, 
change.title)` of rcworker.py 


TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, TheSandDoctor
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[gerritbot]

gerritbot added a comment.


  Change 574111 **merged** by jenkins-bot:
  [pywikibot/core@master] [bugfix] Raise InvalidTitle for title containing 
illegal char
  
  https://gerrit.wikimedia.org/r/574111

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa, gerritbot
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, Beast1978, CBogen, Un1tY, Zkhalido, Hook696, Daryl-TTMG, 
RomaAmorRoma, 0010318400, E.S.A-Sheild, Meekrab2012, joker88john, Viztor, 
DannyS712, CucyNoiD, NebulousIris, Wenyi, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Af420, Darkminds3113, 
Bsandipan, Lordiis, Adik2382, Chicocvenancio, Th3d3v1ls, Ramalepe, Liugev6, 
Tbscho, MayS, WSH1906, Lewizho99, Mdupont, JJMC89, Maathavan, Poyekhali, 
Altostratus, Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, 
El_Grafo, Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[Dvorapa]

Dvorapa added a comment.


  In T181443#5909015 , 
@TheSandDoctor wrote:
  
  > Also: thanks for switching it to a catchable exception! :)
  
  Yeah, the error should be now better.
  
  > @Dvorapa But what would cause it to return it then when looking at images? 
I am sort of confused here.
  
  I can see the issue (it's in text, not in title), but don't know, what causes 
it. Could you describe in detail the steps you took to observe this issue?

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, Beast1978, CBogen, Un1tY, Zkhalido, Hook696, Daryl-TTMG, 
RomaAmorRoma, 0010318400, E.S.A-Sheild, Meekrab2012, joker88john, Viztor, 
DannyS712, CucyNoiD, NebulousIris, Wenyi, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Af420, Darkminds3113, 
Bsandipan, Lordiis, Adik2382, Chicocvenancio, Th3d3v1ls, Ramalepe, Liugev6, 
Tbscho, MayS, WSH1906, Lewizho99, Mdupont, JJMC89, Maathavan, Poyekhali, 
Altostratus, Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, 
El_Grafo, Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[TheSandDoctor]

TheSandDoctor added a comment.


  @Dvorapa But what would cause it to return it then when looking at images? I 
am sort of confused here.

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: TheSandDoctor
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, Beast1978, CBogen, Un1tY, Zkhalido, Hook696, Daryl-TTMG, 
RomaAmorRoma, 0010318400, E.S.A-Sheild, Meekrab2012, joker88john, Viztor, 
DannyS712, CucyNoiD, NebulousIris, Wenyi, Gaboe420, Versusxo, 
Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Af420, Darkminds3113, 
Bsandipan, Lordiis, Adik2382, Chicocvenancio, Th3d3v1ls, Ramalepe, Liugev6, 
Tbscho, MayS, WSH1906, Lewizho99, Mdupont, JJMC89, Maathavan, Poyekhali, 
Altostratus, Taiwania_Justo, Avicennasis, Ixocactus, Wong128hk, mys_721tx, 
El_Grafo, Dinoguy1000, jayvdb, Masti, Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[gerritbot]

gerritbot added a comment.


  Change 574111 had a related patch set uploaded (by Dvorapa; owner: Dvorapa):
  [pywikibot/core@master] [bugfix] Throw InvalidTitle for title containing 
illegal char
  
  https://gerrit.wikimedia.org/r/574111

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[Dvorapa]

Dvorapa added a comment.


  In T181443#5907172 , 
@TheSandDoctor wrote:
  
  > @zhuyifei1999 Do you think that such a raise could be made? The problem 
that I see with both handlings though is that the titles are //not// "invalid" 
as they are the valid image titles on the wiki(?). I am also having this issue 
when it comes to my Commons Corruption Checking task.
  
  \ufffd is invalid, no user can create a page with this in title or upload an 
image with this in name

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Dvorapa, zhuyifei1999, TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, 
Fructibus, CBogen, Zkhalido, Viztor, DannyS712, Wenyi, Chicocvenancio, Tbscho, 
MayS, Mdupont, JJMC89, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, 
Ixocactus, Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, 
Alchimista, Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER' in category add

[TheSandDoctor]

TheSandDoctor added a comment.


  @Fructibus Was this resolved or?

TASK DETAIL
  https://phabricator.wikimedia.org/T181443

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: TheSandDoctor
Cc: TheSandDoctor, Mpaa, Aklapper, pywikibot-bugs-list, Fructibus, CBogen, 
Zkhalido, Viztor, Wenyi, Chicocvenancio, Tbscho, MayS, Mdupont, JJMC89, 
Dvorapa, Poyekhali, Altostratus, Taiwania_Justo, Avicennasis, Ixocactus, 
Wong128hk, mys_721tx, El_Grafo, Dinoguy1000, jayvdb, Masti, Alchimista, 
Steinsplitter, Rxy, Keegan
___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T181443: Pywikibot stops when finding the character \uFFFD - 'REPLACEMENT CHARACTER'

[Mpaa]

Mpaa added a comment.
It is [[Category:Mus�e du quai Branly]] giving problems.TASK DETAILhttps://phabricator.wikimedia.org/T181443EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MpaaCc: Mpaa, Aklapper, pywikibot-bugs-list, Fructibus, Chicocvenancio, Poyekhali, Wong128hk, El_Grafo, Steinsplitter___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs