[Pywikipedia-bugs] [Maniphest] [Triaged] T323413: Category.subcategories(recurse=True) no longer works

[Xqt]

Xqt triaged this task as "High" priority.

TASK DETAIL
  https://phabricator.wikimedia.org/T323413

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Xqt
Cc: Xqt, Aklapper, pywikibot-bugs-list, Dvorapa, PotsdamLamb, Jyoo1011, 
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS, 
Mdupont, JJMC89, Altostratus, Avicennasis, mys_721tx, jayvdb, Masti, Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Updated] T323413: Category.subcategories(recurse=True) no longer works

[Dvorapa]

Dvorapa added a comment.


  It seems this commit broke the behavior: 
rPWBC76cbce788c6db42b5285140d100245c8bdc3cd87 
,
 before it it still works as expected

TASK DETAIL
  https://phabricator.wikimedia.org/T323413

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Xqt, Aklapper, pywikibot-bugs-list, Dvorapa, PotsdamLamb, Jyoo1011, 
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS, 
Mdupont, JJMC89, Altostratus, Avicennasis, mys_721tx, jayvdb, Masti, Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T323413: Category.subcategories(recurse=True) no longer works

[Dvorapa]

Dvorapa added a comment.


  Why does Category.members() calls Category.subcategories()? This seems to be 
completely wrong:
  
  
https://phabricator.wikimedia.org/diffusion/PWBC/browse/master/pywikibot/page/_category.py$196
  
  Also the `member_type='subcat'` is lost in `if recurse:` condition, so it 
tries to get subcategories of articles.

TASK DETAIL
  https://phabricator.wikimedia.org/T323413

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Xqt, Aklapper, pywikibot-bugs-list, Dvorapa, PotsdamLamb, Jyoo1011, 
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS, 
Mdupont, JJMC89, Altostratus, Avicennasis, mys_721tx, jayvdb, Masti, Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Edited] T323413: Category.subcategories(recurse=True) no longer works

[Dvorapa]

Dvorapa updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T323413

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Xqt, Aklapper, pywikibot-bugs-list, Dvorapa, PotsdamLamb, Jyoo1011, 
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS, 
Mdupont, JJMC89, Altostratus, Avicennasis, mys_721tx, jayvdb, Masti, Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Edited] T323413: Category.subcategories(recurse=True) no longer works

[Dvorapa]

Dvorapa updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T323413

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Xqt, Aklapper, pywikibot-bugs-list, Dvorapa, PotsdamLamb, Jyoo1011, 
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS, 
Mdupont, JJMC89, Altostratus, Avicennasis, mys_721tx, jayvdb, Masti, Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Commented On] T323413: Category.subcategories(recurse=True) no longer works

[Dvorapa]

Dvorapa added a comment.


  Reverting `subcategories` method to its state from May fixes the issue:
  
def subcategories(self,
  recurse: Union[int, bool] = False,
  total: Optional[int] = None,
  content: bool = False):
"""
Iterate all subcategories of the current category.

:param recurse: if not False or 0, also iterate subcategories of
subcategories. If an int, limit recursion to this number of
levels. (Example: recurse=1 will iterate direct subcats and
first-level sub-sub-cats, but no deeper.)
:param total: iterate no more than this number of
subcategories in total (at all levels)
:param content: if True, retrieve the content of the current 
version
of each category description page (default False)
"""
if not isinstance(recurse, bool) and recurse:
recurse = recurse - 1
if not hasattr(self, '_subcats'):
self._subcats = []
for member in self.site.categorymembers(
self, member_type='subcat', total=total, 
content=content):
subcat = Category(member)
self._subcats.append(subcat)
yield subcat
if total is not None:
total -= 1
if total == 0:
return
if recurse:
for item in subcat.subcategories(
recurse, total=total, content=content):
yield item
if total is not None:
total -= 1
if total == 0:
return
else:
for subcat in self._subcats:
yield subcat
if total is not None:
total -= 1
if total == 0:
return
if recurse:
for item in subcat.subcategories(
recurse, total=total, content=content):
yield item
if total is not None:
total -= 1
if total == 0:
return

TASK DETAIL
  https://phabricator.wikimedia.org/T323413

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Xqt, Aklapper, pywikibot-bugs-list, Dvorapa, PotsdamLamb, Jyoo1011, 
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS, 
Mdupont, JJMC89, Altostratus, Avicennasis, mys_721tx, jayvdb, Masti, Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Edited] T323413: Category.subcategories(recurse=True) no longer works

[Dvorapa]

Dvorapa updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T323413

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Dvorapa
Cc: Xqt, Aklapper, pywikibot-bugs-list, Dvorapa, PotsdamLamb, Jyoo1011, 
JohnsonLee01, SHEKH, Dijkstra, Khutuck, Zkhalido, Viztor, Wenyi, Tbscho, MayS, 
Mdupont, JJMC89, Altostratus, Avicennasis, mys_721tx, jayvdb, Masti, Alchimista
___
pywikibot-bugs mailing list -- pywikibot-bugs@lists.wikimedia.org
To unsubscribe send an email to pywikibot-bugs-le...@lists.wikimedia.org

[Pywikipedia-bugs] [Maniphest] [Created] T323413: Category.subcategories(recurse=True) no longer works

[Dvorapa]

Dvorapa created this task.
Dvorapa added a project: Pywikibot.
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper.

TASK DESCRIPTION
  **Steps to reproduce**
  
import pywikibot
s=pywikibot.Site('test2', 'wikipedia')
for i in s.allpages(namespace=14, filterredir=False):  # iterate over all 
categories
  if not i.isEmptyCategory() and next(i.categories(total=1), False):  # is 
a subcategory and contains 1 or more subcategories
c = i.subcategories(recurse=True)  # get subcategories
if i in c:  # is there a loop?
  g = set()
  for j in c:  # iterate over members
if g:
  g.add(j)  # add loop member
if j == i:  # found beginning/end of the loop
  if not g:
g.add(i)  # start tracking the loop
  else:
print(g)  # finish loop
break
  
  **Expected behavior**
  This simple code to print category loops worked for years, last time in May 
2022 
.
  
  **Current behavior**
  Lately the code started to fail after a minute of running with this long 
traceback:
  
$ python cykly-v-kategoriich2.py
ERROR: Traceback (most recent call last):
  File "/home/pavel/pywikibot/pywikibot/data/api/_requests.py", line 681, 
in _http_request
response = http.request(self.site, uri=uri,
  File "/home/pavel/pywikibot/pywikibot/comms/http.py", line 233, in request
r = fetch(baseuri, headers=headers, **kwargs)
  File "/home/pavel/pywikibot/pywikibot/comms/http.py", line 399, in fetch
callback(response)
  File "/home/pavel/pywikibot/pywikibot/comms/http.py", line 284, in 
error_handling_callback
raise response from None
  File "/home/pavel/pywikibot/pywikibot/comms/http.py", line 390, in fetch
response = session.request(method, uri,
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 587, 
in request
resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 701, 
in send
r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 489, 
in send
resp = conn.urlopen(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 
703, in urlopen
httplib_response = self._make_request(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 
449, in _make_request
six.raise_from(e, None)
  File "", line 3, in raise_from
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 
444, in _make_request
httplib_response = conn.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
response.begin()
  File "/usr/lib/python3.10/http/client.py", line 337, in begin
self.headers = self.msg = parse_headers(self.fp)
  File "/usr/lib/python3.10/http/client.py", line 236, in parse_headers
return email.parser.Parser(_class=_class).parsestr(hstring)
  File "/usr/lib/python3.10/email/parser.py", line 67, in parsestr
return self.parse(StringIO(text), headersonly=headersonly)
  File "/usr/lib/python3.10/email/parser.py", line 56, in parse
feedparser.feed(data)
  File "/usr/lib/python3.10/email/feedparser.py", line 176, in feed
self._call_parse()
  File "/usr/lib/python3.10/email/feedparser.py", line 180, in _call_parse
self._parse()
  File "/usr/lib/python3.10/email/feedparser.py", line 295, in _parsegen
if self._cur.get_content_maintype() == 'message':
  File "/usr/lib/python3.10/email/message.py", line 594, in 
get_content_maintype
ctype = self.get_content_type()
  File "/usr/lib/python3.10/email/message.py", line 578, in get_content_type
value = self.get('content-type', missing)
  File "/usr/lib/python3.10/email/message.py", line 471, in get
return self.policy.header_fetch_parse(k, v)
  File "/usr/lib/python3.10/email/_policybase.py", line 316, in 
header_fetch_parse
return self._sanitize_header(name, value)
  File "/usr/lib/python3.10/email/_policybase.py", line 287, in 
_sanitize_header
if _has_surrogates(value):
  File "/usr/lib/python3.10/email/utils.py", line 57, in _has_surrogates
s.encode()
RecursionError: maximum recursion depth exceeded while calling a Python 
object

WARNING: Waiting 5.0 seconds before retrying.
ERROR: Traceback (most recent call last):
  File "/home/pavel/pywikibot/pywikibot/data/api/_requests.py", line 681, 
in _http_request
response = http.request(self.site, uri=uri,
  File "/home/pavel/pywikibot/pywikibot/comms/http.py", line 233, in request
r = fetch(baseuri, headers=headers, **kwargs)
  File