[Pywikipedia-bugs] [Maniphest] [Commented On] T209111: APIError: too-many-titles with -start: page generator

2018-11-16 Thread gerritbot 
gerritbot added a comment.
Change 473940 merged by jenkins-bot:
[pywikibot/core@master] [FIX] site.preloadpages: split pagelist in at most max_ids elements

https://gerrit.wikimedia.org/r/473940TASK DETAILhttps://phabricator.wikimedia.org/T209111EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Mpaa, gerritbotCc: gerritbot, Mpaa, Xqt, Phuzion, Aklapper, pywikibot-bugs-list, Framawiki, CucyNoiD, NebulousIris, Wenyi, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Baloch007, Darkminds3113, Bsandipan, Lordiis, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, Tbscho, MayS, Lewizho99, Mdupont, JJMC89, Maathavan, Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T209111: APIError: too-many-titles with -start: page generator

2018-11-15 Thread gerritbot 
gerritbot added a comment.
Change 473940 had a related patch set uploaded (by Mpaa; owner: Mpaa):
[pywikibot/core@master] [FIX] site.preloadpages: split pagelist in max_ids maximum

https://gerrit.wikimedia.org/r/473940TASK DETAILhttps://phabricator.wikimedia.org/T209111EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gerritbotCc: gerritbot, Mpaa, Xqt, Phuzion, Aklapper, pywikibot-bugs-list, Framawiki, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T209111: APIError: too-many-titles with -start: page generator

2018-11-15 Thread Mpaa 
Mpaa added a comment.
The problem is that in site.preloadpages(), max_ids is computed after the pageslist is splitted in chunks of 240.

for sublist in itergroup(pagelist, groupsize):   <--- groupsize = 240
# Do not use p.pageid property as it will force page loading.
pageids = [str(p._pageid) for p in sublist
   if hasattr(p, '_pageid') and p._pageid > 0]
cache = {}
# In case of duplicates, return the first entry.
for priority, page in enumerate(sublist):
try:
cache.setdefault(page.title(with_section=False),
 (priority, page))
except pywikibot.InvalidTitle:
pywikibot.exception()

prio_queue = []
next_prio = 0
rvgen = api.PropertyGenerator(props, site=self)
rvgen.set_maximum_items(-1)  # suppress use of "rvlimit" parameter

parameter = self._paraminfo.parameter('query+info', 'prop')
if self.logged_in() and self.has_right('apihighlimits'):  <--- False
max_ids = int(parameter['highlimit'])
else:
max_ids = int(parameter['limit'])  # T78333, T161783   <--- max_ids = 50

if len(pageids) == len(sublist) and len(set(pageids)) <= max_ids: <--- False
# only use pageids if all pages have them
rvgen.request['pageids'] = set(pageids)
else:
rvgen.request['titles'] = list(cache.keys()) <--- len(cache) = 240  ---> PROBLEM!
rvgen.request['rvprop'] = rvprop
pywikibot.output('Retrieving %s pages from %s.'
 % (len(cache), self))TASK DETAILhttps://phabricator.wikimedia.org/T209111EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MpaaCc: Mpaa, Xqt, Phuzion, Aklapper, pywikibot-bugs-list, Framawiki, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T209111: APIError: too-many-titles with -start: page generator

2018-11-14 Thread Xqt 
Xqt added a comment.
Cannot reproduce it neither with python 3 nor 2:

C:\pwb\GIT\core>py -3 pwb.py weblinkchecker -start:! -simulate -user:xqt -lang:en
Retrieving 240 pages from wikipedia:en.


>>> !! <<<


>>> !!! <<<



C:\pwb\GIT\core>py -3 pwb.py version
Pywikibot: [ssh] pywikibot-core (69d0032, g10396, 2018/11/13, 17:21:12, ok)
Release version: 3.1.dev0
requests version: 2.19.1
  cacerts: C:\python37\lib\site-packages\certifi\cacert.pem
certificate test: ok
Python: 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:06:47) [MSC v.1914 32 bit (Int
el)]



C:\pwb\GIT\core>py -2 pwb.py version
Pywikibot: [ssh] pywikibot-core (69d0032, g10396, 2018/11/13, 17:21:12, ok)
Release version: 3.1.dev0
requests version: 2.9.1
  cacerts: C:\Python27\lib\site-packages\certifi\cacert.pem
certificate test: ok
Python: 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit
(Intel)]TASK DETAILhttps://phabricator.wikimedia.org/T209111EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: XqtCc: Xqt, Phuzion, Aklapper, pywikibot-bugs-list, Framawiki, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs

[Pywikipedia-bugs] [Maniphest] [Commented On] T209111: APIError: too-many-titles with -start: page generator

2018-11-08 Thread Framawiki 
Framawiki added a comment.
Same error as bot user:

tools.framabot@tools-bastion-02:~$ ./pwb.py version
Pywikibot: [https] r-pywikibot-core.git (7ea6fba, g1, 2018/11/05, 17:45:43, ok)
Release version: 3.1.dev0
requests version: 2.2.1
  cacerts: /etc/ssl/certs/ca-certificates.crt
certificate test: ok
Python: 2.7.6 (default, Nov 23 2017, 15:49:48) 
[GCC 4.8.4]
Toolforge hostname: tools-bastion-02
PYWIKIBOT_DIR: Not set
PYWIKIBOT_DIR_PWB: .
PYWIKIBOT_NO_USER_CONFIG: Not set
Config base dir: /data/project/framabot/.pywikibot
Usernames for family "wikinews":
	*: Framabot (no sysop configured)
Usernames for family "wikiquote":
	*: Framabot (no sysop configured)
Usernames for family "wikipedia":
	*: Framabot (no sysop configured)
Usernames for family "meta":
	*: Framabot (no sysop configured)
Usernames for family "wikidata":
	*: Framabot (no sysop configured)
Usernames for family "wikisource":
	*: Framabot (no sysop configured)
Usernames for family "wiktionary":
	*: Framabot (no sysop configured)
Usernames for family "commons":
	*: Framabot (no sysop configured)
Usernames for family "wikivoyage":
	*: Framabot (no sysop configured)
Usernames for family "wikiversity":
	*: Framabot (no sysop configured)
Usernames for family "wikibooks":
	*: Framabot (no sysop configured)TASK DETAILhttps://phabricator.wikimedia.org/T209111EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: FramawikiCc: Phuzion, Aklapper, pywikibot-bugs-list, Framawiki, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Avicennasis, mys_721tx, jayvdb, Dalba, Masti, Alchimista, Rxy___
pywikibot-bugs mailing list
pywikibot-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs