Dominicbm added a comment.
In T253591#6164399 <https://phabricator.wikimedia.org/T253591#6164399>, @Xqt wrote: >> The problem is that, for my use case, I really just want page titles, but I guess since Pywikibot wants to generate all the page objects using all the metadata, there is no way around this error currently. > > `CategorizedPageGenerator `does not load the content by default. To get it you have to use `content=False` as parameter or use an explicit cat.get() later. It's not the page content, but the image metadata. So, if I look at the request in debug mode, even without `content=True`, since `metadata` is always included in `iiprop`, as you can see below, you might get an over-12MB response for certain media files. API request to commons:commons (uses get: False): Headers: {'Content-Type': 'application/x-www-form-urlencoded'} URI: '/w/api.php' Body: 'gcmtitle=Category%3AMedia+contributed+by+the+Digital+Public+Library+of+America&gcmprop=ids%7Ctitle%7Csortkey&gcmtype=page%7Cfile&prop=info%7Cimageinfo%7Ccategoryinfo&inprop=protection&iiprop=timestamp%7Cuser%7Ccomment%7Curl%7Csize%7Csha1%7Cmetadata&iilimit=max&generator=categorymembers&action=query&indexpageids=&continue=&gcmnamespace=6&gcmlimit=1&meta=userinfo&uiprop=blockinfo%7Chasmsg&maxlag=5&format=json' I put `step` lower at `50`, a s you suggested, and was still getting the warning. So now I am trying with `1`, but presumably there will always be a possibility of an API response that is too large as long as the metadata is always retrieved, since there is nothing on the Wikimedia end that prevents a single media file's metadata from exceeding 12MB. Consider the case where you have a 500 page PDF with an OCR text layer, for example. Here is a single file <https://commons.wikimedia.org/w/api.php?action=query&prop=imageinfo&iiprop=commonmetadata&titles=File:Hill%27s_Kannapolis_(Cabarrus_County,_N.C.)_City_Directory_(1953)_-_DPLA_-_b6f29ed7c87879f14be00678df89a6dc.pdf&format=json> in which the API response amounts to almost 3MB because of all the image metadata. A single file 4 times that size, or just 4 such files in a single set of results, and you're getting truncated responses. I am also concerned what the performance implication will be for being forced to use such a low query increment, since I am applying it across the whole results set and not just the requests that were truncated. The category I was attempting to run this on is expected to have at least 700,000 members, for example. Changing `step` for the whole thing drastically increases the number of requests required to complete it. I guess what I was asking for was an argument similar to the `content=` one you mentioned, but where I can do `img_metadata=False`. If I am understanding the cause here, this would solve the problem for me and anyone else with the issue (unless you really want the image metadata, but then you're out of luck in a way Pywikibot can't solve, without T86611 <https://phabricator.wikimedia.org/T86611> getting fixed). TASK DETAIL https://phabricator.wikimedia.org/T253591 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Dominicbm Cc: Xqt, Aklapper, pywikibot-bugs-list, Dominicbm, JohnsonLee01, Dijkstra, Zkhalido, Viztor, Wenyi, Tbscho, MayS, Mdupont, JJMC89, Dvorapa, Altostratus, Avicennasis, mys_721tx, jayvdb, Masti, Alchimista, Rxy
_______________________________________________ pywikibot-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
