Thank you for your help, I got some time to test some of your suggested solutions, and I drastically reduced the time to import my data by creating an item with all its claims in one call. For future readers, site.editEntity() already uses the `wbeditentity` API call, you only need to complete the `data` argument with labels, descriptions, aliases, claims ...
It's great, but I'm trying to optimize even more and If I'm not mistaken, there is no solution to reduce the number of requests to import my items, so the next step would be implementing some sort of multiprocessing, as suggested by Pellegrino Prevete. I tried to implement that, unfortunately Pywikibot raises an APIerror : invalid CSRF token. It sounds like multiple processes share the same CSRF token to create / edit an item, which is a bit weird. All in all, it got me thinking : is it even possible to use multiprocessing with Pywikibot at all ? Here's some pseudo-code to show what I did : ''' from multiprocessing.dummy import Pool pool = Pool(processes=4) results = pool.map(self.process_line, csv) pool.close() pool.join() ''' Where `csv`is a list (already parsed CSV file), `self.process_line` is my method reading the data in the current csv line and creating the item with it. > Le 5 févr. 2019 à 09:35, Andra Waagmeester <[email protected]> a écrit : > > > > 1/ The idea is to read a CSV file, and create an item with its properties for > each line. So I have to loop over thousands of lines and create an item and > multiple claims associated, and it takes quite some time to do so. (atleast 1 > hour to create 1000 items) I guess it's because for each line, I create a new > entity and new claims, which means multiple requests for each line. > > > There is an API call, wbeditentity [1] that allows preparing an item with > multiple claims which are written to WB in one call. its Are you aware of > wikibase universal bot [2] and wikibase-tools [3]? Both cover functionality > that should allow to do what you describe above and both use the wbeditentity > call. > > > > [1] https://www.wikidata.org/w/api.php?action=help&modules=wbeditentity > <https://www.wikidata.org/w/api.php?action=help&modules=wbeditentity> > [2] https://github.com/dcodings/Wikibase_Universal_Bot > <https://github.com/dcodings/Wikibase_Universal_Bot> > [3] https://github.com/stuppie/wikibase-tools > <https://github.com/stuppie/wikibase-tools>_______________________________________________ > pywikibot mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/pywikibot
_______________________________________________ pywikibot mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/pywikibot
