brion added a comment.
In https://phabricator.wikimedia.org/T120452#2227240, @matmarex wrote: > As I understand these are stored as regular MediaWiki pages now, so they have a maximum length of 2 MB. Even naive queries pulling the whole thing into memory would be fast enough at these scales. If we want to think about performance for large data, we should first think about overcoming the length limitation :) Ah I forgot all about that. ;) That'll at least stop people from creating _super_ huge datasets for now... unless they break them into multiple files and create lua modules to stitch them back together. :) That may be acceptable however, and lets people prototype the kinds of crazy things they want until we hate it so much we decide we have to support them better. I tried generating some random tables with this PHP script: https://gist.github.com/brion/46469ac2df31a8eb0e179f50b1967d20 I find I can only successfully save a file of under a megabyte even though the max is 2 megs; a 2 meg file gets rejected complaining that it's over 4 megabytes... I notice the output that comes back to me in the edit window is pretty-printed, which means a lot of indentation and newlines bulking up the storage requirements. That might be on purpose to provide a better diff view? Seems to run reasonably fast to render locally, though it also produces a giant HTML table for the page view that's about 3.5 megs, which should at least compress ok. Editing also means resubmitting the entire data set at once with the edit form. A partial-edit submission system similar to section editing on wiki pages might be nice to reduce submission latency and bandwidth consumption editing a table, but that can probably wait until it needs to be paired with a spreadsheet-like UI. TASK DETAIL https://phabricator.wikimedia.org/T120452 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Yurik, brion Cc: Jdforrester-PERSONAL, Jdforrester-WMF, brion, ThurnerRupert, intracer, TerraCodes, Pokefan95, gerritbot, -jem-, Bawolff, MZMcBride, Alkamid, Milimetric, Thryduulf, JEumerus, MarkTraceur, Yurik, Matanya, ekkis, matmarex, Lydia_Pintscher, Aklapper, Steinsplitter, StudiesWorld, DannyH, Riley_Huntley, D3r1ck01, Izno, JAllemandou, Wikidata-bugs, aude, El_Grafo, Ricordisamoa, Shizhao, Fabrice_Florin, Mbch331, Jay8g, Krenair, jeremyb _______________________________________________ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
