> > I was envisioning that we would produce "diff dumps" in one pass > (presumably in a much shorter time than the fulls we generate now) and > would apply those against previous fulls (in the new format) to produce > new fulls, hopefully also in less time. What do you have in mind for > the production of the new fulls? >
What I originally imagined is that the full dump will be modified directly and a description of the changes made to it will be also written to the diff dump. But now I think that creating the diff and then applying it makes more sense, because it's simpler. But I also think that doing the two at the same time will be faster, because it's less work (no need to read and parse the diff). So what I imagine now is something like this: 1. Read information about a change in a page/revision 2. Create diff object in memory 3. Write the diff object to the diff file 4. Apply the diff object to the full dump > It might be worth seeing how large the resulting en wp history files are > going to be if you compress each revision separaately for version 1 of > this project. My fear is that even with 7z it's going to make the size > unwieldy. If the thought is that it's a first round prototype, not > meant to be run on large projects, that's another story. > I do expect that full dump of enwiki using this compression would be way too big. So yes, this was meant just to have something working, so that I can concentrate on doing compression properly later (after the mid-term). > I'm not sure about removing the restrictions data; someone must have > wanted it, like the other various fields that have crept in over time. > And we should expect there will be more such fields over time... > If I understand the code in XmlDumpWriter.openPage correctly, that data comes from the page_restrictions row [1], which doesn't seem to be used in non-ancient versions of MediaWiki. I did think about versioning the page and revision objects in the dump, but I'm not sure how exactly to handle upgrades from one version to another. For now, I think I'll have just one global "data version" per file, but I'll make sure that adding a version to each object in the future will be possible. > We need to get some of the wikidata users in on the model/format > discussion, to see what use they plan to make of those fields and what > would be most convenient for them. > > It's quite likely that these new fulls will need to be split into chunks > much as we do with the current en wp files. I don't know what that > would mean for the diff files. Currently we split in an arbitrary way > based on sequences of page numbers, writing out separate stub files and > using those for the content dumps. Any thoughts? > If possible, I would prefer to keep everything in a single file. If that won't be possible, I think it makes sense to split on page ids, but make the split id visible (probably in the file name) and unchanging from month to month. If it turns out that a single chunk grows too big, we might consider adding a "split" instruction to diff dumps, but that's probably not necessary now. Petr Onderka [1]: http://www.mediawiki.org/wiki/Manual:Page_table#page_restrictions _______________________________________________ Wikitech-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikitech-l
