[Wikitech-l] [Reminder] Gerrit Cleanup Day: Wed, Sep 23

2015-09-21 Thread Andre Klapper
Reminder: This is in two days. Everybody is very welcome to join and help, especially in areas with unclear maintainership! Please check out https://phabricator.wikimedia.org/T88531 for Gerrit queries and for points of contact per area! Happy reviewing!, andre On Tue, 2015-09-01 at 00:27 +0200,

Re: [Wikitech-l] LDAP extension ownership

2015-09-21 Thread Chris Steipp
On Sep 19, 2015 11:15 AM, "bawolff" wrote: > > maintain is an ambiguous word. WMF has some responsibility to all the > extensions deployed on cluster (imo). If Devunt (and any others who > were knowledgeable of the Josa extension) disappeared, WMF would > default to becoming

Re: [Wikitech-l] LDAP extension ownership

2015-09-21 Thread Ryan Lane
On Mon, Sep 21, 2015 at 8:41 AM, Chris Steipp wrote: > On Sep 19, 2015 11:15 AM, "bawolff" wrote: > > > > maintain is an ambiguous word. WMF has some responsibility to all the > > extensions deployed on cluster (imo). If Devunt (and any others who >

[Wikitech-l] AuthManager (was Re: LDAP extension ownership)

2015-09-21 Thread Brad Jorsch (Anomie)
On Mon, Sep 21, 2015 at 1:22 PM, Ryan Lane wrote: > I know someone is working on an auth framework update, so I'm sure > there'll be some changes necessary for that too. > We're planning on making the changes necessary for AuthManager in WMF-deployed extensions (including

Re: [Wikitech-l] Importing XML Dumps - templates not working

2015-09-21 Thread C. Scott Ananian
Note that Kiwix's "mw-offliner" script ( http://www.openzim.org/wiki/Build_your_ZIM_file#MWoffliner ) does a pretty good job of converting a bunch of wiki pages to HTML, although it starts from a live wiki instance (and a properly-configured Parsoid pointed at it) rather than an XML dump.

[Wikitech-l] Importing XML Dumps - templates not working

2015-09-21 Thread v0id null
Hello Everyone, I've been trying to write a python script that will take an XML dump, and generate all HTML, using Mediawiki itself to handle all the parsing/processing, but I've run into a problem where all the parsed output have warnings that templates couldn't be found. I'm not sure what I'm

Re: [Wikitech-l] Importing XML Dumps - templates not working

2015-09-21 Thread v0id null
http://dumps.wikimedia.org/enwikinews/latest/enwikinews-latest-pages-articles.xml.bz2 this one. I believe this was to contain all latest revisions of all pages. I do see that there are template pages in there, at least, they are pages with a title in the format of Template:[some template name]

Re: [Wikitech-l] Importing XML Dumps - templates not working

2015-09-21 Thread Brion Vibber
A few notes: 1) It sounds like you're recreating all the logic of importing a dump into a SQL database, which may be introducing problems if you have bugs in your code. For instance you may be mistakenly treating namespaces as text strings instead of numbers, or failing to escape things, or

Re: [Wikitech-l] Importing XML Dumps - templates not working

2015-09-21 Thread v0id null
#1: mwdumper has not been updated in a very long time. I did try to use it, but it did not seem to work properly. I don't entirely remember what the problem was but I believe it was related to schema incompatibility. xml2sql comes with a warning about having to rebuild links. Considering that I'm

Re: [Wikitech-l] Importing XML Dumps - templates not working

2015-09-21 Thread John
What kind of dump are you working from? On Mon, Sep 21, 2015 at 2:50 PM, v0id null wrote: > Hello Everyone, > > I've been trying to write a python script that will take an XML dump, and > generate all HTML, using Mediawiki itself to handle all the > parsing/processing, but

Re: [Wikitech-l] Importing XML Dumps - templates not working

2015-09-21 Thread Brion Vibber
Your import process is definitely broken. page_title should be just 'Date', while page_namespace has the numeric key for template pages. -- brion On Mon, Sep 21, 2015 at 12:02 PM, v0id null wrote: > For example, the above mentioned missing template does seem to exist from

Re: [Wikitech-l] Importing XML Dumps - templates not working

2015-09-21 Thread Brion Vibber
On Mon, Sep 21, 2015 at 12:09 PM, v0id null wrote: > #1: mwdumper has not been updated in a very long time. I did try to use it, > but it did not seem to work properly. I don't entirely remember what the > problem was but I believe it was related to schema incompatibility.

Re: [Wikitech-l] Importing XML Dumps - templates not working

2015-09-21 Thread v0id null
For example, the above mentioned missing template does seem to exist from what I can tell: mysql> select page_title from page where page_title='Template:Date'; +---+ | page_title| +---+ | Template:Date | +---+ 1 row in set (0.02 sec) On Mon, Sep 21, 2015