jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/364933 )
Change subject: T114072: Add <section> wrappers to Parsoid output ...................................................................... T114072: Add <section> wrappers to Parsoid output Core implementation ------------------- This patch implements Proposal 1 in https://mediawiki.org/wiki/Parsing/Notes/Section_Wrapping Every section has this structure: <section data-mw-section-id='..'> <h*>..</h*> .. </section> Note that the <h*> element is not present for lead and pseudo-sections. The value of data-mw-section-is id: data-mw-section-id = 0 for the lead section data-mw-section-id = -1 for non-editable sections (see below) data-mw-section-id = -2 for pseudo sections (see below) data-mw-section-id > 0 for everything else this number matches PHP parser / Mediawiki's notion of that section. Additional notes ---------------- * Since ContentTranslation (CX) currently uses <section> tags internally, to ensure that CX can distinguish between its internal use with Parsoid-provided <section> tags, we need to add a distinguishing attribute to our section tags. The data-mw-section-id attribute is present on ALL <section> tags added by Parsoid, and is not present on CX-internal <section> tags. * Non-editable & pseudo sections: For sections where it is not possible to provide that 1-1 guarantee, it suppresses editability by setting the data-mw-section-id attr. to -1. This is the case for sections that are embedded in transclusions or for which there is no equivalent MediaWiki section. Ex 1: {{1x|1=\n=a=\nx\n=b=\ny\n}} Ex 2: <div>x\n=a=\nx\n=b=\ny</div>z For these sections, the DOM subtree rooted at the <section> does not correspond exactly with the wikitext region created by splitting at the headings. For section wrapping in scenarios like in Example 2 on [[mw:Parsing/Notes/Section_Wrapping]], where a block tag contains nested wikitext sections, in order to maintain the invariant that all content is wrapped in some <section> tag, this patch introduces the notion of pseudo-sections where the first child is not a <h*> tag like other sections. * Section-wrapping and template-wrapping interactions: Since section wrapping re-organizes the DOM structure, it can occasionally interact badly with template wrapping by breaking template content continuity semantics. In order to preserve those semantics (whether a client chooses to retain or strip a section tag), this patch does one of two things depending on the scenario: (a) expand the scope of an existing template range to cover the entire content of a section when only part of its content is template-generated. (b) add an additional template wrapping layer on top of the existing one when the expanded scope wraps multiple sections which were previoulsy not template-generated. Parser tests clarify this behavior. In order to support this functionality and not break html2wt when clients strip <section> tags and expose the original template wrapper layer, had to tweak the cleanup pass to leave behind DSR info on nodes that had template marker information. * Added a wrapSections config property to ParsoidConfig which defaults to true. - parserTests.js flips it to false so that we don't have to update every single parser test. Individual tests can enable section wrapping. This is supported in code by examining the wrapSections flag in parsoid config as well as the parsing env. - Added a noWrapSections flag to bin/parse.js which defaults to false * html2wt pass will strip the <section> wrapper before DOM Diffing. This lets us accept DOMs with / without those wrappers. Because of this stripping, the section-wrapping code does not attempt to set DSR offsets for the section tag since doing that properly adds unnecessary code complexity. * HTML version number will be bumped separately. Testing ------- * Updated parserTests.js to process a wrapSections option set on specific parser tests. Added a FIXME in DOMPostProcessor about a parser-pipeline-construction gotcha because of how parsertests shares parser env and cannot select the appropriate pipeline type based on parser test options. * Added a bunch of basic parser tests verifying section wrapping behaviour. * Three selser tests added to blacklist, which are false-positive newline diffs (2 vs 3 newlines which are functionally equivalent wrt wrapping paragraphs). Change-Id: I0f4c19f7ca4ebc88eb85c53ab145b54fde7ab455 --- M bin/parse.js M bin/parserTests.js M config.example.yaml M lib/config/ParsoidConfig.js M lib/html2wt/SelectiveSerializer.js M lib/html2wt/WikitextSerializer.js M lib/utils/DOMUtils.js M lib/wt2html/DOMPostProcessor.js M lib/wt2html/pp/handlers/cleanup.js A lib/wt2html/pp/processors/wrapSections.js M tests/mocha/dsr.js M tests/mocha/heading.ids.js M tests/mocha/linter.js M tests/mocha/parse.js M tests/mocha/regression.specs.js M tests/mocha/test.config.yaml M tests/parserTests-blacklist.js M tests/parserTests.txt 18 files changed, 846 insertions(+), 18 deletions(-) Approvals: C. Scott Ananian: Looks good to me, approved jenkins-bot: Verified diff --git a/bin/parse.js b/bin/parse.js index 7a4f2b5..6c66022 100755 --- a/bin/parse.js +++ b/bin/parse.js @@ -178,6 +178,11 @@ 'boolean': false, 'default': null, }, + 'noSectionWrap': { + description: 'Do not output <section> tags', + 'boolean': true, + 'default': false, + }, }); (function() { @@ -265,6 +270,10 @@ parsoidOptions.localsettings = path.resolve(__dirname, parsoidOptions.localsettings); } + if (argv.noSectionWrap) { + parsoidOptions.wrapSections = !argv.noSectionWrap; + } + var nock, dir, nocksFile; if (argv.record || argv.replay) { prefix = prefix || 'enwiki'; diff --git a/bin/parserTests.js b/bin/parserTests.js index dfcd8da..f9292aa 100755 --- a/bin/parserTests.js +++ b/bin/parserTests.js @@ -704,15 +704,13 @@ this.env.conf.wiki.allowExternalImages = undefined; } - this.env.scrubWikitext = item.options.parsoid && - item.options.parsoid.hasOwnProperty('scrubWikitext') ? - item.options.parsoid.scrubWikitext : - MWParserEnvironment.prototype.scrubWikitext; - - this.env.nativeGallery = item.options.parsoid && - item.options.parsoid.hasOwnProperty('nativeGallery') ? - item.options.parsoid.nativeGallery : - MWParserEnvironment.prototype.nativeGallery; + // Process test-specific options + var env = this.env; + ['scrubWikitext', 'nativeGallery', 'wrapSections'].forEach(function(opt) { + env[opt] = item.options.parsoid && + item.options.parsoid.hasOwnProperty(opt) ? + item.options.parsoid[opt] : MWParserEnvironment.prototype[opt]; + }); this.env.conf.wiki.responsiveReferences = (item.options.parsoid && item.options.parsoid.responsiveReferences) || @@ -1010,6 +1008,9 @@ // Needed for bidi-char-scrubbing html2wt tests. parsoidConfig.scrubBidiChars = true; + // Default to false and let individual tests opt in + parsoidConfig.wrapSections = false; + var extensions = parsoidConfig.defaultNativeExtensions.concat(ParserHook); // Send all requests to the mock API server. diff --git a/config.example.yaml b/config.example.yaml index 1fd7077..e1bd9d7 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -57,6 +57,9 @@ # to load WMF's config for wikipedias, etc. #loadWMF: true + # Adds <section> tags around mediwiki sections (defaults to true) + #wrapSections: false + # A default proxy to connect to the API endpoints. # Default: undefined (no proxying). # Overridden by per-wiki proxy config in setMwApi. diff --git a/lib/config/ParsoidConfig.js b/lib/config/ParsoidConfig.js index 4ff9aa6..99e39c5 100644 --- a/lib/config/ParsoidConfig.js +++ b/lib/config/ParsoidConfig.js @@ -272,6 +272,11 @@ ParsoidConfig.prototype.addHTMLTemplateParameters = false; /** + * Add section wrappers to output + */ +ParsoidConfig.prototype.wrapSections = true; + +/** * @property {boolean|Array} linting Whether to enable linter Backend. * Or an array of enabled lint types */ diff --git a/lib/html2wt/SelectiveSerializer.js b/lib/html2wt/SelectiveSerializer.js index 973268a..66f286d 100644 --- a/lib/html2wt/SelectiveSerializer.js +++ b/lib/html2wt/SelectiveSerializer.js @@ -77,6 +77,12 @@ startTimers.set('html2wt.selser.domDiff', Date.now()); } + // Strip <section> tags, if present. + // This ensures that we can accept HTML from CX / VE + // and other clients that might have stripped them. + DU.stripSectionTags(body); + DU.stripSectionTags(this.env.page.dom); + diff = (new DOMDiff(this.env)).diff(this.env.page.dom, body); if (metrics) { diff --git a/lib/html2wt/WikitextSerializer.js b/lib/html2wt/WikitextSerializer.js index d404bee..ef13570 100644 --- a/lib/html2wt/WikitextSerializer.js +++ b/lib/html2wt/WikitextSerializer.js @@ -1367,6 +1367,12 @@ // scenarios (Ex: <ref> nested in <references>). console.assert(this.env.page.editedDoc, 'Should be set.'); + if (!selserMode) { + // Strip <section> tags + // Selser mode will have done that already before running dom-diff + DU.stripSectionTags(body); + } + this.logType = selserMode ? "trace/selser" : "trace/wts"; this.trace = this.env.log.bind(this.env, this.logType); diff --git a/lib/utils/DOMUtils.js b/lib/utils/DOMUtils.js index 6f1cc61..f6d0365 100644 --- a/lib/utils/DOMUtils.js +++ b/lib/utils/DOMUtils.js @@ -1224,12 +1224,24 @@ /** * Get the first child element or non-IEW text node, ignoring - * whitespace-only text nodes and comments. + * whitespace-only text nodes, comments, and deleted nodes */ firstNonSepChildNode: function(node) { var child = node.firstChild; while (child && !this.isContentNode(child)) { child = child.nextSibling; + } + return child; + }, + + /** + * Get the last child element or non-IEW text node, ignoring + * whitespace-only text nodes, comments, and deleted nodes + */ + lastNonSepChildNode: function(node) { + var child = node.lastChild; + while (child && !this.isContentNode(child)) { + child = child.previousSibling; } return child; }, @@ -2868,6 +2880,24 @@ }); }); +DOMUtils.stripSectionTags = function(node) { + var n = node.firstChild; + while (n) { + var next = n.nextSibling; + if (DU.isElt(n)) { + // Recurse into subtree before stripping this + DU.stripSectionTags(n); + + // Strip <section> tags (except if they have template info) + if (n.nodeName === 'SECTION' && !DU.findFirstEncapsulationWrapperNode(n)) { + DU.migrateChildren(n, n.parentNode, n); + DU.deleteNode(n); + } + } + n = next; + } +}; + if (typeof module === "object") { module.exports.DOMUtils = DOMUtils; } diff --git a/lib/wt2html/DOMPostProcessor.js b/lib/wt2html/DOMPostProcessor.js index 2d9021f..2e3b97c 100644 --- a/lib/wt2html/DOMPostProcessor.js +++ b/lib/wt2html/DOMPostProcessor.js @@ -26,6 +26,7 @@ var migrateTrailingNLs = requireProcessor('migrateTrailingNLs'); var computeDSR = requireProcessor('computeDSR'); var wrapTemplates = requireProcessor('wrapTemplates'); +var wrapSections = requireProcessor('wrapSections'); // handlers var requireHandlers = function(file) { @@ -102,6 +103,28 @@ this.env = env; this.options = options; + /* --------------------------------------------------------------------------- + * FIXME: + * 1. PipelineFactory caches pipelines per env + * 2. PipelineFactory.parse uses a default cache key + * 3. ParserTests uses a shared/global env object for all tests. + * 4. ParserTests also uses PipelineFactory.parse (via env.getContentHandler()) + * => the pipeline constructed for the first test that runs wt2html + * is used for all subsequent wt2html tests + * 5. If we are selectively turning on/off options on a per-test basis + * in parser tests, those options won't work if those options are + * also used to configure pipeline construction (including which DOM passes + * are enabled). + * + * Ex: if (env.wrapSections) { addPP('wrapSections', wrapSections); } + * + * This won't do what you expect it to do. This is primarily a + * parser tests script issue -- but given the abstraction layers that + * are on top of the parser pipeline construction, fixing that is + * not straightforward right now. So, this note is a warning to future + * developers to pay attention to how they construct pipelines. + * --------------------------------------------------------------------------- */ + this.processors = []; var self = this; var addPP = function(name, proc) { @@ -172,13 +195,17 @@ domVisitor.addHandler('td', tableFixer.stripDoubleTDs.bind(tableFixer)); domVisitor.addHandler('td', tableFixer.handleTableCellTemplates.bind(tableFixer)); domVisitor.addHandler('th', tableFixer.handleTableCellTemplates.bind(tableFixer)); - addPP('stripMarkers+(li+table)Fixups', domVisitor.traverse.bind(domVisitor)); + // 4. Add heading anchors + domVisitor.addHandler(null, headings.genAnchors); + addPP('stripMarkers+(li+table)Fixups+headings', domVisitor.traverse.bind(domVisitor)); + + // Add <section> wrappers around sections + addPP('wrapSections', wrapSections); // Save data.parsoid into data-parsoid html attribute. // Make this its own thing so that any changes to the DOM // don't affect other handlers that run alongside it. domVisitor = new DOMTraverser(env); - domVisitor.addHandler(null, headings.genAnchors); domVisitor.addHandler(null, CleanUp.cleanupAndSaveDataParsoid); addPP('cleanupAndSaveDP', domVisitor.traverse.bind(domVisitor)); } diff --git a/lib/wt2html/pp/handlers/cleanup.js b/lib/wt2html/pp/handlers/cleanup.js index 4a69b89..bc39db4 100644 --- a/lib/wt2html/pp/handlers/cleanup.js +++ b/lib/wt2html/pp/handlers/cleanup.js @@ -164,7 +164,14 @@ // // This is only needed for the last top-level node . && (!dp.stx || tplInfo.last !== node) - ) { discardDataParsoid = true; } + // <section> tags and template-wrapping interactions forces + // us to preserve dsr on original first template node because + // we strip <section> tags in html->wt direction and need dsr here. + && (tplInfo.first.nodeName !== 'SECTION' || + !/\bmw:Transclusion\b/.test(node.getAttribute('typeof'))) + ) { + discardDataParsoid = true; + } DU.storeDataAttribs(node, { discardDataParsoid: discardDataParsoid, diff --git a/lib/wt2html/pp/processors/wrapSections.js b/lib/wt2html/pp/processors/wrapSections.js new file mode 100644 index 0000000..15d2806 --- /dev/null +++ b/lib/wt2html/pp/processors/wrapSections.js @@ -0,0 +1,330 @@ +'use strict'; + +var DU = require('../../../utils/DOMUtils.js').DOMUtils; +var Util = require('../../../utils/Util.js').Util; +var JSUtils = require('../../../utils/jsutils.js').JSUtils; + +var lastItem = JSUtils.lastItem; + +function createNewSection(state, rootNode, sectionStack, tplInfo, currSection, node, newLevel, pseudoSection) { + /* Structure for regular (editable or not) sections + * <section data-mw-section-id=".."> + * <h*>..</h*> + * .. + * </section> + * + * Lead sections and pseudo-sections won't have <h*> or <div> tags + */ + var section = { + level: newLevel, + container: state.doc.createElement('section'), + }; + + + /* Step 1. Get section stack to the right nesting level + * 1a. Pop stack till we have a higher-level section. + */ + var stack = sectionStack; + while (stack.length > 0 && newLevel <= lastItem(stack).level) { + stack.pop(); + } + + /* 1b. Push current section onto stack if it is a higher-level section */ + if (currSection && newLevel > currSection.level) { + stack.push(currSection); + } + + /* Step 2: Add new section where it belongs: a parent section OR body */ + var parentSection = stack.length > 0 ? lastItem(stack) : null; + if (parentSection) { + parentSection.container.appendChild(section.container); + } else { + rootNode.insertBefore(section.container, node); + } + + /* Step 3: Add <h*> to the <section> */ + section.container.appendChild(node); + + /* Step 4: Assign data-mw-section-id attribute + * + * CX wants <section> tags with a distinguishing attribute so that + * it can differentiate between its internal use of <section> tags + * with what Parsoid adds. So, we will add a data-mw-section-id + * attribute always. + * + * data-mw-section-id = 0 for the lead section + * data-mw-section-id = -1 for non-editable sections + * Note that templated content cannot be edited directly. + * data-mw-section-id = -2 for pseudo sections + * data-mw-section-id > 0 for everything else and this number + * matches PHP parser / Mediawiki's notion of that section. + * + * The code here handles uneditable sections because of templating. + */ + if (state.inTemplate) { + section.container.setAttribute('data-mw-section-id', -1); + } else if (pseudoSection) { + section.container.setAttribute('data-mw-section-id', -2); + } else { + section.container.setAttribute('data-mw-section-id', state.sectionNumber); + } + + /* Ensure that template continuity is not broken if the section + * tags aren't stripped by a client */ + if (tplInfo && node !== tplInfo.first) { + section.container.setAttribute('about', tplInfo.about); + } + + return section; +} + +function wrapSectionsInDOM(state, currSection, rootNode) { + var tplInfo = null; + var sectionStack = []; + var highestSectionLevel = 7; + var node = rootNode.firstChild; + while (node) { + var next = node.nextSibling; + var addedNode = false; + + // Track entry into templated output + if (!state.inTemplate && DU.isFirstEncapsulationWrapperNode(node)) { + var about = node.getAttribute("about"); + state.inTemplate = true; + tplInfo = { + first: node, + about: about, + last: lastItem(DU.getAboutSiblings(node, about)), + }; + } + + if (/^H[1-6]$/.test(node.nodeName)) { + var level = Number(node.nodeName[1]); + + // HTML <h*> tags don't get section numbers! + if (!DU.isLiteralHTMLNode(node)) { + state.sectionNumber++; + if (level < highestSectionLevel) { + highestSectionLevel = level; + } + currSection = createNewSection(state, rootNode, sectionStack, tplInfo, currSection, node, level); + addedNode = true; + } + } else if (DU.isElt(node)) { + // If we find a higher level nested section, + // (a) Make current section non-editable + // (b) There are 2 options here. + // Best illustrated with an example + // Consider the wiktiext below. + // <div> + // =1= + // b + // </div> + // c + // =2= + // 1. Create a new pseudo-section to wrap 'node' + // There will be a <section> around the <div> which includes 'c'. + // 2. Don't create the pseudo-section by setting 'currSection = null' + // But, this can leave some content outside any top-level section. + // 'c' will not be in any section. + // The code below implements strategy 1. + var nestedHighestSectionLevel = wrapSectionsInDOM(state, null, node); + if (currSection && nestedHighestSectionLevel <= currSection.level) { + currSection.container.setAttribute('data-mw-section-id', -1); + currSection = createNewSection(state, rootNode, sectionStack, tplInfo, currSection, node, nestedHighestSectionLevel, true); + addedNode = true; + } + } + + if (currSection && !addedNode) { + currSection.container.appendChild(node); + } + + if (tplInfo && tplInfo.first === node) { + tplInfo.firstSection = currSection; + } + + // Track exit from templated output + if (tplInfo && tplInfo.last === node) { + // The opening node and closing node of the template + // are in different sections! This might require resolution. + if (currSection !== tplInfo.firstSection) { + tplInfo.lastSection = currSection; + state.templatesToExamine.push(tplInfo); + } + + tplInfo = null; + state.inTemplate = false; + } + + node = next; + } + + // The last section embedded in a non-body DOM element + // should always be marked non-editable since it will have + // the closing tag (ex: </div>) showing up in the source editor + // which we cannot support in a visual editing environment. + if (currSection && !DU.isBody(rootNode)) { + currSection.container.setAttribute('data-mw-section-id', -1); + } + + return highestSectionLevel; +} + +function getDSR(node, start) { + if (node.nodeName !== 'SECTION') { + var dsr = DU.getDataParsoid(node).dsr; + return start ? dsr[0] : dsr[1]; + } + + var offset = 0; + var c = start ? node.firstChild : node.lastChild; + while (c) { + if (!DU.isElt(c)) { + offset += c.textContent.length; + } else { + return getDSR(c, start) + (start ? -offset : offset); + } + c = start ? c.nextSibling : c.previousSibling; + } + + return -1; +} + +function resolveTemplateSectionConflicts(state) { + state.templatesToExamine.forEach(function(tplInfo) { + var s1 = tplInfo.firstSection.container; + var s2 = tplInfo.lastSection.container; + + // Find a common ancestor of s1 and s2 (could be s1) + var s2Ancestors = DU.pathToRoot(s2); + var s1Ancestors = [s1]; + var ancestor = s1; + var i; + while ((i = s2Ancestors.indexOf(ancestor)) < 0) { + ancestor = ancestor.parentNode; + s1Ancestors.push(ancestor); + } + + var n, tplDsr, dmw; + if (ancestor === s1) { + // Scenario 1: s1 is s2's ancestor + s2 = tplInfo.lastSection.container; + if (tplInfo.last.nextSibling) { + // Append the content of the section after the last node to data-mw.parts + var newTplEndOffset = getDSR(s2, false); // will succeed because it traverses non-tpl content + tplDsr = DU.getDataParsoid(tplInfo.first).dsr; + var tplEndOffset = tplDsr[1]; + dmw = DU.getDataMw(tplInfo.first); + dmw.parts.push(state.getSrc(tplEndOffset, newTplEndOffset)); + // Update DSR + tplDsr[1] = newTplEndOffset; + + // Set about attributes on all children of s2 - add span wrappers if required + var span; + for (n = tplInfo.last.nextSibling; n; n = n.nextSibling) { + if (DU.isElt(n)) { + n.setAttribute('about', tplInfo.about); + span = null; + } else { + if (!span) { + span = state.doc.createElement('span'); + span.setAttribute('about', tplInfo.about); + n.parentNode.replaceChild(span, n); + } + span.appendChild(n); + n = span; // to ensure n.nextSibling is correct + } + } + } + } else { + // Scenario 2: s1 and s2 are in different subtrees + // Find children of the common ancestor that are on the + // path from s1 -> ancestor and s2 -> ancestor + var newS1 = s1Ancestors[s1Ancestors.length - 2]; // length >= 2 since we know ancestors != s1 + var newS2 = s2Ancestors[i - 1]; // i >= 1 since we know s2 is not s1's ancestor + var newAbout = state.env.newAboutId(); // new about id for the new wrapping layer + + // Ensure that all children from newS1 and newS2 have about attrs set + for (n = newS1; n !== newS2.nextSibling; n = n.nextSibling) { + n.setAttribute('about', newAbout); + } + + // Update transclusion info + var dsr1 = getDSR(newS1, true); // will succeed because it traverses non-tpl content + var dsr2 = getDSR(newS2, false); // will succeed because it traverses non-tpl content + var tplDP = DU.getDataParsoid(tplInfo.first); + tplDsr = tplDP.dsr; + dmw = Util.clone(DU.getDataMw(tplInfo.first)); + dmw.parts.unshift(state.getSrc(dsr1, tplDsr[0])); + dmw.parts.push(state.getSrc(tplDsr[1], dsr2)); + DU.setDataMw(newS1, dmw); + newS1.setAttribute('typeof', 'mw:Transclusion'); + // Copy the template's parts-information object + // which has white-space information for formatting + // the transclusion and eliminates dirty-diffs. + DU.setDataParsoid(newS1, { pi: tplDP.pi, dsr: [ dsr1, dsr2 ] }); + } + }); +} + +function hasOnlyWSAndComments(node) { + if (!DU.isContentNode(node)) { + return true; + } + + var n = node.firstChild; + while (n) { + if (DU.isContentNode(n)) { + return false; + } + n = n.nextSibling; + } + + return true; +} + +function wrapSections(rootNode, env, options, atTopLevel) { + if (!atTopLevel || (!env.conf.parsoid.wrapSections && !env.wrapSections)) { + return; + } + + var doc = rootNode.ownerDocument; + var leadSection = { + container: doc.createElement('section'), + // lowest possible level since we don't want + // any nesting of h-tags in the lead section + level: 6, + lead: true, + }; + leadSection.container.setAttribute('data-mw-section-id', 0); + + // Global state + var state = { + env: env, + doc: doc, + rootNode: rootNode, + sectionNumber: 0, + inTemplate: false, + templatesToExamine: [], + getSrc: function(s, e) { + return this.env.page.src.substring(s,e); + }, + }; + wrapSectionsInDOM(state, leadSection, rootNode); + resolveTemplateSectionConflicts(state); + + // Insert lead sections conditionally. + // Suppress those with whitespace + comments only + var n = leadSection.container; + if (hasOnlyWSAndComments(n)) { + // comments and white-space nodes + DU.migrateChildren(n, n.parentNode, n); + } else { + rootNode.insertBefore(leadSection.container, rootNode.firstChild); + } +} + +module.exports = { + wrapSections: wrapSections, +}; diff --git a/tests/mocha/dsr.js b/tests/mocha/dsr.js index fbf8058..3bcbcab 100644 --- a/tests/mocha/dsr.js +++ b/tests/mocha/dsr.js @@ -12,7 +12,7 @@ // FIXME: MWParserEnvironment.getParserEnv and switchToConfig both require // mwApiMap to be setup. This forces us to load WMF config. Fixing this // will require some changes to ParsoidConfig and MWParserEnvironment. -var parsoidConfig = new ParsoidConfig(null, { loadWMF: true, defaultWiki: 'enwiki' }); +var parsoidConfig = new ParsoidConfig(null, { wrapSections: false, loadWMF: true, defaultWiki: 'enwiki' }); var parse = function(src, options) { return helpers.parse(parsoidConfig, src, options).then(function(ret) { return ret.doc; diff --git a/tests/mocha/heading.ids.js b/tests/mocha/heading.ids.js index 6505776..f03e3a6 100644 --- a/tests/mocha/heading.ids.js +++ b/tests/mocha/heading.ids.js @@ -11,7 +11,7 @@ // FIXME: MWParserEnvironment.getParserEnv and switchToConfig both require // mwApiMap to be setup. This forces us to load WMF config. Fixing this // will require some changes to ParsoidConfig and MWParserEnvironment. -var parsoidConfig = new ParsoidConfig(null, { loadWMF: true, defaultWiki: 'enwiki' }); +var parsoidConfig = new ParsoidConfig(null, { wrapSections: false, loadWMF: true, defaultWiki: 'enwiki' }); var parse = function(src, options) { return helpers.parse(parsoidConfig, src, options).then(function(ret) { return ret.doc; diff --git a/tests/mocha/linter.js b/tests/mocha/linter.js index ab8bb5b..af5d6d7 100644 --- a/tests/mocha/linter.js +++ b/tests/mocha/linter.js @@ -17,7 +17,7 @@ // will require some changes to ParsoidConfig and MWParserEnvironment. // Parsing the `[[file:...]]` tags below may also require running the // mock API to answer imageinfo queries. - var parsoidConfig = new ParsoidConfig(null, { defaultWiki: 'enwiki', loadWMF: true, linting: true }); + var parsoidConfig = new ParsoidConfig(null, { wrapSections: false, defaultWiki: 'enwiki', loadWMF: true, linting: true }); // Undo freezing so we can tweak it below parsoidConfig.linter = Util.clone(parsoidConfig.linter); var parseWT = function(wt, opts) { diff --git a/tests/mocha/parse.js b/tests/mocha/parse.js index 7f7f8bc..e36d133 100644 --- a/tests/mocha/parse.js +++ b/tests/mocha/parse.js @@ -18,7 +18,7 @@ // There are also specific dependencies on enwiki contents // (subpage support and the {Lowercase title}} template) // which ought to be factored out and mocked, longer-term. - var parsoidConfig = new ParsoidConfig(null, { loadWMF: true, defaultWiki: 'enwiki' }); + var parsoidConfig = new ParsoidConfig(null, { wrapSections: false, loadWMF: true, defaultWiki: 'enwiki' }); var parse = function(src, options) { return helpers.parse(parsoidConfig, src, options).then(function(ret) { return ret.doc; diff --git a/tests/mocha/regression.specs.js b/tests/mocha/regression.specs.js index 7895d77..4d94b6f 100644 --- a/tests/mocha/regression.specs.js +++ b/tests/mocha/regression.specs.js @@ -11,7 +11,7 @@ // FIXME: MWParserEnvironment.getParserEnv and switchToConfig both require // mwApiMap to be setup. This forces us to load WMF config. Fixing this // will require some changes to ParsoidConfig and MWParserEnvironment. -var parsoidConfig = new ParsoidConfig(null, { loadWMF: true, defaultWiki: 'enwiki' }); +var parsoidConfig = new ParsoidConfig(null, { wrapSections: false, loadWMF: true, defaultWiki: 'enwiki' }); var parse = function(src, options) { return helpers.parse(parsoidConfig, src, options).then(function(ret) { return ret.doc; diff --git a/tests/mocha/test.config.yaml b/tests/mocha/test.config.yaml index 0104551..15fdd2e 100644 --- a/tests/mocha/test.config.yaml +++ b/tests/mocha/test.config.yaml @@ -2,6 +2,7 @@ - conf: loadWMF: true useSelser: true + wrapSections: false # section wrappers get in the way strictAcceptCheck: true #traceFlags: # - 'wts' diff --git a/tests/parserTests-blacklist.js b/tests/parserTests-blacklist.js index 6e6cd23..955672e 100644 --- a/tests/parserTests-blacklist.js +++ b/tests/parserTests-blacklist.js @@ -1852,6 +1852,9 @@ add("selser", "Normalizations should be restricted to edited content [1,2,0,0,0]", "a\n\nhg3nd9\n\n= =\nb"); add("selser", "Normalizations should be restricted to edited content [1,2,2,3,0]", "a\n\ngerlhn\n\n1gcvetf\n\n= =\nb"); add("selser", "4a. Table cells without escapable prefixes after edits manual", "{|\n| id=\"x\" | -\n|}"); +add("selser", "Section wrapping with template-generated sections (bad nesting 1) [[1,0,4,0],1,[3,0,0]]", "= 1 =\n1ccv4ok\n\n{{echo|1=\n= 2 =\nb\n== 2.1 ==\nc\n}}\n\nd\n\n\ne"); +add("selser", "Section wrapping with template-generated sections (bad nesting 2) [0,0,[3,0,[4]]]", "= 1 =\na\n\n{{echo|1=\n== 1.2 ==\nb\n= 2 =\nc\n}}\n\nd\n\n\nbl6xpg"); +add("selser", "Section wrapping with editable lead section + div overlapping multiple sections [[4,3],[4,2,[2],0],[1,4,1,2],[4,0,1,3,0]]", "hrfp19\n\nn5dm7x\n\n1euy0ph\n\n1o3yomza\n<div style=\"border:1px solid red;\" data-foobar=\"188ni0p\">\nb\n\n== 1.1 ==\nc\n\n= 2 =\nd\n</div>o4asuo\ne\n1dh6a6e\n\n13vv0lg\n\nf\n\n== 3.1 ==\ng"); // ### DO NOT REMOVE THIS LINE ### (end of automatically-generated section) diff --git a/tests/parserTests.txt b/tests/parserTests.txt index 7cb9950..d90cad2 100644 --- a/tests/parserTests.txt +++ b/tests/parserTests.txt @@ -29709,3 +29709,403 @@ <p><a href="#Foo_bar">#Foo bar</a> </p> !! end + +## ------------------------------ +## Parsoid section-wrapping tests +## ------------------------------ +!! test +Section wrapping for well-nested sections (no leading content) +!! options +parsoid={ + "wrapSections": true +} +!! wikitext += 1 = +a + += 2 = +b + +== 2.1 == +c + +== 2.2 == +d + +=== 2.2.1 === +e + += 3 = +f +!! html/parsoid +<section data-mw-section-id="1"><h1 id="1"> 1 </h1> +<p>a</p> + +</section><section data-mw-section-id="2"><h1 id="2"> 2 </h1> +<p>b</p> + +<section data-mw-section-id="3"><h2 id="2.1"> 2.1 </h2> +<p>c</p> + +</section><section data-mw-section-id="4"><h2 id="2.2"> 2.2 </h2> +<p>d</p> + +<section data-mw-section-id="5"><h3 id="2.2.1"> 2.2.1 </h3> +<p>e</p> + +</section></section></section><section data-mw-section-id="6"><h1 id="3"> 3 </h1> +<p>f</p> + +</section> +!! end + +!! test +Section wrapping for well-nested sections (with leading content) +!! options +parsoid={ + "wrapSections": true +} +!! wikitext +Para 1. + +Para 2 with a <div>nested in it</div> + +Para 3. + += 1 = +a + += 2 = +b + +== 2.1 == +c +!! html/parsoid +<section data-mw-section-id="0"><p>Para 1.</p> + +<p>Para 2 with a </p><div>nested in it</div> + +<p>Para 3.</p> + +</section><section data-mw-section-id="1"><h1 id="1"> 1 </h1> +<p>a</p> + +</section><section data-mw-section-id="2"><h1 id="2"> 2 </h1> +<p>b</p> + +<section data-mw-section-id="3"><h2 id="2.1"> 2.1 </h2> +<p>c</p> + +</section></section> +!! end + +!! test +Section wrapping with template-generated sections (good nesting 1) +!! options +parsoid={ + "wrapSections": true +} +!! wikitext += 1 = +a + +{{echo|1= +== 1.1 == +b +}} + +== 1.2 == +c + += 2 = +d +!! html/parsoid +<section data-mw-section-id="1"><h1 id="1"> 1 </h1> +<p>a</p> + +<section data-mw-section-id="-1"><h2 about="#mwt1" typeof="mw:Transclusion" id="1.1" data-parsoid='{"dsr":[9,33,null,null],"pi":[[{"k":"1","named":true,"spc":["","","\n","\n"]}]]}' data-mw='{"parts":[{"template":{"target":{"wt":"echo","href":"./Template:Echo"},"params":{"1":{"wt":"== 1.1 ==\nb"}},"i":0}}]}'> 1.1 </h2><span about="#mwt1"> +</span><p about="#mwt1">b</p> +</section><section data-mw-section-id="3"><h2 id="1.2"> 1.2 </h2> +<p>c</p> + +</section></section><section data-mw-section-id="4"><h1 id="2"> 2 </h1> +<p>d</p></section> +!! end + +# In this example, the template scope is mildly expanded to incorporate the +# trailing newline after the transclusion since that is part of section 1.1.1 +!! test +Section wrapping with template-generated sections (good nesting 2) +!! options +parsoid={ + "wrapSections": true, + "modes": ["wt2html", "wt2wt"] +} +!! wikitext += 1 = +a + +{{echo|1= +== 1.1 == +b +=== 1.1.1 === +d +}} += 2 = +e +!! html/parsoid +<section data-mw-section-id="1"><h1 id="1"> 1 </h1> +<p>a</p> + +<section data-mw-section-id="-1"><h2 about="#mwt1" typeof="mw:Transclusion" id="1.1" data-parsoid='{"dsr":[9,50,null,null],"pi":[[{"k":"1","named":true,"spc":["","","\n","\n"]}]]}' data-mw='{"parts":[{"template":{"target":{"wt":"echo","href":"./Template:Echo"},"params":{"1":{"wt":"== 1.1 ==\nb\n=== 1.1.1 ===\nd"}},"i":0}},"\n"]}'> 1.1 </h2><span about="#mwt1"> +</span><p about="#mwt1">b</p><span about="#mwt1"> +</span><section data-mw-section-id="-1" about="#mwt1"><h3 about="#mwt1" id="1.1.1"> 1.1.1 </h3><span about="#mwt1"> +</span><p about="#mwt1">d</p><span about="#mwt1"> +</span></section></section></section><section data-mw-section-id="4" data-parsoid="{}"><h1 id="2"> 2 </h1> +<p>e</p></section> +!! end + +# In this example, the template scope is mildly expanded to incorporate the +# trailing newline after the transclusion since that is part of section 1.2.1 +!! test +Section wrapping with template-generated sections (good nesting 3) +!! options +parsoid={ + "wrapSections": true, + "modes": ["wt2html", "wt2wt"] +} +!! wikitext += 1 = +a + +{{echo|1= +x +== 1.1 == +b +==1.2== +c +===1.2.1=== +d +}} += 2 = +e +!! html/parsoid +<section data-mw-section-id="1" data-parsoid="{}"><h1 id="1"> 1 </h1> +<p>a</p> + +<p about="#mwt1" typeof="mw:Transclusion" data-parsoid='{"dsr":[9,60,0,0],"pi":[[{"k":"1","named":true,"spc":["","","\n","\n"]}]]}' data-mw='{"parts":[{"template":{"target":{"wt":"echo","href":"./Template:Echo"},"params":{"1":{"wt":"x\n== 1.1 ==\nb\n==1.2==\nc\n===1.2.1===\nd"}},"i":0}},"\n"]}'>x</p><span about="#mwt1"> +</span><section data-mw-section-id="-1" about="#mwt1"><h2 about="#mwt1" id="1.1"> 1.1 </h2><span about="#mwt1"> +</span><p about="#mwt1">b</p><span about="#mwt1"> +</span></section><section data-mw-section-id="-1" about="#mwt1"><h2 about="#mwt1" id="1.2">1.2</h2><span about="#mwt1"> +</span><p about="#mwt1">c</p><span about="#mwt1"> +</span><section data-mw-section-id="-1" about="#mwt1"><h3 about="#mwt1" id="1.2.1">1.2.1</h3><span about="#mwt1"> +</span><p about="#mwt1">d</p><span about="#mwt1"> +</span></section></section></section><section data-mw-section-id="5"><h1 id="2"> 2 </h1> +<p>e</p></section> +!! end + +# Because of section-wrapping and template-wrapping interactions, +# the scope of the template is expanded so that the template markup +# is valid in the presence of <section> tags. +!! test +Section wrapping with template-generated sections (bad nesting 1) +!! options +parsoid={ + "wrapSections": true +} +!! wikitext += 1 = +a + +{{echo|1= += 2 = +b +== 2.1 == +c +}} + +d + += 3 = +e +!! html/parsoid +<section data-mw-section-id="1"><h1 id="1"> 1 </h1> +<p>a</p> + +</section><section data-mw-section-id="-1"><h1 about="#mwt1" typeof="mw:Transclusion" id="2" data-parsoid='{"dsr":[9,45,null,null],"pi":[[{"k":"1","named":true,"spc":["","","\n","\n"]}]]}' data-mw='{"parts":[{"template":{"target":{"wt":"echo","href":"./Template:Echo"},"params":{"1":{"wt":"= 2 =\nb\n== 2.1 ==\nc"}},"i":0}},"\n\nd\n\n"]}'> 2 </h1><span about="#mwt1"> +</span><p about="#mwt1">b</p><span about="#mwt1"> +</span><section data-mw-section-id="-1" about="#mwt1"><h2 about="#mwt1" id="2.1"> 2.1 </h2><span about="#mwt1"> +</span><p about="#mwt1">c</p><span about="#mwt1"> + +</span><p about="#mwt1">d</p><span about="#mwt1"> + +</span></section></section><section data-mw-section-id="4"><h1 id="3"> 3 </h1> +<p>e</p></section> +!! end + +# Because of section-wrapping and template-wrapping interactions, +# additional template wrappers are added to <section> tags +# so that template wrapping semantics are valid whether section +# tags are retained or stripped. But, the template scope can expand +# greatly when accounting for section tags. +!! test +Section wrapping with template-generated sections (bad nesting 2) +!! options +parsoid={ + "wrapSections": true, + "modes": ["wt2html", "wt2wt"] +} +!! wikitext += 1 = +a + +{{echo|1= +== 1.2 == +b += 2 = +c +}} + +d + += 3 = +e +!! html/parsoid +<section data-mw-section-id="1" about="#mwt1" typeof="mw:Transclusion" data-mw='{"parts":["= 1 =\na\n\n",{"template":{"target":{"wt":"echo","href":"./Template:Echo"},"params":{"1":{"wt":"== 1.2 ==\nb\n= 2 =\nc"}},"i":0}},"\n\nd\n\n"]}'><h1 id="1"> 1 </h1> +<p>a</p> + +<section data-mw-section-id="-1"><h2 about="#mwt1" typeof="mw:Transclusion" id="1.2" data-mw='{"parts":[{"template":{"target":{"wt":"echo","href":"./Template:Echo"},"params":{"1":{"wt":"== 1.2 ==\nb\n= 2 =\nc"}},"i":0}}]}'> 1.2 </h2><span about="#mwt1"> +</span><p about="#mwt1">b</p><span about="#mwt1"> +</span></section></section><section data-mw-section-id="-1" about="#mwt1"><h1 about="#mwt1" id="2"> 2 </h1><span about="#mwt1"> +</span><p about="#mwt1">c</p> + +<p>d</p> +</section><section data-mw-section-id="4" data-parsoid="{}"><h1 id="3"> 3 </h1> +<p>e</p></section> +!! end + +!! test +Section wrapping with uneditable lead section + div wrapping multiple sections +!! options +parsoid={ + "wrapSections": true +} +!! wikitext +foo + +<div style="border:1px solid red;"> += 1 = +a + +== 1.1 == +b + += 2 = +c +</div> + += 3 = +d + +== 3.1 == +e +!! html/parsoid +<section data-mw-section-id="-1"><p>foo</p> + +</section><section data-mw-section-id="-2"><div style="border:1px solid red;"> +<section data-mw-section-id="1"><h1 id="1"> 1 </h1> +<p>a</p> + +<section data-mw-section-id="2"><h2 id="1.1"> 1.1 </h2> +<p>b</p> + +</section></section><section data-mw-section-id="-1"><h1 id="2"> 2 </h1> +<p>c</p> +</section></div> + +</section><section data-mw-section-id="4"><h1 id="3"> 3 </h1> +<p>d</p> + +<section data-mw-section-id="5"><h2 id="3.1"> 3.1 </h2> +<p>e</p> +</section></section> +!! end + +!! test +Section wrapping with editable lead section + div overlapping multiple sections +!! options +parsoid={ + "wrapSections": true +} +!! wikitext +foo + += 1 = +a +<div style="border:1px solid red;"> +b + +== 1.1 == +c + += 2 = +d +</div> +e + += 3 = +f + +== 3.1 == +g +!! html/parsoid +<section data-mw-section-id="0"><p>foo</p> + +</section><section data-mw-section-id="-1"><h1 id="1"> 1 </h1> +<p>a</p> +</section><section data-mw-section-id="-2"><div style="border:1px solid red;"> +<p>b</p> + +<section data-mw-section-id="2"><h2 id="1.1"> 1.1 </h2> +<p>c</p> + +</section><section data-mw-section-id="-1"><h1 id="2"> 2 </h1> +<p>d</p> +</section></div> +<p>e</p> + +</section><section data-mw-section-id="4"><h1 id="3"> 3 </h1> +<p>f</p> + +<section data-mw-section-id="5"><h2 id="3.1"> 3.1 </h2> +<p>g</p> +</section></section> +!! end + +!! test +HTML header tags should not be wrapped in section tags +!! options +parsoid={ + "wrapSections": true +} +!! wikitext +foo + +<h1>a</h1> + += b = + +<h1>c</h1> + += d = +!! html/parsoid +<section data-mw-section-id="0"><p>foo</p> + +<h1 id="a" data-parsoid='{"stx":"html"}'>a</h1> + +</section><section data-mw-section-id="1"><h1 id="b"> b </h1> + +<h1 id="c" data-parsoid='{"stx":"html"}'>c</h1> + +</section><section data-mw-section-id="2"><h1 id="d"> d </h1></section> +!! end -- To view, visit https://gerrit.wikimedia.org/r/364933 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I0f4c19f7ca4ebc88eb85c53ab145b54fde7ab455 Gerrit-PatchSet: 36 Gerrit-Project: mediawiki/services/parsoid Gerrit-Branch: master Gerrit-Owner: Subramanya Sastry <ssas...@wikimedia.org> Gerrit-Reviewer: Aklapper <aklap...@wikimedia.org> Gerrit-Reviewer: BearND <bsitzm...@wikimedia.org> Gerrit-Reviewer: C. Scott Ananian <canan...@wikimedia.org> Gerrit-Reviewer: Esanders <esand...@wikimedia.org> Gerrit-Reviewer: GWicke <gwi...@wikimedia.org> Gerrit-Reviewer: Mattflaschen <mflasc...@wikimedia.org> Gerrit-Reviewer: Santhosh <santhosh.thottin...@gmail.com> Gerrit-Reviewer: Sbailey <sbai...@wikimedia.org> Gerrit-Reviewer: Subramanya Sastry <ssas...@wikimedia.org> Gerrit-Reviewer: Tim Starling <tstarl...@wikimedia.org> Gerrit-Reviewer: jenkins-bot <> _______________________________________________ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
