[
https://issues.apache.org/jira/browse/LUCENE-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14074743#comment-14074743
]
David Smiley commented on LUCENE-5740:
--------------------------------------
bq. David, the other thing I worry about is: a fully fledged version of this
would allow finer-grained specification of tags, along the lines of XPath, but
that would be a much much bigger task... I don't think such a goal should hold
up what you're thinking about.
Yeah I thought of that and agree it's not worth worrying about right now. My
initial use of this will strip generated HTML that is already fairly clean, and
will strip these tags purely by element name. I have no need/plans for more
complicated matching.
bq. Don't forget that HTMLStripCharFilter must be able to handle (i.e. not
throw an error, and maximize useful extracted content) non-well-formed content.
Assuming you'll see closing tags could be a problem here; some HTML doesn't
have these in some cases.
If a tag that is to be stripped opens, then I propose the next close tag at the
same level (whatever it's name may be) is where the strip ends:
{noformat}<body> bodyStart <p> paraStart <foo> <b>bold</b> paraEnd </p>
bodyEnd</body>{noformat}
Notice there is no {{</foo>}}. Stripping tag foo would yield only the text
tokens bodyStart, paraStart, and bodyEnd. I think it's not realistic to expect
better than that, not to mention that this issue is optional and would come
with disclaimers on this matter.
bq. It might be better to just track nested tags of the same type as the
current tag being stripped, rather than all tags
I don't think that adds any value (at least I don't see it yet), and it hurts
the bad-html case like the "foo" example above. In that same example, only
same-name tags would mean that bodyEnd would not get emitted. Right?
> Add stripContentOfTags option to HTMLStripCharFilter
> ----------------------------------------------------
>
> Key: LUCENE-5740
> URL: https://issues.apache.org/jira/browse/LUCENE-5740
> Project: Lucene - Core
> Issue Type: New Feature
> Components: modules/analysis
> Reporter: David Smiley
> Assignee: David Smiley
>
> HTMLStripCharFilter should have an option to strip out the sub-content of
> certain elements. It already does this for SCRIPT & STYLE but it should be
> configurable to add more. I don't want certain elements to have their
> contents to be searchable.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
