[
https://issues.apache.org/jira/browse/TIKA-2683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547119#comment-16547119
]
ASF GitHub Bot commented on TIKA-2683:
--------------------------------------
karanjeets commented on a change in pull request #243: Fix for TIKA-2683
contributed by karanjeets
URL: https://github.com/apache/tika/pull/243#discussion_r203181822
##########
File path:
tika-parsers/src/main/java/org/apache/tika/parser/html/BoilerpipeContentHandler.java
##########
@@ -230,14 +229,20 @@ public void endDocument() throws SAXException {
case CONTINUE:
// Now emit characters that are valid. Note that
boilerpipe pre-increments the character index, so
// we have to follow suit.
- for (char[] chars : element.getCharacters()) {
+ for (int i = 0; i < element.getCharacters().size();
i++) {
+ char[] chars = element.getCharacters().get(i);
curCharsIndex++;
+ boolean isValidCharacterRun =
validCharacterRuns.get(curCharsIndex);
- if (validCharacterRuns.get(curCharsIndex)) {
+ //
https://issues.apache.org/jira/projects/TIKA/issues/TIKA-2683
+ // Allow exempted characters to be written
+ if (isValidCharacterRun ||
+ (chars.length == 1 &&
whitelistCharSet.contains(chars[0]))) {
delegate.characters(chars, 0, chars.length);
//
https://issues.apache.org/jira/browse/TIKA-961
- if (!Character.isWhitespace(chars[chars.length
- 1])) {
+ if (isValidCharacterRun && i ==
element.getCharacters().size() - 1
Review comment:
It seems `element.getCharacters()` returns a section which corresponds to a
tag in the HTML web page. Inside that there are sub-sections created by char
length or other heuristics based on the BoilerPipe algorithm.
This check ensures that we are detecting/adding whitespace characters at the
end of section and not in between. The current behavior doesn't take care of
this and you may see improper new-line characters between the text (Example -
https://en.wikipedia.org/wiki/Blobfish)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Missing space and inappropriate new-line in Boilerpipe extracted text
> ---------------------------------------------------------------------
>
> Key: TIKA-2683
> URL: https://issues.apache.org/jira/browse/TIKA-2683
> Project: Tika
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.18
> Environment: Replicable everywhere in all environments
> Reporter: Karanjeet Singh
> Priority: Major
> Labels: Boilerplate_Removal, boilerpipe, parser
> Fix For: 1.19
>
>
> Boilerpipe extractor in Tika miss to capture the space and new-line character
> in HTML.
> Also, additional new-line characters are inserted in between the text.
> *Example URL* - [https://en.wikipedia.org/wiki/Blobfish]
> Missing space in "family Psychrolutidae" and additional new-line characters
> around round brackets '('
>
> Related issue reported long back -
> https://issues.apache.org/jira/browse/TIKA-961
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
