[
https://issues.apache.org/jira/browse/TIKA-2683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545856#comment-16545856
]
ASF GitHub Bot commented on TIKA-2683:
--------------------------------------
kkrugler commented on a change in pull request #243: Fix for TIKA-2683
contributed by karanjeets
URL: https://github.com/apache/tika/pull/243#discussion_r202854453
##########
File path:
tika-parsers/src/main/java/org/apache/tika/parser/html/BoilerpipeContentHandler.java
##########
@@ -230,14 +229,20 @@ public void endDocument() throws SAXException {
case CONTINUE:
// Now emit characters that are valid. Note that
boilerpipe pre-increments the character index, so
// we have to follow suit.
- for (char[] chars : element.getCharacters()) {
+ for (int i = 0; i < element.getCharacters().size();
i++) {
+ char[] chars = element.getCharacters().get(i);
curCharsIndex++;
+ boolean isValidCharacterRun =
validCharacterRuns.get(curCharsIndex);
- if (validCharacterRuns.get(curCharsIndex)) {
+ //
https://issues.apache.org/jira/projects/TIKA/issues/TIKA-2683
+ // Allow exempted characters to be written
+ if (isValidCharacterRun ||
+ (chars.length == 1 &&
whitelistCharSet.contains(chars[0]))) {
delegate.characters(chars, 0, chars.length);
//
https://issues.apache.org/jira/browse/TIKA-961
- if (!Character.isWhitespace(chars[chars.length
- 1])) {
+ if (isValidCharacterRun && i ==
element.getCharacters().size() - 1
Review comment:
Could you explain why we now want to check for `i ==
element.getCharacters().size() - 1`, where that wasn't previously done by the
patch for TIKA-961?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Missing space and inappropriate new-line in Boilerpipe extracted text
> ---------------------------------------------------------------------
>
> Key: TIKA-2683
> URL: https://issues.apache.org/jira/browse/TIKA-2683
> Project: Tika
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.18
> Environment: Replicable everywhere in all environments
> Reporter: Karanjeet Singh
> Priority: Major
> Labels: Boilerplate_Removal, boilerpipe, parser
> Fix For: 1.19
>
>
> Boilerpipe extractor in Tika miss to capture the space and new-line character
> in HTML.
> Also, additional new-line characters are inserted in between the text.
> *Example URL* - [https://en.wikipedia.org/wiki/Blobfish]
> Missing space in "family Psychrolutidae" and additional new-line characters
> around round brackets '('
>
> Related issue reported long back -
> https://issues.apache.org/jira/browse/TIKA-961
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
