Hello all, I have an interesting problem here. More of a challenge.
I have been doing text cleansing for bad characters and all. Then I have another interesting problem. Extracted a public PDF with Tika does not necessary mean you will get clean text because the original PDF might have different fonts within a section that will cause weird behaviors. If you then divide it into Senteces via OpenNLP you will then get some interesting sentences. Trying to parse those sentences then it gets worse. I am showing an example bellow and I would like to ask about solutions to it, considering the text can be noisy. I do not think that it will be easy to fix the Sentence Parser. Here is what I think on approaching it: Instead, the best way to do is to look at the sentences poorly parsed, parse them and extract the inner (S) from the parse as separate sentences. What would you suggest? Here is an example of a piece of text extracted with Tika from a public pdf. This part is what OpenNLP considered to be a sentence: ---- related research DocumentsBrief: Your Next Portal should Be An Engagement WorkplaceFebruary 3, 2014Microsoft Aims sharePoint To The CloudJanuary 27, 2014setting The Technology Foundation For Your social Business And Collaboration strategyJuly 29, 2013The Forrester wave : enterprise social Platforms, Q2 2014The 13 Providers That Matter Most And How They stack Upby rob Koplowitzwith Peter Burris and Nancy Wang2257913JUNE 5, 2014For CIos The Forrester Wave : Enterprise social Platforms, Q2 2014 2 2014, Forrester Research, Inc. Reproduction Prohibited June 5, 2014 eNTeRPRIse sOCIaL PLaTFORM MaRKeT MaTuRes aMID CONsOLIDaTION aND INTegRaTIONThe enterprise social platform is no longer in its infancy as offerings become increasingly functional. ---- It is now parsed as follows: (S (S (S (NP (VBN related) (NN research) (NNP DocumentsBrief:) (NNP Your) (NNP Next) (NNP Portal)) (VP (MD should) (VP (VB Be) (NP (NP (DT An) (NNP Engagement) (NNP WorkplaceFebruary) (CD 3,) (JJ 2014Microsoft) (NNP Aims) (NN sharePoint)) (PP (TO To) (NP (DT The) (NNP CloudJanuary))))))) (VP (VBD 27,) (S (VP (VBG 2014setting) (NP (NP (NP (DT The) (NNP Technology) (NNP Foundation)) (PP (IN For) (NP (PRP$ Your) (JJ social) (NNP Business)))) (CC And) (NP (NNP Collaboration))) (PP (RB strategyJuly) (NP (CD 29,) (CD 2013The) (NNP Forrester) (NN wave))))))) (: :) (S (VP (VB enterprise) (NP (JJ social) (NN Platforms,)) (PP (IN Q2) (NP (NP (DT 2014The) (CD 13) (NNS Providers)) (NP (NP (DT That) (NNP Matter) (JJS Most)) (SBAR (S (CC And) (SBAR (WHADVP (WRB How)) (S (NP (PRP They)) (VP (VBP stack) (PP (IN Upby) (NP (NP (NN rob)) (PP (IN Koplowitzwith) (NP (NP (NNP Peter) (NNP Burris)) (CC and) (NP (NP (NNP Nancy) (NNP Wang2257913JUNE) (CD 5,)) (PP (IN 2014For) (NP (NP (NNP CIos)) (NP (DT The) (NNP Forrester) (NNP Wave)) (: :) (S (NP (NP (NP (NP (NN Enterprise) (JJ social) (NN Platforms,)) (PP (IN Q2) (NP (CD 2014) (CD 2) (JJ 2014,) (NNP Forrester) (NNP Research,) (NNP Inc.) (NNP Reproduction) (NNP Prohibited) (NNP June) (CD 5,) (CD 2014) (NN eNTeRPRIse) (NN sOCIaL)))) (NP (NNP PLaTFORM) (NNP MaRKeT) (NNP MaTuRes) (NN aMID) (NN CONsOLIDaTION))) (PP (IN aND) (NP (DT INTegRaTIONThe) (NN enterprise) (JJ social) (NN platform)))) (VP (VP (VBZ is) (ADVP (RB no) (RB longer)) (PP (IN in) (NP (PRP$ its) (NN infancy))) (SBAR (IN as) (S (NP (NNS offerings)) (VP (VBP become) (ADVP (RB increasingly)))))) (VBG functional.))))))))))))))))))))) Notice that I have more than one (S (S (S And then I have the first correct structure as (S (NP ..... (VP..... What is the best way to deal with it?
