On Monday 20 June 2011 18:24:39 Marseld Dedgjonaj wrote: > Thank you Markus. > I tested my two examples with ParserChecker and as I see, I have to > separated issues. > > First one: When I run ParserChecker in the url that contains url of the > first example, it will parse it correctly and encode "ü". But the issue > still remains. What cause the deformation of the url to: " > http://www.mydomain.com/LAJME_GOSSIP_CATEGORY/1105130159/Article-Gisele-B > ├Æ ╞├ > Γ¼├ÆΓ¼á├óΓ¼Γ‑ó├Æ╞├óΓ¼┬á├Æ┬ó├óΓ¼a┬¼├óΓ¼~┬ó├Æ╞├Γ¼"├Æ┬ó├óΓ¼a┬¼├&┬í├Æ╞├óΓ¼┼í├Æ > Γ¼a├┬╝ndchen-eshte-ende-modelja-me-e-paguar-. aspx"? > > And the second issue: I run the ParserChecker with url that contains url of > the second example, and it will list in "outlink:" the url of the second > example. I see the page source and this url is not in the page source. Why > does nutch create this url? Is there a way to fix it?
I don't know about your document source. Perhaps you can publish a simple test page online where we can replicate this behaviour. Then please describe what you expect and what you get. > > Thank you again Markus. > Marseldi > > > -----Original Message----- > From: Markus Jelsma [mailto:[email protected]] > Sent: Monday, June 20, 2011 1:17 PM > To: [email protected] > Cc: Marseld Dedgjonaj > Subject: Re: Problem in nutch parsing. > > Hmm, Nutch should parse that URL without a problem. I've done crawls of > wikipedia and had no trouble downloading and parsing and indexing of > non-latin URL's. > > You can test parsing using bin/nutch org.apache.nutch.parse.ParserChecker > <URL> to see how it parses and how it detects outlinks. > > On Monday 20 June 2011 13:02:15 Marseld Dedgjonaj wrote: > > Hi Markus, > > My question nr.1. is, why nutch does not parse correctly the "ü" in > > url.(this is in the first example) My question nr.2. is: Where does nutch > > found this url " http://www.mydomain.com/LIGJE/601270007/11.1.1.e";. I see > > in the website where nutch supposed to found this url and I see that is > > this frase (some text "11.1.1.e" some text) inside a <span> tag. It seems > > that nutch consider it as a link. > > > > Any idea how to correct this? > > > > Any help will be appreciated. > > Marseld > > > > > > -----Original Message----- > > From: Markus Jelsma [mailto:[email protected]] > > Sent: Monday, June 20, 2011 12:43 AM > > To: [email protected] > > Cc: Marseld Dedgjonaj > > Subject: Re: Problem in nutch parsing. > > > > What is your question? If you have any, provide more details of what you > > expect, what your configuration is (url filters?) and what you're > > getting. > > > > > Hello everybody, > > > > > > I use nutch-1.2 and I use it to crawl my website. > > > > > > I see some url in fetched list that doesn’t exist in website. > > > > > > > > > > > > Examples: > > > > > > http://www.mydomain.com/LAJME_GOSSIP_CATEGORY/1105130159/Article-Gisele > > > -B ├Æ ╞├ Γ¼├ÆΓ¼á├óΓ¼Γ‑ó├Æ╞├óΓ¼┬á├Æ┬ó├óΓ¼a┬¼├óΓ¼~┬ó├Æ╞├ > > > Γ¼"├Æ┬ó├óΓ¼a┬¼├&┬í├Æ╞├óΓ¼┼í├ÆΓ¼a├┬╝ndchen-eshte-ende-modelja-me-e-pagua > > > r- . aspx" > > > > > > http://www.mydomain.com/LIGJE/601270007/11.1.1.e > > > > > > > > > > > > I think nutch is not parsing correctly in this case. > > > > > > > > > > > > Thanks in advance. > > > > > > Best Regards, > > > > > > Marseld > > > > > > > > > > > > > > > > > > > > > <p class="MsoNormal"><span style="color: rgb(31, 73, 125);">Gjeni > > > <b>Punë të Mirë</b> dhe <b>të Mirë për > > > Punë</b>... Vizitoni: <a target="_blank" > > > href="http://www.punaime.al/";>www.punaime.al</a></span></p> <p><a > > > target="_blank" href="http://www.punaime.al/";><span > > > style="text-decoration: none;"><img width="165" height="31" border="0" > > > alt="punaime" src="http://www.ikub.al/images/punaime.al_small.png"; > > > /></span></a></p> > > > > <p class="MsoNormal"><span style="color: rgb(31, 73, 125);">Gjeni > > <b>Punë të Mirë</b> dhe <b>të Mirë për > > Punë</b>... Vizitoni: <a target="_blank" > > href="http://www.punaime.al/";>www.punaime.al</a></span></p> <p><a > > target="_blank" href="http://www.punaime.al/";><span > > style="text-decoration: none;"><img width="165" height="31" border="0" > > alt="punaime" src="http://www.ikub.al/images/punaime.al_small.png"; > > /></span></a></p> -- Markus Jelsma - CTO - Openindex http://www.linkedin.com/in/markus17 050-8536620 / 06-50258350
![http://www.ikub.al/images/punaime.al_small.png"](http://www.ikub.al/images/punaime.al_small.png"){kind=link}