Skip to site navigation (Press enter)
Update - added
and
(Re: [Nutch-dev] Patch to recognize <area ..> tags.)</span></a></span> </h1> <p class="darkgray font13"> <span class="sender pipe"><a href="/search?l=nutch-developers@lists.sourceforge.net&q=from:%22Andrzej+Bialecki%22" rel="nofollow"><span itemprop="author" itemscope itemtype="http://schema.org/Person"><span itemprop="name">Andrzej Bialecki</span></span></a></span> <span class="date"><a href="/search?l=nutch-developers@lists.sourceforge.net&q=date:20040521" rel="nofollow">Fri, 21 May 2004 15:54:00 -0700</a></span> </p> </div> <div itemprop="articleBody" class="msgBody"> <!--X-Body-of-Message--> <pre style="margin: 0em;">Doug Cutting wrote:</pre><br> <pre style="margin: 0em;">Hi,</pre><br> <tt>Here's an updated patch, which cleans up the logic and adds some extensibility. Currently it supports the references in the following HTML elements: <a>, <area>, <frame> and <iframe>. It's easy to add more elements, but I couldn't think of any others...</tt><br> <br> <pre style="margin: 0em;">Andrzej.</pre><br> <blockquote style="border-left: #5555EE solid 0.2em; margin: 0em; padding-left: 0.85em"><pre style="margin: 0em;">I just applied this patch. Thanks!</pre><br> <pre style="margin: 0em;">Doug</pre><br> <pre style="margin: 0em;">Andrzej Bialecki wrote:</pre><br> <blockquote style="border-left: #5555EE solid 0.2em; margin: 0em; padding-left: 0.85em"><pre style="margin: 0em;">Hi,</pre><br> <tt>The attached patch makes a world of difference in my case. I'm trying to index some graphics-rich websites, and many links are hidden within image maps. This patch allows us to collect and traverse links contained in <area href="..."> elements.</tt><br> <br> <tt>Enjoy! </tt></blockquote><pre style="margin: 0em;"><br></pre><br> <tt><br>-------------------------------------------------------<br> This SF.Net email is sponsored by: Oracle 10g<br> Get certified on the hottest thing ever to hit the market... Oracle 10g. Take an Oracle 10g class now, and we'll give you the exam FREE.<br> <a href="http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click">http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click</a><br> _______________________________________________<br> Nutch-developers mailing list<br> [EMAIL PROTECTED]<br> <a href="https://lists.sourceforge.net/lists/listinfo/nutch-developers">https://lists.sourceforge.net/lists/listinfo/nutch-developers</a></tt><br> <br> <br> </blockquote><pre style="margin: 0em;"><br></pre><br> <pre style="margin: 0em;">-- Best regards, Andrzej Bialecki</pre><br> <pre style="margin: 0em;">------------------------------------------------- Software Architect, System Integration Specialist CEN/ISSS EC Workshop, ECIMF project chair EU FP6 E-Commerce Expert/Evaluator ------------------------------------------------- FreeBSD developer (<a href="http://www.freebsd.org">http://www.freebsd.org</a>)</pre><br> <pre>Index: DOMContentUtils.java =================================================================== RCS file: /cvsroot/nutch/nutch/src/java/net/nutch/util/DOMContentUtils.java,v retrieving revision 1.4 diff -b -d -u -r1.4 DOMContentUtils.java --- DOMContentUtils.java 20 May 2004 19:19:29 -0000 1.4 +++ DOMContentUtils.java 21 May 2004 22:02:47 -0000 @@ -3,6 +3,7 @@ import java.net.URL; import java.net.MalformedURLException; import java.util.ArrayList; +import java.util.HashMap; import net.nutch.fetcher.Outlink; @@ -17,6 +18,25 @@ */ public class DOMContentUtils { + public static class LinkParams { + public String attrName; + public int childLen; + + public LinkParams(String attrName, int childLen) { + this.attrName = attrName; + this.childLen = childLen; + } + } + + public static HashMap linkParams = new HashMap(); + + static { + linkParams.put("a", new LinkParams("href", 1)); + linkParams.put("area", new LinkParams("href", 0)); + linkParams.put("frame", new LinkParams("src", 0)); + linkParams.put("iframe", new LinkParams("src", 0)); + } + /** * This method takes a [EMAIL PROTECTED] StringBuffer} and a DOM [EMAIL PROTECTED] Node}, * and will append all the content text found beneath the DOM node to @@ -128,13 +148,11 @@ // this only covers a few cases of empty links that are symptomatic // of nekohtml's DOM-fixup process... private static boolean shouldThrowAwayLink(Node node, NodeList children, - int childLen) { - if (node.getNodeName().equalsIgnoreCase("area")) { - return false; - } + int childLen, LinkParams params) { if (childLen == 0) { // this has no inner structure - return true; + if (params.childLen == 0) return false; + else return true; } else if ((childLen == 1) && (children.item(0).getNodeType() == Node.ELEMENT_NODE) && ("a".equalsIgnoreCase(children.item(0).getNodeName()))) { @@ -204,10 +222,10 @@ childLen= children.getLength(); if (node.getNodeType() == Node.ELEMENT_NODE) { - if ("a".equalsIgnoreCase(node.getNodeName()) || - "area".equalsIgnoreCase(node.getNodeName())) { + LinkParams params = (LinkParams)linkParams.get(node.getNodeName().toLowerCase()); + if (params != null) { - if (shouldThrowAwayLink(node, children, childLen)) { + if (shouldThrowAwayLink(node, children, childLen, params)) { // this has no inner structure or just a single nested // anchor-- toss it! } else { @@ -218,7 +236,7 @@ NamedNodeMap attrs = node.getAttributes(); String target= null; for (int i= 0; i < attrs.getLength(); i++ ) { - if ("href".equalsIgnoreCase(attrs.item(i).getNodeName())) { + if (params.attrName.equalsIgnoreCase(attrs.item(i).getNodeName())) { target= attrs.item(i).getNodeValue(); break; } </pre> </div> <div class="msgButtons margintopdouble"> <ul class="overflow"> <li class="msgButtonItems"><a class="button buttonleft " accesskey="p" href="msg00721.html">Previous message</a></li> <li class="msgButtonItems textaligncenter"><a class="button" accesskey="c" href="thrd11.html#00724">View by thread</a></li> <li class="msgButtonItems textaligncenter"><a class="button" accesskey="i" href="mail11.html#00724">View by date</a></li> <li class="msgButtonItems textalignright"><a class="button buttonright " accesskey="n" href="msg00674.html">Next message</a></li> </ul> </div> <a name="tslice"></a> <div class="tSliceList margintopdouble"> <ul class="icons monospace"> <li class="icons-email"><span class="subject"><a href="msg00670.html">[Nutch-dev] Patch to recognize <area ..> tags.</a></span> <span class="sender italic">Andrzej Bialecki</span></li> <li><ul> <li class="icons-email"><span class="subject"><a href="msg00707.html">Re: [Nutch-dev] Patch to recognize <area ..> t...</a></span> <span class="sender italic">Doug Cutting</span></li> <li><ul> <li class="icons-email"><span class="subject"><a href="msg00721.html">Javascript parsing (Re: [Nutch-dev] Patch to rec...</a></span> <span class="sender italic">Andrzej Bialecki</span></li> <li class="icons-email tSliceCur"><span class="subject"></span> <span class="sender italic">Andrzej Bialecki</span></li> </ul> </ul> </ul> </div> <div class="overflow msgActions margintopdouble"> <div class="msgReply" > <h2> Reply via email to </h2> <form method="POST" action="/mailto.php"> <input type="hidden" name="subject" value="Update - added <frame> and <iframe> (Re: [Nutch-dev] Patch to recognize <area ..> tags.)"> <input type="hidden" name="msgid" value="40AE8822.1080902@getopt.org"> <input type="hidden" name="relpath" value="nutch-developers@lists.sourceforge.net/msg00724.html"> <input type="submit" value=" Andrzej Bialecki "> </form> </div> </div> </div> <div class="aside" role="complementary"> <div class="logo"> <a href="/"><img src="/logo.png" width=247 height=88 alt="The Mail Archive"></a> </div> <form class="overflow" action="/search" method="get"> <input type="hidden" name="l" value="nutch-developers@lists.sourceforge.net"> <label class="hidden" for="q">Search the site</label> <input class="submittext" type="text" id="q" name="q" placeholder="Search nutch-developers"> <input class="submitbutton" name="submit" type="image" src="/submit.png" alt="Submit"> </form> <div class="nav margintop" id="nav" role="navigation"> <ul class="icons font16"> <li class="icons-home"><a href="/">The Mail Archive home</a></li> <li class="icons-list"><a href="/nutch-developers@lists.sourceforge.net/">nutch-developers - all messages</a></li> <li class="icons-about"><a href="/nutch-developers@lists.sourceforge.net/info.html">nutch-developers - about the list</a></li> <li class="icons-expand"><a href="/search?l=nutch-developers@lists.sourceforge.net&q=subject:%22Update+%5C-+added+%3Cframe%3E+and+%3Ciframe%3E+%5C%28Re%5C%3A+%5C%5BNutch%5C-dev%5C%5D+Patch+to+recognize+%3Carea+..%3E+tags.%5C%29%22&o=newest&f=1" title="e" id="e">Expand</a></li> <li class="icons-prev"><a href="msg00721.html" title="p">Previous message</a></li> <li class="icons-next"><a href="msg00674.html" title="n">Next message</a></li> </ul> </div> <div class="listlogo margintopdouble"> </div> <div class="margintopdouble"> </div> </div> </div> <div class="footer" role="contentinfo"> <ul> <li><a href="/">The Mail Archive home</a></li> <li><a href="/faq.html#newlist">Add your mailing list</a></li> <li><a href="/faq.html">FAQ</a></li> <li><a href="/faq.html#support">Support</a></li> <li><a href="/faq.html#privacy">Privacy</a></li> <li class="darkgray">40AE8822.1080902@getopt.org</li> </ul> </div> </body> </html> <script>(function(){function c(){var b=a.contentDocument||a.contentWindow.document;if(b){var d=b.createElement('script');d.innerHTML="window.__CF$cv$params={r:'9ec44dc75fff0acb',t:'MTc3NjE4NjQxNQ=='};var a=document.createElement('script');a.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js';document.getElementsByTagName('head')[0].appendChild(a);";b.getElementsByTagName('head')[0].appendChild(d)}}if(document.body){var a=document.createElement('iframe');a.height=1;a.width=1;a.style.position='absolute';a.style.top=0;a.style.left=0;a.style.border='none';a.style.visibility='hidden';document.body.appendChild(a);if('loading'!==document.readyState)c();else if(window.addEventListener)document.addEventListener('DOMContentLoaded',c);else{var e=document.onreadystatechange||function(){};document.onreadystatechange=function(b){e(b);'loading'!==document.readyState&&(document.onreadystatechange=e,c())}}}})();</script>