Skip to site navigation (Press enter)
packages: pldnotify.awk - use perl HTML::Tree to extract
and
f...</span></a></span> </h1> <p class="darkgray font13"> <span class="sender pipe"><a href="/search?l=pld-cvs-commit@lists.pld-linux.org&q=from:%22glen%22" rel="nofollow"><span itemprop="author" itemscope itemtype="http://schema.org/Person"><span itemprop="name">glen</span></span></a></span> <span class="date"><a href="/search?l=pld-cvs-commit@lists.pld-linux.org&q=date:20101005" rel="nofollow">Tue, 05 Oct 2010 04:02:55 -0700</a></span> </p> </div> <div itemprop="articleBody" class="msgBody"> <!--X-Body-of-Message--> <pre>Author: glen Date: Tue Oct 5 11:02:43 2010 GMT Module: packages Tag: HEAD ---- Log message: - use perl HTML::Tree to extract <A> and <IFRAME> from source</pre><pre> ---- Files affected: packages: pldnotify.awk (1.93 -> 1.94) ---- Diffs: ================================================================ Index: packages/pldnotify.awk diff -u packages/pldnotify.awk:1.93 packages/pldnotify.awk:1.94 --- packages/pldnotify.awk:1.93 Tue Oct 5 12:33:45 2010 +++ packages/pldnotify.awk Tue Oct 5 13:02:36 2010 @@ -193,6 +193,22 @@ return link } +# use perl HTML::TreeBuilder module to extract links from html +# it returns TAGNAME LINK in output which is pretty stright forward to parse in awk +function extract_links_cmd(tmpfile) { + return "perl -MHTML::TreeBuilder -e ' \ + my $content = join q//, <>; \ + my $root = new HTML::TreeBuilder; \ + $root->parse($content); \ + my $links_r = $root->extract_links(); \ + \ + for (@{$root->extract_links(qw(a iframe))}) { \ + my($link, $element, $attr, $tag) = @$_; \ + print $tag, q/ /, $link, $/; \ + } \ + ' " tmpfile +} + # get all <A HREF=..> tags from specified URL function get_links(url,filename, errno,link,oneline,retval,odp,wholeodp,lowerodp,tmpfile,cmd) { @@ -272,6 +288,43 @@ retval = ("WGET ERROR: " errno ": " wholeerr) return retval } + system("rm -f " tmpfileerr) + + urldir = url; + sub(/[^\/]+$/, "", urldir) + +if (USE_PERL) { + cmd = extract_links_cmd(tmpfile) + while (cmd | getline) { + tag = $1 + link = substr($0, length(tag) + 2) + + if (tag == "iframe") { + d("Frame: " link) + if (url !~ /\//) { + link = (urldir link) + d("Frame->: " link) + } + + if (link_seen(link)) { + continue + } + retval = (retval " " get_links(link)) + } + + if (link_seen(link)) { + continue + } + + retval = (retval " " link) + d("href(): " link) + } + close(cmd) + system("rm -f " tmpfile) + + d("Returning: [" retval "]") + return retval +} wholeodp = "" d("Reading success response...") @@ -280,12 +333,7 @@ # d("Response: " wholeodp) } d("Reponse read done...") - system("rm -f " tmpfile) - system("rm -f " tmpfileerr) - - urldir = url; - sub(/[^\/]+$/, "", urldir) while (match(wholeodp, /<([aA]|[fF][rR][aA][mM][eE])[ \t][^>]*>/) > 0) { d("Processing links...") @@ -359,7 +407,7 @@ } } - d("Returning: " retval) + d("Returning: [" retval "]") return retval } ================================================================ ---- CVS-web: <a rel="nofollow" href="http://cvs.pld-linux.org/cgi-bin/cvsweb.cgi/packages/pldnotify.awk?r1=1.93&r2=1.94&f=u">http://cvs.pld-linux.org/cgi-bin/cvsweb.cgi/packages/pldnotify.awk?r1=1.93&r2=1.94&f=u</a> _______________________________________________ pld-cvs-commit mailing list pld-cvs-commit@lists.pld-linux.org <a rel="nofollow" href="http://lists.pld-linux.org/mailman/listinfo/pld-cvs-commit">http://lists.pld-linux.org/mailman/listinfo/pld-cvs-commit</a> </pre> </div> <div class="msgButtons margintopdouble"> <ul class="overflow"> <li class="msgButtonItems"><a class="button buttonleft " accesskey="p" href="msg233124.html">Previous message</a></li> <li class="msgButtonItems textaligncenter"><a class="button" accesskey="c" href="thrd12.html#233130">View by thread</a></li> <li class="msgButtonItems textaligncenter"><a class="button" accesskey="i" href="mail10.html#233130">View by date</a></li> <li class="msgButtonItems textalignright"><a class="button buttonright " accesskey="n" href="msg233131.html">Next message</a></li> </ul> </div> <a name="tslice"></a> <div class="tSliceList margintopdouble"> <ul class="icons monospace"> <li class="icons-email"><span class="subject"><a href="msg232354.html">packages: pldnotify.awk - generic rss upgrade checker - add hudson u...</a></span> <span class="sender italic">glen</span></li> <li><ul> <li class="icons-email"><span class="subject"><a href="msg232645.html">packages: pldnotify.awk - handle cgit snapshot urls (libva.spec...</a></span> <span class="sender italic">arekm</span></li> <li><ul> <li class="icons-email"><span class="subject"><a href="msg233115.html">packages: pldnotify.awk - simplify, add downloads.sourcefor...</a></span> <span class="sender italic">glen</span></li> <li><ul> <li class="icons-email"><span class="subject"><a href="msg233116.html">packages: pldnotify.awk - gawk for strftime - error out...</a></span> <span class="sender italic">glen</span></li> <li><ul> <li class="icons-email"><span class="subject"><a href="msg233124.html">packages: pldnotify.awk - report postfix only when ...</a></span> <span class="sender italic">glen</span></li> <li><ul> <li class="icons-email tSliceCur"><span class="subject">packages: pldnotify.awk - use perl HTML::Tree ...</span> <span class="sender italic">glen</span></li> <li><ul> <li class="icons-email"><span class="subject"><a href="msg233131.html">packages: pldnotify.awk - make urls unique...</a></span> <span class="sender italic">glen</span></li> <li><ul> <li class="icons-email"><span class="subject"><a href="msg233132.html">packages: pldnotify.awk - use mawk, so...</a></span> <span class="sender italic">glen</span></li> <li class="icons-email"><span class="subject"><a href="msg234124.html">packages: pldnotify.awk - revert downl...</a></span> <span class="sender italic">arekm</span></li> <li class="icons-email"><span class="subject"><a href="msg234950.html">packages: pldnotify.awk - xulrunner ha...</a></span> <span class="sender italic">glen</span></li> </ul> </ul> </ul> </ul> </ul> </ul> </ul> </ul> </div> <div class="overflow msgActions margintopdouble"> <div class="msgReply" > <h2> Reply via email to </h2> <form method="POST" action="/mailto.php"> <input type="hidden" name="subject" value="packages: pldnotify.awk - use perl HTML::Tree to extract <A> and <IFRAME> f..."> <input type="hidden" name="msgid" value="packages_pldnotify.awk.1.94@pld-linux.org"> <input type="hidden" name="relpath" value="pld-cvs-commit@lists.pld-linux.org/msg233130.html"> <input type="submit" value=" glen "> </form> </div> </div> </div> <div class="aside" role="complementary"> <div class="logo"> <a href="/"><img src="/logo.png" width=247 height=88 alt="The Mail Archive"></a> </div> <form class="overflow" action="/search" method="get"> <input type="hidden" name="l" value="pld-cvs-commit@lists.pld-linux.org"> <label class="hidden" for="q">Search the site</label> <input class="submittext" type="text" id="q" name="q" placeholder="Search pld-cvs-commit"> <input class="submitbutton" name="submit" type="image" src="/submit.png" alt="Submit"> </form> <div class="nav margintop" id="nav" role="navigation"> <ul class="icons font16"> <li class="icons-home"><a href="/">The Mail Archive home</a></li> <li class="icons-list"><a href="/pld-cvs-commit@lists.pld-linux.org/">pld-cvs-commit - all messages</a></li> <li class="icons-about"><a href="/pld-cvs-commit@lists.pld-linux.org/info.html">pld-cvs-commit - about the list</a></li> <li class="icons-expand"><a href="/search?l=pld-cvs-commit@lists.pld-linux.org&q=subject:%22packages%5C%3A+pldnotify.awk+%5C-+use+perl+HTML%5C%3A%5C%3ATree+to+extract+%3CA%3E+and%09%3CIFRAME%3E+f...%22&o=newest&f=1" title="e" id="e">Expand</a></li> <li class="icons-prev"><a href="msg233124.html" title="p">Previous message</a></li> <li class="icons-next"><a href="msg233131.html" title="n">Next message</a></li> </ul> </div> <div class="listlogo margintopdouble"> </div> <div class="margintopdouble"> </div> </div> </div> <div class="footer" role="contentinfo"> <ul> <li><a href="/">The Mail Archive home</a></li> <li><a href="/faq.html#newlist">Add your mailing list</a></li> <li><a href="/faq.html">FAQ</a></li> <li><a href="/faq.html#support">Support</a></li> <li><a href="/faq.html#privacy">Privacy</a></li> <li class="darkgray">packages_pldnotify.awk.1.94@pld-linux.org</li> </ul> </div> </body> </html> <script>(function(){function c(){var b=a.contentDocument||a.contentWindow.document;if(b){var d=b.createElement('script');d.innerHTML="window.__CF$cv$params={r:'9c2b7a883fb32e5c',t:'MTc2OTIxNTIxNw=='};var a=document.createElement('script');a.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js';document.getElementsByTagName('head')[0].appendChild(a);";b.getElementsByTagName('head')[0].appendChild(d)}}if(document.body){var a=document.createElement('iframe');a.height=1;a.width=1;a.style.position='absolute';a.style.top=0;a.style.left=0;a.style.border='none';a.style.visibility='hidden';document.body.appendChild(a);if('loading'!==document.readyState)c();else if(window.addEventListener)document.addEventListener('DOMContentLoaded',c);else{var e=document.onreadystatechange||function(){};document.onreadystatechange=function(b){e(b);'loading'!==document.readyState&&(document.onreadystatechange=e,c())}}}})();</script>