Author: siren Date: Sat May 27 10:01:55 2006 New Revision: 409839 URL: http://svn.apache.org/viewvc?rev=409839&view=rev Log: added some sample plugins
Added: lucene/nutch/trunk/contrib/web2/plugins/ lucene/nutch/trunk/contrib/web2/plugins/build-plugin.xml (with props) lucene/nutch/trunk/contrib/web2/plugins/build.xml (with props) lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/README.txt lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/build.xml lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/lib/ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/lib/oscache-2.1.jar (with props) lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/oscache-2.1.LICENSE.txt lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/plugin.xml lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/conf/ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/conf/tiles-defs.xml lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/java/ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/java/org/ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/java/org/apache/ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/java/org/apache/nutch/ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/java/org/apache/nutch/webapp/ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/java/org/apache/nutch/webapp/CacheManager.java lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/java/org/apache/nutch/webapp/controller/ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/java/org/apache/nutch/webapp/controller/CachingSearchController.java lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/test/ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/web/ lucene/nutch/trunk/contrib/web2/plugins/web-more/ lucene/nutch/trunk/contrib/web2/plugins/web-more/build.xml lucene/nutch/trunk/contrib/web2/plugins/web-more/lib/ lucene/nutch/trunk/contrib/web2/plugins/web-more/plugin.xml lucene/nutch/trunk/contrib/web2/plugins/web-more/src/ lucene/nutch/trunk/contrib/web2/plugins/web-more/src/conf/ lucene/nutch/trunk/contrib/web2/plugins/web-more/src/conf/tiles-defs.xml lucene/nutch/trunk/contrib/web2/plugins/web-more/src/java/ lucene/nutch/trunk/contrib/web2/plugins/web-more/src/java/org/ lucene/nutch/trunk/contrib/web2/plugins/web-more/src/java/org/apache/ lucene/nutch/trunk/contrib/web2/plugins/web-more/src/java/org/apache/nutch/ lucene/nutch/trunk/contrib/web2/plugins/web-more/src/java/org/apache/nutch/webapp/ lucene/nutch/trunk/contrib/web2/plugins/web-more/src/java/org/apache/nutch/webapp/controller/ lucene/nutch/trunk/contrib/web2/plugins/web-more/src/java/org/apache/nutch/webapp/controller/MoreController.java lucene/nutch/trunk/contrib/web2/plugins/web-more/src/test/ lucene/nutch/trunk/contrib/web2/plugins/web-more/src/web/ lucene/nutch/trunk/contrib/web2/plugins/web-more/src/web/more.jsp lucene/nutch/trunk/contrib/web2/plugins/web-resources/ lucene/nutch/trunk/contrib/web2/plugins/web-resources/build.xml lucene/nutch/trunk/contrib/web2/plugins/web-resources/lib/ lucene/nutch/trunk/contrib/web2/plugins/web-resources/plugin.xml lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/ lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/conf/ lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/conf/tiles-defs.xml lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/java/ lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/resources/ lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/resources/image.gif (with props) lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/test/ lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/web/ lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/web/footer.jsp Modified: lucene/nutch/trunk/contrib/web2/README.txt lucene/nutch/trunk/contrib/web2/build.xml Modified: lucene/nutch/trunk/contrib/web2/README.txt URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/README.txt?rev=409839&r1=409838&r2=409839&view=diff ============================================================================== --- lucene/nutch/trunk/contrib/web2/README.txt (original) +++ lucene/nutch/trunk/contrib/web2/README.txt Sat May 27 10:01:55 2006 @@ -27,25 +27,37 @@ To compile you need to fist build your nutch (core and plugins) after that run ant war to generate war. -The nutch plugins are no included in the generated war and you +The nutch plugins are not included in the generated war and you need to properly configure where your plugins are. This is achieved by editing the nutch configuration file <NUTCH_HOME>/conf/nutch-site.xml the configuration parameter you need to edit is named 'plugin.folders' +Compiling web2 plugins: + +ant compile-plugins (inside contrib/web2) + +After compiling you must enable plugins, please refer to nutch +documentation Todo: --provide some samples of ui plugins --move "more" functionality to plugin --remove table structures from html to allow more flexible css layouts --add mechanism for adding binary items (ie. images) +-Remove table structures from html to allow more flexible +css layouts Directory contents /lib - contains required additional libraties, all Licenced and maintained by ASF. - struts.jar (version 1.2.9) and libraries required by struts. + contains required additional libraries, all licenced and + maintained by ASF. files: + struts.jar (version 1.2.9) and libraries required by struts: + commons-beanutils.jar + commons-collections-3.0.jar + commons-digester.jar + + jstl.jar + standard.jar + /res contains stylesheets to transform static html pages or page snippets @@ -66,3 +78,58 @@ this file +Web ui plugin source directory structure + +/lib + contains libraries required by extension + +/src/conf + configuration files for example tiles-defs.xml + +/src/java + java source files + +/src/test + java unit test source files + +/src/web + jsp files + +/src/resources + any resources that needs to be exposed to public url space + (images, css, js...) + +/build.xml + +/plugin.xml + + +JSP Templating + +Referencing jsp files from tiles-defs: + +Referencing jsp resources inside nutch war + +absolute path, must start with /WEB-INF + + <definition name="searchPage" extends=".layout" + controllerClass="org.apache.nutch.webapp.controller.CachingSearchController"> + <put name="title" value="cached search"/> + <put name="pageBody" value="/WEB-INF/jsp/search.jsp" /> + </definition> + +Referencing jsp resources inside plugins + +absolute path, must start with /plugin/ + + + <definition name="searchPage" extends=".layout" + controllerClass="org.apache.nutch.webapp.controller.CachingSearchController"> + <put name="title" value="cached search"/> + <put name="pageBody" value="/plugin/search.jsp" /> + </definition> + + +Static resources + + Modified: lucene/nutch/trunk/contrib/web2/build.xml URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/build.xml?rev=409839&r1=409838&r2=409839&view=diff ============================================================================== --- lucene/nutch/trunk/contrib/web2/build.xml (original) +++ lucene/nutch/trunk/contrib/web2/build.xml Sat May 27 10:01:55 2006 @@ -14,7 +14,7 @@ <property name="build.test" location="${build.dir}/test" /> <property name="src.dir" location="${root}/src/main/java" /> - <property name="src.test" location="${root}/src/test" /> + <property name="src.test" location="${root}/src/main/test" /> <property name="conf.dir" location="${nutch.root}/conf" /> <property name="deploy.dir" location="target" /> @@ -23,8 +23,10 @@ <property name="web.src.dir" value="${basedir}/src/main/webapp"/> <property name="docs.dir" value="${build.dir}/${name}/"/> <property name="build.plugins" value="${nutch.root}/build/plugins"/> + <property name="build.src" value="${build.dir}/jsp-src"/> <property file="${nutch.root}/default.properties"/> + <property name="web.final.name" value="${name}-${version}" /> @@ -38,14 +40,20 @@ <fileset refid="lib.jars" /> <pathelement location="${nutch.root}/build/classes" /> <fileset dir="${nutch.root}/lib"> - <include name="servlet-api.jar" /> - <include name="hadoop-0.1.1.jar" /> - </fileset> - <fileset dir="${nutch.root}/lib/jetty-ext"> - <include name="jsp-api.jar" /> + <include name="**/*.jar" /> </fileset> </path> + <!-- the unit test classpath --> + <dirname property="plugins.classpath.dir" file="${build.plugins}"/> + <path id="test.classpath"> + <pathelement location="${build.test}" /> + <pathelement location="${conf.dir}"/> + <pathelement location="${test.src.dir}"/> + <pathelement location="${plugins.classpath.dir}"/> + <path refid="classpath"/> + </path> + <!-- Init --> @@ -82,7 +90,7 @@ </target> <!-- - generate the nutch.xml (servlet context) file + generate the nutch.xml (servlet context) file --> <target name="generate-context"> <!-- xmlcatalog definition for xslt task --> @@ -96,7 +104,11 @@ </xslt> </target> - <target name="war" depends="jar, generate-context, generate-pages"> + <target name="compile-plugins" depends="init"> + <ant dir="plugins" target="deploy" inheritAll="false"/> + </target> + + <target name="war" depends="generate-context, generate-pages, jar"> <war destfile="${build.dir}/${web.final.name}.war" webxml="src/main/webapp/WEB-INF/web.xml"> <fileset dir="${web.src.dir}" excludes="WEB-INF/web.xml"/> <zipfileset dir="${docs.src}" includes="include/*.html" /> @@ -106,7 +118,13 @@ <include name="lucene-*.jar" /> <include name="hadoop-*.jar" /> <include name="commons-lang-*.jar" /> + <include name="commons-logging-*.jar" /> + <include name="xerces-*.jar" /> </lib> + <lib dir="${nutch.root}/lib/jetty-ext"> + <include name="jasper-runtime*.jar" /> + </lib> + <fileset dir="${nutch.root}/docs" includes="img/**"/> @@ -121,10 +139,11 @@ <include name="*.jar" /> </lib> <classes dir="${conf.dir}" excludes="**/*.template" /> -<!-- no plugins + <!-- <zipfileset prefix="WEB-INF/classes/plugins" dir="${build.plugins}" > <exclude name="parse-*/**"/> <exclude name="protocol-*/**"/> + <exclude name="urlfilter-*/**"/> </zipfileset> --> </war> @@ -138,10 +157,13 @@ <copy file="${build.dir}/${web.final.name}.war" todir="${deploy.dir}" failonerror="false" /> </target> - <!-- + <taskdef classname="org.apache.jasper.JspC" name="jsp-compile" > + <classpath refid="classpath"/> + </taskdef> + <!-- Compile test code --> - <target name="compile-test" depends="compile" if="test.available"> + <target name="compile-test" depends="compile"> <javac encoding="${build.encoding}" srcdir="${src.test}" includes="**/*.java" destdir="${build.test}" debug="${javac.debug}" optimize="${javac.optimize}" target="${javac.version}" source="${javac.version}" deprecation="${javac.deprecation}"> <classpath refid="test.classpath" /> </javac> @@ -150,7 +172,7 @@ <!-- Run unit tests --> - <target name="test" depends="compile-test, deploy" if="test.available"> + <target name="test" depends="compile-test"> <junit printsummary="yes" haltonfailure="no" fork="yes" errorProperty="tests.failed" failureProperty="tests.failed"> <sysproperty key="test.data" value="${build.test}/data" /> <sysproperty key="test.input" value="${root}/data" /> Added: lucene/nutch/trunk/contrib/web2/plugins/build-plugin.xml URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/build-plugin.xml?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/build-plugin.xml (added) +++ lucene/nutch/trunk/contrib/web2/plugins/build-plugin.xml Sat May 27 10:01:55 2006 @@ -0,0 +1,238 @@ +<?xml version="1.0"?> + +<!-- Imported by plugin build.xml files to define default targets. --> + +<project> + + <property name="name" value="${ant.project.name}"/> + <property name="root" value="${basedir}"/> + + <!-- load plugin-specific properties first --> + <property file="${user.home}/${name}.build.properties" /> + <property file="${root}/build.properties" /> + + <property name="nutch.root" location="${root}/../../../.."/> + + <property name="web.root" location="../../src/main/webapp"/> + <property name="web.lib" location="../../lib"/> + <property name="src.dir" location="${root}/src/java"/> + <property name="resources.dir" location="${root}/src/resources"/> + <property name="web.dir" location="${root}/src/web"/> + <property name="src.test" location="${root}/src/test"/> + + <available file="${src.test}" type="dir" property="test.available"/> + + <property name="conf.dir" location="${nutch.root}/conf"/> + + <property name="build.dir" location="${nutch.root}/build/${name}"/> + <property name="build.src.tmp" location="${nutch.root}/build/${name}/tmp/"/> + <property name="build.web.dir" location="${build.src.tmp}/_web"/> + <property name="build.src.dir" location="${build.src.tmp}/_src"/> + <property name="build.classes" location="${build.dir}/classes"/> + <property name="build.test" location="${build.dir}/test"/> + + <property name="deploy.dir" location="${nutch.root}/build/plugins/${name}"/> + + <!-- load nutch defaults last so that they can be overridden above --> + <property file="${nutch.root}/default.properties" /> + + <path id="plugin.deps"/> + + <fileset id="lib.jars" dir="${root}" includes="lib/*.jar"/> + + <!-- the normal classpath --> + <path id="classpath"> + <pathelement location="${build.classes}"/> + <pathelement location="${nutch.root}/build/classes"/> + <fileset dir="${nutch.root}/contrib/web2/target"> + <include name="*.jar" /> + </fileset> + <fileset dir="${nutch.root}/lib/"> + <include name="**/*.jar" /> + </fileset> + <fileset dir="${web.lib}"> + <include name="*.jar" /> + </fileset> + <fileset refid="lib.jars"/> + <path refid="plugin.deps"/> + </path> + + <!-- the unit test classpath --> + <path id="test.classpath"> + <pathelement location="${build.test}" /> + <pathelement location="${nutch.root}/build/test/classes"/> + <pathelement location="${nutch.root}/src/test"/> + <pathelement location="${conf.dir}"/> + <pathelement location="${nutch.root}/build"/> + <path refid="classpath"/> + </path> + + <!-- ====================================================== --> + <!-- Stuff needed by all targets --> + <!-- ====================================================== --> + <target name="init"> + <mkdir dir="${build.dir}"/> + <mkdir dir="${build.classes}"/> + <mkdir dir="${build.test}"/> + <mkdir dir="${build.src.dir}"/> + <mkdir dir="${build.src.dir}"/> +<mkdir dir="${build.src.tmp}"/> + <antcall target="init-plugin"/> + </target> + + <!-- to be overridden by sub-projects --> + <target name="init-plugin"/> + + <!-- + ! Used to build plugin compilation dependencies + ! (to be overridden by plugins) + !--> + <target name="deps-jar"/> + + <!-- + ! Used to deploy plugin runtime dependencies + ! (to be overridden by plugins) + !--> + <target name="deps-test"/> + + <!-- ====================================================== --> + <!-- Compile the Java files --> + <!-- ====================================================== --> + <target name="compile" depends="init,deps-jar,compile-jsp"> + <echo message="Compiling plugin: ${name}"/> + <javac + encoding="${build.encoding}" + srcdir="${src.dir};${build.src.dir}" + includes="**/*.java" + destdir="${build.classes}" + debug="${javac.debug}" + optimize="${javac.optimize}" + target="${javac.version}" + source="${javac.version}" + deprecation="${javac.deprecation}"> + <classpath refid="classpath"/> + </javac> + </target> + + <target name="compile-core"> + <ant target="compile-core" inheritall="false" dir="${nutch.root}"/> + <ant target="compile"/> + </target> + + <!-- ================================================================== --> + <!-- Make plugin .jar --> + <!-- ================================================================== --> + <!-- --> + <!-- ================================================================== --> + <target name="jar" depends="compile"> + <jar + jarfile="${build.dir}/${name}.jar" + basedir="${build.classes}" + > + <zipfileset prefix="META-INF" dir="${build.src.tmp}"> + <include name="*.xml"/> + </zipfileset> + </jar> + </target> + + <target name="jar-core" depends="compile-core"> + <jar + jarfile="${build.dir}/${name}.jar" + basedir="${build.classes}" + /> + </target> + + <!-- ================================================================== --> + <!-- Deploy plugin to ${deploy.dir} --> + <!-- ================================================================== --> + <!-- --> + <!-- ================================================================== --> + <target name="deploy" depends="jar, deps-test"> + <mkdir dir="${deploy.dir}"/> + <copy file="plugin.xml" todir="${deploy.dir}" + preservelastmodified="true"/> + <copy file="${build.dir}/${name}.jar" todir="${deploy.dir}" failonerror="false"/> + <copy todir="${deploy.dir}" flatten="true"> + <fileset refid="lib.jars"/> + </copy> + </target> + + <!-- ================================================================== --> + <!-- Compile test code --> + <!-- ================================================================== --> + <target name="compile-test" depends="compile" if="test.available"> + <javac + encoding="${build.encoding}" + srcdir="${src.test}" + includes="**/*.java" + destdir="${build.test}" + debug="${javac.debug}" + optimize="${javac.optimize}" + target="${javac.version}" + source="${javac.version}" + deprecation="${javac.deprecation}"> + <classpath refid="test.classpath"/> + </javac> + </target> + + <!-- ================================================================== --> + <!-- Run unit tests --> + <!-- ================================================================== --> + <target name="test" depends="compile-test, deploy" if="test.available"> + <echo message="Testing plugin: ${name}"/> + + <junit printsummary="yes" haltonfailure="no" fork="yes" + errorProperty="tests.failed" failureProperty="tests.failed"> + <sysproperty key="test.data" value="${build.test}/data"/> + <sysproperty key="test.input" value="${root}/data"/> + <classpath refid="test.classpath"/> + <formatter type="plain" /> + <batchtest todir="${build.test}" unless="testcase"> + <fileset dir="${src.test}" + includes="**/Test*.java" excludes="**/${test.exclude}.java" /> + </batchtest> + <batchtest todir="${build.test}" if="testcase"> + <fileset dir="${src.test}" includes="**/${testcase}.java"/> + </batchtest> + </junit> + + <fail if="tests.failed">Tests failed!</fail> + + </target> + + <!-- ================================================================== --> + <!-- Clean. Delete the build files, and their directories --> + <!-- ================================================================== --> + <target name="clean"> + <delete dir="${build.dir}"/> + <delete dir="${deploy.dir}"/> + </target> + + <target name="prepare-web"> + <delete dir="${build.web.dir}"/> + <copy todir="${build.web.dir}" > + <fileset dir="${web.root}"> + <include name="**/common.jsp"/> + <include name="**/*.tld"/> + <include name="**/web.xml"/> + </fileset> + </copy> + <copy todir="${build.web.dir}"> + <fileset dir="${web.dir}"> + <include name="**/*.jsp"/> + </fileset> + </copy> + </target> + <!-- + Compile jsp -> java + --> + <target name="compile-jsp" depends="prepare-web"> + <jsp-compile + uriroot="${build.web.dir}" + outputdir="${build.src.dir}" + package="org.apache.nutch.web2.plugin" + webxml="${build.src.tmp}/jsp-servlet-mappings.xml" + /> + </target> + +</project> Propchange: lucene/nutch/trunk/contrib/web2/plugins/build-plugin.xml ------------------------------------------------------------------------------ svn:executable = * Added: lucene/nutch/trunk/contrib/web2/plugins/build.xml URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/build.xml?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/build.xml (added) +++ lucene/nutch/trunk/contrib/web2/plugins/build.xml Sat May 27 10:01:55 2006 @@ -0,0 +1,37 @@ +<?xml version="1.0"?> + +<project name="Nutch" default="deploy-core" basedir="."> + + <target name="deploy-core"> + <ant target="compile-core" inheritall="false" dir="../../../"/> + <ant target="deploy"/> + </target> + + <!-- ====================================================== --> + <!-- Build & deploy all the plugin jars. --> + <!-- ====================================================== --> + <target name="deploy"> + <ant dir="web-caching-oscache" target="deploy"/> + <ant dir="web-more" target="deploy"/> + <ant dir="web-resources" target="deploy"/> + </target> + + <!-- ====================================================== --> + <!-- Test all of the plugins. --> + <!-- ====================================================== --> + <target name="test"> + <parallel threadCount="2"> + <ant dir="web-caching-oscache" target="test"/> + </parallel> + </target> + + <!-- ====================================================== --> + <!-- Clean all of the plugins. --> + <!-- ====================================================== --> + <target name="clean"> + <ant dir="web-caching-oscache" target="clean"/> + <ant dir="web-resources" target="clean"/> + <ant dir="web-more" target="clean"/> + </target> + +</project> Propchange: lucene/nutch/trunk/contrib/web2/plugins/build.xml ------------------------------------------------------------------------------ svn:executable = * Added: lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/README.txt URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/README.txt?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/README.txt (added) +++ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/README.txt Sat May 27 10:01:55 2006 @@ -0,0 +1,3 @@ +Cache search results in on disc cache. + +This plugin is made only for demonstration purposes. Added: lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/build.xml URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/build.xml?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/build.xml (added) +++ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/build.xml Sat May 27 10:01:55 2006 @@ -0,0 +1,19 @@ +<?xml version="1.0"?> + +<project name="web-caching-oscache" default="jar-core"> + + <import file="../build-plugin.xml"/> + <property name="nutch.root" location="${root}/../../../../"/> + + <target name="init-plugin"> + <echo>Copying UI configuration</echo> + <copy todir="${build.classes}"> + <fileset dir="src/conf" includes="**/*"/> + </copy> + <echo>Copying UI templates</echo> + <copy todir="${deploy.dir}/web"> + <fileset dir="src/web" includes="**/*"/> + </copy> + </target> + +</project> Added: lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/lib/oscache-2.1.jar URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/lib/oscache-2.1.jar?rev=409839&view=auto ============================================================================== Binary file - no diff available. Propchange: lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/lib/oscache-2.1.jar ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/oscache-2.1.LICENSE.txt URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/oscache-2.1.LICENSE.txt?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/oscache-2.1.LICENSE.txt (added) +++ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/oscache-2.1.LICENSE.txt Sat May 27 10:01:55 2006 @@ -0,0 +1,50 @@ +/* ==================================================================== + * The OpenSymphony Software License, Version 1.1 + * + * (this license is derived and fully compatible with the Apache Software + * License - see http://www.apache.org/LICENSE.txt) + * + * Copyright (c) 2001-2004 The OpenSymphony Group. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * 3. The end-user documentation included with the redistribution, + * if any, must include the following acknowledgment: + * "This product includes software developed by the + * OpenSymphony Group (http://www.opensymphony.com/)." + * Alternately, this acknowledgment may appear in the software itself, + * if and wherever such third-party acknowledgments normally appear. + * + * 4. The names "OpenSymphony" and "The OpenSymphony Group" + * must not be used to endorse or promote products derived from this + * software without prior written permission. For written + * permission, please contact [EMAIL PROTECTED] . + * + * 5. Products derived from this software may not be called "OpenSymphony" + * or "OSCache", nor may "OpenSymphony" or "OSCache" appear in their + * name, without prior written permission of the OpenSymphony Group. + * + * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR IMPLIED + * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + * DISCLAIMED. IN NO EVENT SHALL THE APACHE SOFTWARE FOUNDATION OR + * ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * ==================================================================== + */ \ No newline at end of file Added: lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/plugin.xml URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/plugin.xml?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/plugin.xml (added) +++ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/plugin.xml Sat May 27 10:01:55 2006 @@ -0,0 +1,27 @@ +<?xml version="1.0" encoding="UTF-8"?> +<plugin + id="web-caching-oscache" + name="Search result caching" + version="1.0.0" + provider-name="apache.org"> + + <runtime> + <library name="web-caching-oscache.jar"> + <export name="*"/> + </library> + + <library name="oscache-2.1.jar"/> + </runtime> + + <requires> + <import plugin="nutch-extensionpoints"/> + </requires> + + <extension id="org.apache.nutch.webapp.extension.UIExtensionPoint" + name="Nutch ui extension point" + point="org.apache.nutch.webapp.extension.UIExtensionPoint"> + <implementation id="web-caching-oscache" + class="org.apache.nutch.webapp.extension.UIExtension.VoidImplementation"/> + </extension> + +</plugin> Added: lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/conf/tiles-defs.xml URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/conf/tiles-defs.xml?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/conf/tiles-defs.xml (added) +++ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/conf/tiles-defs.xml Sat May 27 10:01:55 2006 @@ -0,0 +1,10 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE tiles-definitions PUBLIC "-//Apache Software Foundation//DTD Tiles Configuration 1.1//EN" "http://jakarta.apache.org/struts/dtds/tiles-config_1_1.dtd"> +<tiles-definitions> + <!-- Search --> + <definition name="searchPage" extends="template" + controllerClass="org.apache.nutch.webapp.controller.CachingSearchController"> + <put name="title" value="search.title" /> + <put name="pageBody" value="search" /> + </definition> +</tiles-definitions> Added: lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/java/org/apache/nutch/webapp/CacheManager.java URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/java/org/apache/nutch/webapp/CacheManager.java?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/java/org/apache/nutch/webapp/CacheManager.java (added) +++ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/java/org/apache/nutch/webapp/CacheManager.java Sat May 27 10:01:55 2006 @@ -0,0 +1,66 @@ +package org.apache.nutch.webapp; + +import org.apache.hadoop.conf.Configuration; +import org.apache.nutch.webapp.common.Search; + +import com.opensymphony.oscache.base.Cache; +import com.opensymphony.oscache.base.CacheEntry; +import com.opensymphony.oscache.base.EntryRefreshPolicy; +import com.opensymphony.oscache.base.NeedsRefreshException; + +/** + * CacheManager for + */ +public class CacheManager { + + static final String CACHE_KEY="cache"; + + class NutchReferehPolicy implements EntryRefreshPolicy { + + private static final long serialVersionUID = 1L; + + public boolean needsRefresh(CacheEntry arg0) { + return false; + } + } + + EntryRefreshPolicy policy=new NutchReferehPolicy(); + + Cache cache; + + protected CacheManager(){ + cache=new Cache(true,true,false,true,"com.opensymphony.oscache.base.algorithm.UnlimitedCache",Integer.MAX_VALUE); + } + + public static CacheManager getInstance(Configuration conf){ + CacheManager cache=(CacheManager)conf.getObject(CACHE_KEY); + + if(cache==null) { + cache = new CacheManager(); + + conf.setObject(CACHE_KEY, cache); + } + return cache; + } + + /** + * Get Search object from cache + * @param id key + * @return + * @throws NeedsRefreshException + */ + public Search getSearch(String id) throws NeedsRefreshException { + return (Search) cache.getFromCache(id); + } + + /** + * Put Search object in cache + * + * @param id key + * @param search the search to cache + */ + public void putSearch(String id, Search search){ + cache.putInCache(id,search,policy); + } + +} Added: lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/java/org/apache/nutch/webapp/controller/CachingSearchController.java URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/java/org/apache/nutch/webapp/controller/CachingSearchController.java?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/java/org/apache/nutch/webapp/controller/CachingSearchController.java (added) +++ lucene/nutch/trunk/contrib/web2/plugins/web-caching-oscache/src/java/org/apache/nutch/webapp/controller/CachingSearchController.java Sat May 27 10:01:55 2006 @@ -0,0 +1,56 @@ +package org.apache.nutch.webapp.controller; + +import java.io.IOException; + +import javax.servlet.ServletContext; +import javax.servlet.ServletException; +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; + +import org.apache.nutch.webapp.CacheManager; +import org.apache.nutch.webapp.common.Search; +import org.apache.nutch.webapp.common.ServiceLocator; +import org.apache.struts.tiles.ComponentContext; + +import com.opensymphony.oscache.base.NeedsRefreshException; + +/** + * This naive search result caching implementation is just an example of + * extending the web ui. + */ +public class CachingSearchController extends SearchController { + + public void nutchPerform(ComponentContext tileContext, + HttpServletRequest request, HttpServletResponse response, + ServletContext servletContext) throws ServletException, IOException { + + Search search = null; + boolean requiresUpdate = false; + + // key used for caching + String key = request.getQueryString(); + + ServiceLocator locator = getServiceLocator(request); + + if (key != null) { + try { + search = CacheManager.getInstance(locator.getConfiguration()) + .getSearch(key); + request.setAttribute("resultInfo", search.getResultInfo()); + request.setAttribute("nutchSearch", search); + + LOG.fine("Using cached"); + } catch (NeedsRefreshException e) { + requiresUpdate = true; + LOG.fine("Cache update required"); + } + } + if (key!=null && (search == null || requiresUpdate)) { + LOG.fine("Cache miss"); + super.nutchPerform(tileContext, request, response, servletContext); + search = (Search) request.getAttribute(SearchController.REQ_ATTR_SEARCH); + CacheManager.getInstance(locator.getConfiguration()).putSearch(key, + search); + } + } +} Added: lucene/nutch/trunk/contrib/web2/plugins/web-more/build.xml URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-more/build.xml?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/web-more/build.xml (added) +++ lucene/nutch/trunk/contrib/web2/plugins/web-more/build.xml Sat May 27 10:01:55 2006 @@ -0,0 +1,19 @@ +<?xml version="1.0"?> + +<project name="web-more" default="jar-core"> + + <import file="../build-plugin.xml"/> + <property name="nutch.root" location="${root}/../../../../"/> + + <target name="init-plugin"> + <echo>Copying UI configuration</echo> + <copy todir="${build.classes}"> + <fileset dir="src/conf" includes="**/*"/> + </copy> + <echo>Copying UI templates</echo> + <copy todir="${deploy.dir}/web"> + <fileset dir="src/web" includes="**/*"/> + </copy> + </target> + +</project> Added: lucene/nutch/trunk/contrib/web2/plugins/web-more/plugin.xml URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-more/plugin.xml?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/web-more/plugin.xml (added) +++ lucene/nutch/trunk/contrib/web2/plugins/web-more/plugin.xml Sat May 27 10:01:55 2006 @@ -0,0 +1,25 @@ +<?xml version="1.0" encoding="UTF-8"?> +<plugin + id="web-more" + name="diplay information from index-more plgin" + version="1.0.0" + provider-name="apache.org"> + + <runtime> + <library name="web-more.jar"> + <export name="*"/> + </library> + </runtime> + + <requires> + <import plugin="nutch-extensionpoints"/> + </requires> + + <extension id="org.apache.nutch.webapp.extension.UIExtensionPoint" + name="Nutch ui extension point" + point="org.apache.nutch.webapp.extension.UIExtensionPoint"> + <implementation id="web-more" + class="org.apache.nutch.webapp.extension.UIExtension.VoidImplementation"/> + </extension> + +</plugin> Added: lucene/nutch/trunk/contrib/web2/plugins/web-more/src/conf/tiles-defs.xml URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-more/src/conf/tiles-defs.xml?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/web-more/src/conf/tiles-defs.xml (added) +++ lucene/nutch/trunk/contrib/web2/plugins/web-more/src/conf/tiles-defs.xml Sat May 27 10:01:55 2006 @@ -0,0 +1,12 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE tiles-definitions PUBLIC "-//Apache Software Foundation//DTD Tiles Configuration 1.1//EN" "http://jakarta.apache.org/struts/dtds/tiles-config_1_1.dtd"> +<tiles-definitions> + <definition name="more" path="/plugin/more.jsp" controllerClass="org.apache.nutch.webapp.controller.MoreController"> + <putList name="moreNames" id="moreNames"> + <add value="primaryType" type="string"/> + <add value="subType" type="string"/> + <add value="contentLength" type="string"/> + <add value="lastModified" type="string"/> + </putList> + </definition> +</tiles-definitions> Added: lucene/nutch/trunk/contrib/web2/plugins/web-more/src/java/org/apache/nutch/webapp/controller/MoreController.java URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-more/src/java/org/apache/nutch/webapp/controller/MoreController.java?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/web-more/src/java/org/apache/nutch/webapp/controller/MoreController.java (added) +++ lucene/nutch/trunk/contrib/web2/plugins/web-more/src/java/org/apache/nutch/webapp/controller/MoreController.java Sat May 27 10:01:55 2006 @@ -0,0 +1,72 @@ +/* + * Copyright 2006 The Apache Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nutch.webapp.controller; + +import java.io.IOException; +import java.util.Date; + +import javax.servlet.ServletContext; +import javax.servlet.ServletException; +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; + +import org.apache.nutch.webapp.common.SearchResultBean; +import org.apache.struts.tiles.ComponentContext; + +/** + * This is the controller used when rendering a hit. Basically what + * happens here is that we get some meta data values from HitDetails + * and stick them into request object so the jsp can display them. + * + */ +public class MoreController extends NutchController { + + public void nutchPerform(ComponentContext tileContext, + HttpServletRequest request, HttpServletResponse response, + ServletContext servletContext) throws ServletException, IOException { + + SearchResultBean hit = (SearchResultBean) request.getAttribute("hit"); + + if (hit != null) { + + // Content-Type + String primaryType = hit.getDetails().getValue("primaryType"); + String subType = hit.getDetails().getValue("subType"); + + String contentType = subType; + if (contentType == null) + contentType = primaryType; + if (contentType != null) { + request.setAttribute("contentType", contentType); + } + // Content-Length + String contentLength = hit.getDetails().getValue("contentLength"); + if (contentLength != null) { + request.setAttribute("contentLength", contentLength); + } + + // Last-Modified + String lastModified = hit.getDetails().getValue("lastModified"); + if (lastModified != null) { + long millis = new Long(lastModified).longValue(); + Date date = new Date(millis); + request.setAttribute("lastModified", date); + } + } else { + LOG.info("hit was null?"); + } + } +} Added: lucene/nutch/trunk/contrib/web2/plugins/web-more/src/web/more.jsp URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-more/src/web/more.jsp?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/web-more/src/web/more.jsp (added) +++ lucene/nutch/trunk/contrib/web2/plugins/web-more/src/web/more.jsp Sat May 27 10:01:55 2006 @@ -0,0 +1,27 @@ +<?xml version="1.0" encoding="UTF-8"?> +<jsp:root xmlns:jsp="http://java.sun.com/JSP/Page" version="1.2" + xmlns:c="/tags/jstl-c" xmlns:tiles="/tags/struts-tiles" + xmlns:fmt="/tags/jstl-fmt"> + <jsp:directive.page session="false" /> + <c:if + test="${contentType!=null || contentLength!=null ||lastModified!=null }"> + <c:if test="${contentLength!=null}"> + <fmt:message key="search.contentType"> + <fmt:param value="${contentType}" /> + </fmt:message> + </c:if> + <c:if test="${contentLength!=null}"> + <fmt:message key="search.contentLength"> + <fmt:param value="${contentLength}" /> + </fmt:message> + </c:if> + <c:if test="${contentLength!=null}"> + <fmt:message key="search.lastModified"> + <fmt:param> + <fmt:formatDate value="${lastModified}" /> + </fmt:param> + </fmt:message> + </c:if> + <br /> + </c:if> +</jsp:root> Added: lucene/nutch/trunk/contrib/web2/plugins/web-resources/build.xml URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-resources/build.xml?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/web-resources/build.xml (added) +++ lucene/nutch/trunk/contrib/web2/plugins/web-resources/build.xml Sat May 27 10:01:55 2006 @@ -0,0 +1,24 @@ +<?xml version="1.0"?> + +<project name="web-resources" default="jar-core"> + + <import file="../build-plugin.xml" /> + <property name="nutch.root" location="${root}/../../../../" /> + + <target name="init-plugin"> + <echo>Copying resources templates</echo> + <copy todir="${build.classes}/resources"> + <fileset dir="${resources.dir}" includes="**/*" /> + </copy> + <echo>Copying UI configuration</echo> + <copy todir="${build.classes}"> + <fileset dir="src/conf" includes="**/*"/> + </copy> + <echo>Copying UI templates</echo> + <copy todir="${deploy.dir}/web"> + <fileset dir="src/web" includes="**/*"/> + </copy> + </target> + + +</project> Added: lucene/nutch/trunk/contrib/web2/plugins/web-resources/plugin.xml URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-resources/plugin.xml?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/web-resources/plugin.xml (added) +++ lucene/nutch/trunk/contrib/web2/plugins/web-resources/plugin.xml Sat May 27 10:01:55 2006 @@ -0,0 +1,25 @@ +<?xml version="1.0" encoding="UTF-8"?> +<plugin + id="web-resources" + name="Resources plugin" + version="1.0.0" + provider-name="apache.org"> + + <runtime> + <library name="web-resources.jar"> + <export name="*"/> + </library> + </runtime> + + <requires> + <import plugin="nutch-extensionpoints"/> + </requires> + + <extension id="org.apache.nutch.webapp.extension.UIExtensionPoint" + name="Nutch ui extension point" + point="org.apache.nutch.webapp.extension.UIExtensionPoint"> + <implementation id="web-resources" + class="org.apache.nutch.webapp.extension.UIExtension.VoidImplementation"/> + </extension> + +</plugin> Added: lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/conf/tiles-defs.xml URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/conf/tiles-defs.xml?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/conf/tiles-defs.xml (added) +++ lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/conf/tiles-defs.xml Sat May 27 10:01:55 2006 @@ -0,0 +1,5 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE tiles-definitions PUBLIC "-//Apache Software Foundation//DTD Tiles Configuration 1.1//EN" "dtd/tiles-config_1_1.dtd"> +<tiles-definitions> + <definition name="footer" path="/plugin/footer.jsp"/> +</tiles-definitions> \ No newline at end of file Added: lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/resources/image.gif URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/resources/image.gif?rev=409839&view=auto ============================================================================== Binary file - no diff available. Propchange: lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/resources/image.gif ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/web/footer.jsp URL: http://svn.apache.org/viewvc/lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/web/footer.jsp?rev=409839&view=auto ============================================================================== --- lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/web/footer.jsp (added) +++ lucene/nutch/trunk/contrib/web2/plugins/web-resources/src/web/footer.jsp Sat May 27 10:01:55 2006 @@ -0,0 +1,14 @@ +<%@ include file="/WEB-INF/jsp/common.jsp"%> +<table bgcolor="3333ff" align="right"> + <tr> + <td bgcolor="ff9900"> + <hr /> + This footer is from resources plugin + <hr /> + <a href="opensearch?query=pelepele" /><font color="ffffff"><b>RSS</b> + </font></a></td> + </tr> +</table> +<a href="http://wiki.apache.org/nutch/FAQ"> <img border="0" + src="resources/web-resources/image.gif"> </a> +</div>