No hassle Markus, I am fully supportive of your initiatives to upgrade to the newer Hadoop API's. This was initially the reason I am trying to get BIGTOP-284 [1] off the ground. As we all know the Hadoop ecosystem is moving at an alarming rate, it's great to have guys with the drive to keep up with it... or at least to try :0)
At least we know the failing tests can be resolved easily. Thanks Markus. [1] https://issues.apache.org/jira/browse/BIGTOP-284 On Mon, Dec 26, 2011 at 11:34 AM, Markus Jelsma <[email protected]> wrote: > There was an upgrade to 0.22.0 but i'm likely going to downgrade it again to > the latest 0.20.x version. Initial tests showed good results but 0.22 is > very unstable when fetching large lists. I've asked about this on the mapred > user list but nothing so far. > > cheers > >> Hi Guys, > >> > >> Our trunk builds have been broken since migrating to new Hadoop 0.20.2 > >> and migrating CrawlDBScanner to new MR API e.g. trunk build [1] 1698. > >> Looking to the stack trace, I'm assuming that this has to do with how > >> we are specifying the new file reads. Hopefully this shouldn't be too > >> hard to solve so maybe we can get on to it at some stage in the near > >> future. > >> > >> I just want to say Merry Christmas to EVERYONE celebrating and happy > >> holidays to everyone else who may not be. > >> > >> Best > >> > >> Lewis > >> > >> [1] https://builds.apache.org/view/M-R/view/Nutch/job/Nutch-trunk/1698/ > >> On Sat, Dec 24, 2011 at 7:36 AM, Apache Jenkins Server > >> > >> <[email protected]> wrote: > >> > See <https://builds.apache.org/job/Nutch-trunk/1702/changes> > >> > > >> > Changes: > >> > > >> > [markus] Updated pom to reflect Hadoop upgrade > >> > > >> > ------------------------------------------ > >> > [...truncated 2836 lines...] > >> > resolve-default: > >> > [ivy:resolve] :: loading settings :: file = > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/ivy/ivysettings.xml > >> > > >> > compile: > >> > [echo] Compiling plugin: urlfilter-validator > >> > [javac] > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/src/plugin/build-plugin.xml:117: warning: 'includeantruntime' > >> > was not set, defaulting to build.sysclasspath=last; set to false for > >> > repeatable builds [javac] Compiling 1 source file to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlfilter-validator/classes > >> > > >> > jar: > >> > [jar] Building jar: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlfilter-validator/urlfilter-validator.jar > >> > > >> > deps-test: > >> > > >> > deploy: > >> > [copy] Copying 1 file to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/plugins/urlfilter-validator > >> > > >> > copy-generated-lib: > >> > [copy] Copying 1 file to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/plugins/urlfilter-validator > >> > > >> > init: > >> > [mkdir] Created dir: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlmeta [mkdir] Created dir: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlmeta/classes [mkdir] Created dir: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlmeta/test [mkdir] Created dir: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/plugins/urlmeta > >> > > >> > init-plugin: > >> > > >> > deps-jar: > >> > > >> > clean-lib: > >> > > >> > resolve-default: > >> > [ivy:resolve] :: loading settings :: file = > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/ivy/ivysettings.xml > >> > > >> > compile: > >> > [echo] Compiling plugin: urlmeta > >> > [javac] > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/src/plugin/build-plugin.xml:117: warning: 'includeantruntime' > >> > was not set, defaulting to build.sysclasspath=last; set to false for > >> > repeatable builds [javac] Compiling 2 source files to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlmeta/classes > >> > > >> > jar: > >> > [jar] Building jar: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlmeta/urlmeta.jar > >> > > >> > deps-test: > >> > > >> > deploy: > >> > [copy] Copying 1 file to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/plugins/urlmeta > >> > > >> > copy-generated-lib: > >> > [copy] Copying 1 file to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/plugins/urlmeta > >> > > >> > init: > >> > [mkdir] Created dir: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlnormalizer-basic [mkdir] Created dir: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlnormalizer-basic/classes [mkdir] Created dir: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlnormalizer-basic/test [mkdir] Created dir: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/plugins/urlnormalizer-basic > >> > > >> > init-plugin: > >> > > >> > deps-jar: > >> > > >> > clean-lib: > >> > > >> > resolve-default: > >> > [ivy:resolve] :: loading settings :: file = > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/ivy/ivysettings.xml > >> > > >> > compile: > >> > [echo] Compiling plugin: urlnormalizer-basic > >> > [javac] > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/src/plugin/build-plugin.xml:117: warning: 'includeantruntime' > >> > was not set, defaulting to build.sysclasspath=last; set to false for > >> > repeatable builds [javac] Compiling 1 source file to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlnormalizer-basic/classes > >> > > >> > jar: > >> > [jar] Building jar: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlnormalizer-basic/urlnormalizer-basic.jar > >> > > >> > deps-test: > >> > > >> > deploy: > >> > [copy] Copying 1 file to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/plugins/urlnormalizer-basic > >> > > >> > copy-generated-lib: > >> > [copy] Copying 1 file to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/plugins/urlnormalizer-basic > >> > > >> > init: > >> > [mkdir] Created dir: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlnormalizer-pass [mkdir] Created dir: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlnormalizer-pass/classes [mkdir] Created dir: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlnormalizer-pass/test [mkdir] Created dir: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/plugins/urlnormalizer-pass > >> > > >> > init-plugin: > >> > > >> > deps-jar: > >> > > >> > clean-lib: > >> > > >> > resolve-default: > >> > [ivy:resolve] :: loading settings :: file = > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/ivy/ivysettings.xml > >> > > >> > compile: > >> > [echo] Compiling plugin: urlnormalizer-pass > >> > [javac] > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/src/plugin/build-plugin.xml:117: warning: 'includeantruntime' > >> > was not set, defaulting to build.sysclasspath=last; set to false for > >> > repeatable builds [javac] Compiling 1 source file to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlnormalizer-pass/classes > >> > > >> > jar: > >> > [jar] Building jar: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlnormalizer-pass/urlnormalizer-pass.jar > >> > > >> > deps-test: > >> > > >> > deploy: > >> > [copy] Copying 1 file to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/plugins/urlnormalizer-pass > >> > > >> > copy-generated-lib: > >> > [copy] Copying 1 file to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/plugins/urlnormalizer-pass [mkdir] Created dir: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlnormalizer-regex/test/data [copy] Copying 4 files to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlnormalizer-regex/test/data > >> > > >> > init: > >> > [mkdir] Created dir: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlnormalizer-regex/classes [mkdir] Created dir: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/plugins/urlnormalizer-regex > >> > > >> > init-plugin: > >> > > >> > deps-jar: > >> > > >> > clean-lib: > >> > > >> > resolve-default: > >> > [ivy:resolve] :: loading settings :: file = > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/ivy/ivysettings.xml > >> > > >> > compile: > >> > [echo] Compiling plugin: urlnormalizer-regex > >> > [javac] > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/src/plugin/build-plugin.xml:117: warning: 'includeantruntime' > >> > was not set, defaulting to build.sysclasspath=last; set to false for > >> > repeatable builds [javac] Compiling 1 source file to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlnormalizer-regex/classes [javac] Note: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > >> > nk/trunk/src/plugin/urlnormalizer-regex/src/java/org/apache/nutch/net/url > >> > normalizer/regex/RegexURLNormalizer.java uses unchecked or unsafe > >> > operations. [javac] Note: Recompile with -Xlint:unchecked for details. > >> > > >> > jar: > >> > [jar] Building jar: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/urlnormalizer-regex/urlnormalizer-regex.jar > >> > > >> > deps-test: > >> > > >> > init: > >> > > >> > init-plugin: > >> > > >> > clean-lib: > >> > > >> > resolve-default: > >> > [ivy:resolve] :: loading settings :: file = > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/ivy/ivysettings.xml > >> > > >> > compile: > >> > > >> > jar: > >> > > >> > deps-test: > >> > > >> > deploy: > >> > > >> > copy-generated-lib: > >> > > >> > deploy: > >> > [copy] Copying 1 file to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/plugins/urlnormalizer-regex > >> > > >> > copy-generated-lib: > >> > [copy] Copying 1 file to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/plugins/urlnormalizer-regex > >> > > >> > compile: > >> > > >> > job: > >> > [jar] Building jar: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/nutch-2011-12-24_07-29-25.job > >> > > >> > resolve-test: > >> > [taskdef] Could not load definitions from resource > >> > org/sonar/ant/antlib.xml. It could not be found. > >> > > >> > copy-libs: > >> > > >> > compile-core-test: > >> > [javac] > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build.xml:320: warning: 'includeantruntime' was not set, > >> > defaulting to build.sysclasspath=last; set to false for repeatable > >> > builds [javac] Compiling 32 source files to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/test/classes [javac] Note: Some input files use or > >> > override a deprecated API. [javac] Note: Recompile with > >> > -Xlint:deprecation for details. > >> > [javac] Note: Some input files use unchecked or unsafe operations. > >> > [javac] Note: Recompile with -Xlint:unchecked for details. > >> > > >> > test-core: > >> > [mkdir] Created dir: > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/test/data [copy] Copying 90 files to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/test/data [copy] Copying 1 file to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/test/classes [copy] Copying 1 file to > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-tru > >> > nk/trunk/build/test/classes [junit] Running > >> > org.apache.nutch.crawl.TestCrawlDbMerger > >> > [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 5.721 sec > >> > [junit] Test org.apache.nutch.crawl.TestCrawlDbMerger FAILED > >> > [junit] Running org.apache.nutch.crawl.TestGenerator > >> > [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 59.082 >> > sec > >> > [junit] Running org.apache.nutch.crawl.TestInjector > >> > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 13.243 >> > sec > >> > [junit] Running org.apache.nutch.crawl.TestLinkDbMerger > >> > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.266 sec > >> > [junit] Running org.apache.nutch.crawl.TestSignatureFactory > >> > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.449 sec > >> > [junit] Running org.apache.nutch.fetcher.TestFetcher > >> > [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 21.793 >> > sec > >> > [junit] Running org.apache.nutch.indexer.TestIndexingFilters > >> > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.133 sec > >> > [junit] Running org.apache.nutch.metadata.TestMetadata > >> > [junit] Tests run: 10, Failures: 0, Errors: 0, Time elapsed: 0.067 >> > sec > >> > [junit] Running org.apache.nutch.metadata.TestSpellCheckedMetadata > >> > [junit] Tests run: 11, Failures: 0, Errors: 0, Time elapsed: 3.342 >> > sec > >> > [junit] Running org.apache.nutch.net.TestURLFilters > >> > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.406 sec > >> > [junit] Running org.apache.nutch.net.TestURLNormalizers > >> > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.643 sec > >> > [junit] Running org.apache.nutch.parse.TestOutlinkExtractor > >> > [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.062 sec > >> > [junit] Running org.apache.nutch.parse.TestParseData > >> > [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.044 sec > >> > [junit] Running org.apache.nutch.parse.TestParseText > >> > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.068 sec > >> > [junit] Running org.apache.nutch.parse.TestParserFactory > >> > [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 1.014 sec > >> > [junit] Running org.apache.nutch.plugin.TestPluginSystem > >> > [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 1.523 sec > >> > [junit] Running org.apache.nutch.protocol.TestContent > >> > [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.565 sec > >> > [junit] Running org.apache.nutch.protocol.TestProtocolFactory > >> > [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.889 sec > >> > [junit] Running org.apache.nutch.segment.TestSegmentMerger > >> > [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 184.385 > >> > sec [junit] Test org.apache.nutch.segment.TestSegmentMerger FAILED > >> > [junit] Running org.apache.nutch.util.TestEncodingDetector > >> > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.555 sec > >> > [junit] Running org.apache.nutch.util.TestGZIPUtils > >> > [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 1.107 sec > >> > [junit] Running org.apache.nutch.util.TestNodeWalker > >> > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.906 sec > >> > [junit] Running org.apache.nutch.util.TestPrefixStringMatcher > >> > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.14 sec > >> > [junit] Running org.apache.nutch.util.TestStringUtil > >> > [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.056 sec > >> > [junit] Running org.apache.nutch.util.TestSuffixStringMatcher > >> > [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.108 sec > >> > [junit] Running org.apache.nutch.util.TestURLUtil > >> > [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 0.412 sec > >> > > >> > BUILD FAILED > >> > >> > /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Nutch-trun > >> > k/trunk/build.xml:395: Tests failed! > >> > > >> > Total time: 6 minutes 22 seconds > >> > Build step 'Execute shell' marked build as failure > >> > Recording test results > >> > Publishing Javadoc -- Lewis
