Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-Crawler changes)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r//#review6280 --- trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/MetExtractorProductCrawler.java https://reviews.apache.org/r//#comment13623 Nope, it's backwards compatible... This exception is only thrown if you specify a naming convention ID which does not exist... If you don't specify the ID... which is what the peeps don't specify... Then the code will never reach this IF statement... The IF statement it is contained in will be false - brian On 2012-03-22 06:09:52, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r// --- (Updated 2012-03-22 06:09:52) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- - Introduced NamingConvention support, which for MetExtractorProductCrawler now takes an ID for the NamingConvention to use and AutoDetectProductCrawler has a new element namingConvention class=/ in MimeExtractorRepo - Also cleaned up handleFile(File)... documented better and is now public and returns the IngestResult for what happened when called. This addresses bug OODT-426. https://issues.apache.org/jira/browse/OODT-426 Diffs - trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/AutoDetectProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/MetExtractorProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/ProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/StdProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigMetKeys.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigReader.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorRepo.java 1302790 trunk/crawler/src/main/resources/cmd-line-options.xml 1302790 trunk/crawler/src/main/resources/crawler-config.xml 1302790 trunk/crawler/src/main/resources/examples/mime-extractor-map.xml 1302790 trunk/crawler/src/main/resources/naming-beans.xml PRE-CREATION Diff: https://reviews.apache.org/r//diff Testing --- Still need to unit-test up cas-crawler Thanks, brian
Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-Crawler changes)
On 2012-03-22 07:15:57, Chris Mattmann wrote: trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigReader.java, line 90 https://reviews.apache.org/r//diff/1/?file=94495#file94495line90 does this make it so that Exception is never thrown and that this the default? This just allows you to specify a default... See other comment about exception - brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r//#review6214 --- On 2012-03-22 06:09:52, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r// --- (Updated 2012-03-22 06:09:52) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- - Introduced NamingConvention support, which for MetExtractorProductCrawler now takes an ID for the NamingConvention to use and AutoDetectProductCrawler has a new element namingConvention class=/ in MimeExtractorRepo - Also cleaned up handleFile(File)... documented better and is now public and returns the IngestResult for what happened when called. This addresses bug OODT-426. https://issues.apache.org/jira/browse/OODT-426 Diffs - trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/AutoDetectProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/MetExtractorProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/ProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/StdProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigMetKeys.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigReader.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorRepo.java 1302790 trunk/crawler/src/main/resources/cmd-line-options.xml 1302790 trunk/crawler/src/main/resources/crawler-config.xml 1302790 trunk/crawler/src/main/resources/examples/mime-extractor-map.xml 1302790 trunk/crawler/src/main/resources/naming-beans.xml PRE-CREATION Diff: https://reviews.apache.org/r//diff Testing --- Still need to unit-test up cas-crawler Thanks, brian
Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-Crawler changes)
On 2012-03-26 15:50:17, Chris Mattmann wrote: trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/ProductCrawler.java, line 326 https://reviews.apache.org/r//diff/2/?file=95394#file95394line326 Are all of these @VisibleForTesting coupling our test system too much with the code? Just wondering... I just changing the visibility on these methods from private to package level... then i notated them with the @VisibleForTesting to then make this clear to other developers that the only reason for these methods being package level is to make them unit-test-able On 2012-03-26 15:50:17, Chris Mattmann wrote: trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/StdProductCrawler.java, line 71 https://reviews.apache.org/r//diff/2/?file=95395#file95395line71 Should we augment the ProductCrawler super class to declare this function as an abstract method since all sub class crawlers implement it? it is... this method does need a @Override above it to make it clear... i'll add this - brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r//#review6349 --- On 2012-03-25 01:55:32, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r// --- (Updated 2012-03-25 01:55:32) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- - Introduced NamingConvention support, which for MetExtractorProductCrawler now takes an ID for the NamingConvention to use and AutoDetectProductCrawler has a new element namingConvention class=/ in MimeExtractorRepo - Also cleaned up handleFile(File)... documented better and is now public and returns the IngestResult for what happened when called. This addresses bug OODT-426. https://issues.apache.org/jira/browse/OODT-426 Diffs - trunk/crawler/src/main/resources/naming-beans.xml PRE-CREATION trunk/crawler/src/test/org/apache/oodt/cas/crawl/TestProductCrawler.java PRE-CREATION trunk/crawler/src/main/resources/cmd-line-options.xml 1302790 trunk/crawler/src/main/resources/crawler-config.xml 1302790 trunk/crawler/src/main/resources/examples/mime-extractor-map.xml 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorRepo.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/StdProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigMetKeys.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigReader.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/ProductCrawler.java 1302790 trunk/crawler/pom.xml 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/AutoDetectProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/MetExtractorProductCrawler.java 1302790 Diff: https://reviews.apache.org/r//diff Testing --- Still need to unit-test up cas-crawler Thanks, brian
Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-Crawler changes)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r// --- (Updated 2012-03-27 00:47:30.189828) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Changes --- - Updated unit-tests -- ProductCrawler has a unit-test for each possible path that can be taken through handleFile(File) and verifies the appropriate methods where called Summary --- - Introduced NamingConvention support, which for MetExtractorProductCrawler now takes an ID for the NamingConvention to use and AutoDetectProductCrawler has a new element namingConvention class=/ in MimeExtractorRepo - Also cleaned up handleFile(File)... documented better and is now public and returns the IngestResult for what happened when called. This addresses bug OODT-426. https://issues.apache.org/jira/browse/OODT-426 Diffs (updated) - trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/ProductCrawler.java 1305657 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/MetExtractorProductCrawler.java 1305657 trunk/crawler/pom.xml 1305657 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/AutoDetectProductCrawler.java 1305657 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/StdProductCrawler.java 1305657 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MetExtractorSpec.java 1305657 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigMetKeys.java 1305657 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigReader.java 1305657 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorRepo.java 1305657 trunk/crawler/src/main/resources/cmd-line-options.xml 1305657 trunk/crawler/src/main/resources/crawler-config.xml 1305657 trunk/crawler/src/main/resources/examples/mime-extractor-map.xml 1305657 trunk/crawler/src/main/resources/naming-beans.xml PRE-CREATION trunk/crawler/src/test/org/apache/oodt/cas/crawl/StateAwareProductCrawler.java PRE-CREATION trunk/crawler/src/test/org/apache/oodt/cas/crawl/TestProductCrawler.java PRE-CREATION trunk/crawler/src/test/org/apache/oodt/cas/crawl/typedetection/TestMimeExtractorConfigReader.java PRE-CREATION Diff: https://reviews.apache.org/r//diff Testing (updated) --- Wrote several unit-tests Thanks, brian
Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-Crawler changes)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r// --- (Updated 2012-03-25 01:55:32.563950) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Changes --- with some unit-tests... few more to go Summary --- - Introduced NamingConvention support, which for MetExtractorProductCrawler now takes an ID for the NamingConvention to use and AutoDetectProductCrawler has a new element namingConvention class=/ in MimeExtractorRepo - Also cleaned up handleFile(File)... documented better and is now public and returns the IngestResult for what happened when called. This addresses bug OODT-426. https://issues.apache.org/jira/browse/OODT-426 Diffs (updated) - trunk/crawler/src/main/resources/naming-beans.xml PRE-CREATION trunk/crawler/src/test/org/apache/oodt/cas/crawl/TestProductCrawler.java PRE-CREATION trunk/crawler/src/main/resources/cmd-line-options.xml 1302790 trunk/crawler/src/main/resources/crawler-config.xml 1302790 trunk/crawler/src/main/resources/examples/mime-extractor-map.xml 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorRepo.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/StdProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigMetKeys.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigReader.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/ProductCrawler.java 1302790 trunk/crawler/pom.xml 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/AutoDetectProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/MetExtractorProductCrawler.java 1302790 Diff: https://reviews.apache.org/r//diff Testing --- Still need to unit-test up cas-crawler Thanks, brian
Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-Crawler changes)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r//#review6214 --- trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/MetExtractorProductCrawler.java https://reviews.apache.org/r//#comment13396 will this cause back incompat changes for peeps with default crawl scripts that use MetExtractorProductCrawler? If so, can we default it to something to appease that? trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigReader.java https://reviews.apache.org/r//#comment13397 good job adding this! trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigReader.java https://reviews.apache.org/r//#comment13398 does this make it so that Exception is never thrown and that this the default? trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorRepo.java https://reviews.apache.org/r//#comment13399 same comment as above -- default naming convention ensures that exception is never thrown? - Chris On 2012-03-22 06:09:52, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r// --- (Updated 2012-03-22 06:09:52) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- - Introduced NamingConvention support, which for MetExtractorProductCrawler now takes an ID for the NamingConvention to use and AutoDetectProductCrawler has a new element namingConvention class=/ in MimeExtractorRepo - Also cleaned up handleFile(File)... documented better and is now public and returns the IngestResult for what happened when called. This addresses bug OODT-426. https://issues.apache.org/jira/browse/OODT-426 Diffs - trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/AutoDetectProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/MetExtractorProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/ProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/StdProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigMetKeys.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigReader.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorRepo.java 1302790 trunk/crawler/src/main/resources/cmd-line-options.xml 1302790 trunk/crawler/src/main/resources/crawler-config.xml 1302790 trunk/crawler/src/main/resources/examples/mime-extractor-map.xml 1302790 trunk/crawler/src/main/resources/naming-beans.xml PRE-CREATION Diff: https://reviews.apache.org/r//diff Testing --- Still need to unit-test up cas-crawler Thanks, brian
Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-Crawler changes)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r//#review6215 --- looking good happy to review unit tests when they are there. LGTM dude you rule! - Chris On 2012-03-22 06:09:52, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r// --- (Updated 2012-03-22 06:09:52) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- - Introduced NamingConvention support, which for MetExtractorProductCrawler now takes an ID for the NamingConvention to use and AutoDetectProductCrawler has a new element namingConvention class=/ in MimeExtractorRepo - Also cleaned up handleFile(File)... documented better and is now public and returns the IngestResult for what happened when called. This addresses bug OODT-426. https://issues.apache.org/jira/browse/OODT-426 Diffs - trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/AutoDetectProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/MetExtractorProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/ProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/StdProductCrawler.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigMetKeys.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigReader.java 1302790 trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorRepo.java 1302790 trunk/crawler/src/main/resources/cmd-line-options.xml 1302790 trunk/crawler/src/main/resources/crawler-config.xml 1302790 trunk/crawler/src/main/resources/examples/mime-extractor-map.xml 1302790 trunk/crawler/src/main/resources/naming-beans.xml PRE-CREATION Diff: https://reviews.apache.org/r//diff Testing --- Still need to unit-test up cas-crawler Thanks, brian
