Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-PGE Changes)
On 2012-04-04 02:12:41, Paul Ramirez wrote: trunk/pge/src/main/resources/examples/Crawler/action-beans.xml, lines 29-37 https://reviews.apache.org/r/4628/diff/1/?file=98806#file98806line29 I'd define these properties in another file and then include them here. This is only a suggestion and not a just but I see the properties as something that could likely be changed or set to a fixed value and if we factor it out of here we can keep people from touching this file too much. I think this file just makes peoples heads spin at first but the properties don't (i.e. it hides the Spring goodness in a good way). done On 2012-04-04 02:12:41, Paul Ramirez wrote: trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml, lines 42-43 https://reviews.apache.org/r/4628/diff/1/?file=98813#file98813line42 Put these examples inside comment tags as they wouldn't work as they existed anyhow. Also putting a longer description in the comment would help (i.e. one or more of these is not as helpful as what it does functionally. Why did we remove the files tag? Is this no longer supported? If it is then I recommend putting it back in but commented out. For instance, I'd expect that instead of metadata keys you want to set more of what will be done with that custom metadata would be of use. Also an example of multivalued metadata. Added a TODO at the top of this file... The reader for this file still needs to be updated... so when i update it i'll make this file a working example when i write the unit-tests for it - brian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4628/#review6670 --- On 2012-04-03 21:56:17, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4628/ --- (Updated 2012-04-03 21:56:17) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- CAS-PGE Changes to this issue... - Renaming and Metadata extraction removed from CAS-PGE and instead CAS-PGE now uses AutoDetectProductCrawler instead of StdProductCrawler This addresses bug OODT-426. https://issues.apache.org/jira/browse/OODT-426 Diffs - trunk/pge/pom.xml 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/OutputDir.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfig.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigBuilder.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigMetKeys.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RegExprOutputFiles.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RenamingConv.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/XmlFilePgeConfigBuilder.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/ExternExtractorMetWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/FilenameExtractorWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/PcsMetFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/SciPgeConfigFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/metlist/MetadataListPcsMetFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/xslt/XslTransformWriter.java 1302648 trunk/pge/src/main/resources/examples/Crawler/action-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/crawler-config.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/mime-extractor-map.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/mime-types.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/naming-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/precondition-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/MetadataOutputFiles/metadata-output.xml 1302648 trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml 1302648 trunk/pge/src/test/org/apache/oodt/cas/pge/TestPGETaskInstance.java 1302781 Diff: https://reviews.apache.org/r/4628/diff Testing --- Several Unit-tests Thanks, brian
Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-PGE Changes)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4628/ --- (Updated 2012-04-06 02:16:10.469275) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Changes --- Updates per comments in reviews Summary --- CAS-PGE Changes to this issue... - Renaming and Metadata extraction removed from CAS-PGE and instead CAS-PGE now uses AutoDetectProductCrawler instead of StdProductCrawler This addresses bug OODT-426. https://issues.apache.org/jira/browse/OODT-426 Diffs (updated) - trunk/pge/src/main/resources/examples/Crawler/naming-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/precondition-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/MetadataOutputFiles/metadata-output.xml 1302648 trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml 1302648 trunk/pge/src/test/org/apache/oodt/cas/pge/TestPGETaskInstance.java 1302781 trunk/pge/src/main/resources/examples/Crawler/mime-types.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/mime-extractor-map.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/crawler-config.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/filename.extractor.config.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/action-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/action-beans.properties PRE-CREATION trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/metlist/MetadataListPcsMetFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/xslt/XslTransformWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/SciPgeConfigFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/XmlFilePgeConfigBuilder.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/ExternExtractorMetWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/FilenameExtractorWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/PcsMetFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RenamingConv.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigBuilder.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigMetKeys.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RegExprOutputFiles.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/OutputDir.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfig.java 1302648 trunk/pge/pom.xml 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1302648 Diff: https://reviews.apache.org/r/4628/diff Testing --- Several Unit-tests Thanks, brian
Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-PGE Changes)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4628/#review6734 --- Ship it! - Chris On 2012-04-06 02:16:10, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4628/ --- (Updated 2012-04-06 02:16:10) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- CAS-PGE Changes to this issue... - Renaming and Metadata extraction removed from CAS-PGE and instead CAS-PGE now uses AutoDetectProductCrawler instead of StdProductCrawler This addresses bug OODT-426. https://issues.apache.org/jira/browse/OODT-426 Diffs - trunk/pge/src/main/resources/examples/Crawler/naming-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/precondition-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/MetadataOutputFiles/metadata-output.xml 1302648 trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml 1302648 trunk/pge/src/test/org/apache/oodt/cas/pge/TestPGETaskInstance.java 1302781 trunk/pge/src/main/resources/examples/Crawler/mime-types.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/mime-extractor-map.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/crawler-config.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/filename.extractor.config.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/action-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/action-beans.properties PRE-CREATION trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/metlist/MetadataListPcsMetFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/xslt/XslTransformWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/SciPgeConfigFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/XmlFilePgeConfigBuilder.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/ExternExtractorMetWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/FilenameExtractorWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/PcsMetFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RenamingConv.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigBuilder.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigMetKeys.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RegExprOutputFiles.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/OutputDir.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfig.java 1302648 trunk/pge/pom.xml 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1302648 Diff: https://reviews.apache.org/r/4628/diff Testing --- Several Unit-tests Thanks, brian
Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-PGE Changes)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4628/#review6735 --- Ship it! LGTM sounds good. - Chris On 2012-04-06 02:16:10, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4628/ --- (Updated 2012-04-06 02:16:10) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- CAS-PGE Changes to this issue... - Renaming and Metadata extraction removed from CAS-PGE and instead CAS-PGE now uses AutoDetectProductCrawler instead of StdProductCrawler This addresses bug OODT-426. https://issues.apache.org/jira/browse/OODT-426 Diffs - trunk/pge/src/main/resources/examples/Crawler/naming-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/precondition-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/MetadataOutputFiles/metadata-output.xml 1302648 trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml 1302648 trunk/pge/src/test/org/apache/oodt/cas/pge/TestPGETaskInstance.java 1302781 trunk/pge/src/main/resources/examples/Crawler/mime-types.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/mime-extractor-map.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/crawler-config.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/filename.extractor.config.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/action-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/action-beans.properties PRE-CREATION trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/metlist/MetadataListPcsMetFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/xslt/XslTransformWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/SciPgeConfigFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/XmlFilePgeConfigBuilder.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/ExternExtractorMetWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/FilenameExtractorWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/PcsMetFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RenamingConv.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigBuilder.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigMetKeys.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RegExprOutputFiles.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/OutputDir.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfig.java 1302648 trunk/pge/pom.xml 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1302648 Diff: https://reviews.apache.org/r/4628/diff Testing --- Several Unit-tests Thanks, brian
Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-PGE Changes)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4628/#review6686 --- trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java https://reviews.apache.org/r/4628/#comment14489 this seems like an ancillary change to this patch. However, it's a useful functionality so I don't feel strongly about separating it out. Just be wary of stuff like this (b/c as it grows) it can take away from the purpose of the patch ;) trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/OutputDir.java https://reviews.apache.org/r/4628/#comment14496 +like - Chris On 2012-04-03 21:56:17, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4628/ --- (Updated 2012-04-03 21:56:17) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- CAS-PGE Changes to this issue... - Renaming and Metadata extraction removed from CAS-PGE and instead CAS-PGE now uses AutoDetectProductCrawler instead of StdProductCrawler This addresses bug OODT-426. https://issues.apache.org/jira/browse/OODT-426 Diffs - trunk/pge/pom.xml 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/OutputDir.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfig.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigBuilder.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigMetKeys.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RegExprOutputFiles.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RenamingConv.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/XmlFilePgeConfigBuilder.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/ExternExtractorMetWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/FilenameExtractorWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/PcsMetFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/SciPgeConfigFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/metlist/MetadataListPcsMetFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/xslt/XslTransformWriter.java 1302648 trunk/pge/src/main/resources/examples/Crawler/action-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/crawler-config.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/mime-extractor-map.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/mime-types.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/naming-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/precondition-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/MetadataOutputFiles/metadata-output.xml 1302648 trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml 1302648 trunk/pge/src/test/org/apache/oodt/cas/pge/TestPGETaskInstance.java 1302781 Diff: https://reviews.apache.org/r/4628/diff Testing --- Several Unit-tests Thanks, brian
Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-PGE Changes)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4628/#review6670 --- trunk/pge/src/main/resources/examples/Crawler/action-beans.xml https://reviews.apache.org/r/4628/#comment14432 I'd define these properties in another file and then include them here. This is only a suggestion and not a just but I see the properties as something that could likely be changed or set to a fixed value and if we factor it out of here we can keep people from touching this file too much. I think this file just makes peoples heads spin at first but the properties don't (i.e. it hides the Spring goodness in a good way). trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml https://reviews.apache.org/r/4628/#comment14431 Put these examples inside comment tags as they wouldn't work as they existed anyhow. Also putting a longer description in the comment would help (i.e. one or more of these is not as helpful as what it does functionally. Why did we remove the files tag? Is this no longer supported? If it is then I recommend putting it back in but commented out. For instance, I'd expect that instead of metadata keys you want to set more of what will be done with that custom metadata would be of use. Also an example of multivalued metadata. - Paul On 2012-04-03 21:56:17, brian Foster wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4628/ --- (Updated 2012-04-03 21:56:17) Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas Bennett. Summary --- CAS-PGE Changes to this issue... - Renaming and Metadata extraction removed from CAS-PGE and instead CAS-PGE now uses AutoDetectProductCrawler instead of StdProductCrawler This addresses bug OODT-426. https://issues.apache.org/jira/browse/OODT-426 Diffs - trunk/pge/pom.xml 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/OutputDir.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfig.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigBuilder.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigMetKeys.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RegExprOutputFiles.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RenamingConv.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/XmlFilePgeConfigBuilder.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/ExternExtractorMetWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/FilenameExtractorWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/PcsMetFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/SciPgeConfigFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/metlist/MetadataListPcsMetFileWriter.java 1302648 trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/xslt/XslTransformWriter.java 1302648 trunk/pge/src/main/resources/examples/Crawler/action-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/crawler-config.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/mime-extractor-map.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/mime-types.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/naming-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/Crawler/precondition-beans.xml PRE-CREATION trunk/pge/src/main/resources/examples/MetadataOutputFiles/metadata-output.xml 1302648 trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml 1302648 trunk/pge/src/test/org/apache/oodt/cas/pge/TestPGETaskInstance.java 1302781 Diff: https://reviews.apache.org/r/4628/diff Testing --- Several Unit-tests Thanks, brian
