Jenkins build is still unstable: oodt-trunk #364

2012-03-22 Thread Apache Jenkins Server 
See

Re: crawler actionIds

2012-03-22 Thread holenoter 
hey tom,ya... i was thinking about that... i have to go back and look at how i handled boolean type options in cas-cli... i think i stuff a "true" in the value so that there is always a value... add this comment to review board when you get in tomorrow... i'll start looking into it in the meantime though.-brianOn Mar 22, 2012, at 03:01 PM, Thomas Bennett  wrote:Hey Brian,  I checked out you're diff, but could not log into review board and JIRA from my phone (no cached pwords - groan), so I'll post a quick question here if that's okay.  I'm just wondering about Boolean switches?  I think when OptionInstance.getValues().size() == 0.  Not sure if this is a valid case, but just checking.  Cheers, Tom  Sent from my iPhone  On 22 Mar 2012, at 11:35 PM, Brian Foster  wrote:  > hey tom, >  > okay... got a patch up: https://issues.apache.org/jira/browse/OODT-428 >  > patch it into your crawler an let me know if it solves your problem >  > Also, on another note... that fix you did for *Metadata* to *MetadataFile* CrawlerAction... was it for: MoveMetadataToFailureDir -> MoveMetadataFileToFailureDir... or did i just find a new typo of mine again? >  > -brian >  > On Mar 22, 2012, at 01:36 PM, Brian Foster  wrote: >  >> hey thomas, >>  >> i know what the problem is... working on the fix right now... i'll create an issue describing the problem and post the fix so you can patch it in as soon as i have something. >>  >> -brian >>  >> On Mar 22, 2012, at 07:07 AM, Thomas Bennett  wrote: >>  >>> Sorry forgot to mention that I'm using 0.4. ;-) >>>  >>> On 22 March 2012 16:02, Thomas Bennett  wrote: >>>  >>> > Hi, >>> > >>> > I can't seem to add multiple actionIds when I launch a crawler. Specifying >>> > a single actionId seems to work. >>> > >>> > I'm just wondering if I need to specify the actionId's in a different way. >>> > Note I'm not using the script + abs paths to configuration files. >>> > >>> > Cheers, >>> > Tom >>> > >>> > Command executed: >>> > >>> > ~$ java -Dorg.apache.oodt.cas.cli.debug=true >>> > -Djava.util.logging.config.file=/var/kat/katconfig/static/oodt/etc/logging.properties >>> > -Djava.ext.dirs=/usr/local/oodt/cas-crawler/lib >>> > -Dorg.apache.oodt.cas.crawl.bean.repo=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/crawler-config.xml >>> > -Dorg.apache.oodt.cas.cli.action.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-actions.xml >>> > -Dorg.apache.oodt.cas.cli.option.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-options.xml >>> > org.apache.oodt.cas.crawl.CrawlerLauncher --operation --launchAutoCrawler >>> > --productPath /var/kat/data --filemgrUrl http://127.0.0.1:9101--mimeExtractorRepo >>> > /var/kat/katconfig/static/oodt/cas-crawler/policy/mime-extractor-map.xml >>> > --clientTransferer >>> > org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory >>> > --failureDir /var/kat/failed/ --actionIds DeleteDataFile DeleteMetadataFile >>> > >>> > Error message: >>> > Setting property 'AutoDetectProductCrawler.mimeExtractorRepo' >>> > Setting property 'StdProductCrawler.clientTransferer' >>> > Setting property 'MetExtractorProductCrawler.clientTransferer' >>> > Setting property 'AutoDetectProductCrawler.clientTransferer' >>> > Setting property 'StdProductCrawler.filemgrUrl' >>> > Setting property 'MetExtractorProductCrawler.filemgrUrl' >>> > Setting property 'AutoDetectProductCrawler.filemgrUrl' >>> > Setting property 'StdProductCrawler.actionIds' >>> > Setting property 'MetExtractorProductCrawler.actionIds' >>> > Setting property 'AutoDetectProductCrawler.actionIds' >>> > Setting property 'StdProductCrawler.productPath' >>> > Setting property 'MetExtractorProductCrawler.productPath' >>> > Setting property 'AutoDetectProductCrawler.productPath' >>> > Setting property 'MoveMetadataFileToFailureDir.toDir' >>> > Setting property 'MovePushpullMetFileToFailureDir.toDir' >>> > Setting property 'MoveDataFileToFailureDir.toDir' >>> > Exception in thread "main" java.lang.RuntimeException: >>> > org.apache.oodt.cas.cli.exception.CmdLineActionException: Failed to launch >>> > crawler : No bean named 'DeleteDataFile,DeleteMetadataFile' is defined >>> > at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:191) >>> > at org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36) >>> > Caused by: org.apache.oodt.cas.cli.exception.CmdLineActionException: >>> > Failed to launch crawler : No bean named >>> > 'DeleteDataFile,DeleteMetadataFile' is defined >>> > at >>> > org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:61) >>> > at >>> > org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) >>> > at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) >>> > ... 1 more >>> > Caused by: >>> > org.springframework.beans.factory.NoSuchBeanDefinitionException:

Re: crawler actionIds

2012-03-22 Thread Thomas Bennett 
Hey Brian,

I checked out you're diff, but could not log into review board and JIRA from my 
phone (no cached pwords - groan), so I'll post a quick question here if that's 
okay.

I'm just wondering about Boolean switches?

I think when OptionInstance.getValues().size() == 0.

Not sure if this is a valid case, but just checking.

Cheers,
Tom

Sent from my iPhone

On 22 Mar 2012, at 11:35 PM, Brian Foster  wrote:

> hey tom,
> 
> okay... got a patch up: https://issues.apache.org/jira/browse/OODT-428
> 
> patch it into your crawler an let me know if it solves your problem
> 
> Also, on another note... that fix you did for *Metadata* to *MetadataFile* 
> CrawlerAction... was it for: MoveMetadataToFailureDir -> 
> MoveMetadataFileToFailureDir... or did i just find a new typo of mine again?
> 
> -brian
> 
> On Mar 22, 2012, at 01:36 PM, Brian Foster  wrote:
> 
>> hey thomas,
>> 
>> i know what the problem is... working on the fix right now... i'll create an 
>> issue describing the problem and post the fix so you can patch it in as soon 
>> as i have something.
>> 
>> -brian
>> 
>> On Mar 22, 2012, at 07:07 AM, Thomas Bennett  wrote:
>> 
>>> Sorry forgot to mention that I'm using 0.4. ;-)
>>> 
>>> On 22 March 2012 16:02, Thomas Bennett  wrote:
>>> 
>>> > Hi,
>>> >
>>> > I can't seem to add multiple actionIds when I launch a crawler. Specifying
>>> > a single actionId seems to work.
>>> >
>>> > I'm just wondering if I need to specify the actionId's in a different way.
>>> > Note I'm not using the script + abs paths to configuration files.
>>> >
>>> > Cheers,
>>> > Tom
>>> >
>>> > Command executed:
>>> >
>>> > ~$ java -Dorg.apache.oodt.cas.cli.debug=true
>>> > -Djava.util.logging.config.file=/var/kat/katconfig/static/oodt/etc/logging.properties
>>> > -Djava.ext.dirs=/usr/local/oodt/cas-crawler/lib
>>> > -Dorg.apache.oodt.cas.crawl.bean.repo=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/crawler-config.xml
>>> > -Dorg.apache.oodt.cas.cli.action.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-actions.xml
>>> > -Dorg.apache.oodt.cas.cli.option.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-options.xml
>>> > org.apache.oodt.cas.crawl.CrawlerLauncher --operation --launchAutoCrawler
>>> > --productPath /var/kat/data --filemgrUrl 
>>> > http://127.0.0.1:9101--mimeExtractorRepo
>>> > /var/kat/katconfig/static/oodt/cas-crawler/policy/mime-extractor-map.xml
>>> > --clientTransferer
>>> > org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
>>> > --failureDir /var/kat/failed/ --actionIds DeleteDataFile 
>>> > DeleteMetadataFile
>>> >
>>> > Error message:
>>> > Setting property 'AutoDetectProductCrawler.mimeExtractorRepo'
>>> > Setting property 'StdProductCrawler.clientTransferer'
>>> > Setting property 'MetExtractorProductCrawler.clientTransferer'
>>> > Setting property 'AutoDetectProductCrawler.clientTransferer'
>>> > Setting property 'StdProductCrawler.filemgrUrl'
>>> > Setting property 'MetExtractorProductCrawler.filemgrUrl'
>>> > Setting property 'AutoDetectProductCrawler.filemgrUrl'
>>> > Setting property 'StdProductCrawler.actionIds'
>>> > Setting property 'MetExtractorProductCrawler.actionIds'
>>> > Setting property 'AutoDetectProductCrawler.actionIds'
>>> > Setting property 'StdProductCrawler.productPath'
>>> > Setting property 'MetExtractorProductCrawler.productPath'
>>> > Setting property 'AutoDetectProductCrawler.productPath'
>>> > Setting property 'MoveMetadataFileToFailureDir.toDir'
>>> > Setting property 'MovePushpullMetFileToFailureDir.toDir'
>>> > Setting property 'MoveDataFileToFailureDir.toDir'
>>> > Exception in thread "main" java.lang.RuntimeException:
>>> > org.apache.oodt.cas.cli.exception.CmdLineActionException: Failed to launch
>>> > crawler : No bean named 'DeleteDataFile,DeleteMetadataFile' is defined
>>> > at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:191)
>>> > at org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36)
>>> > Caused by: org.apache.oodt.cas.cli.exception.CmdLineActionException:
>>> > Failed to launch crawler : No bean named
>>> > 'DeleteDataFile,DeleteMetadataFile' is defined
>>> > at
>>> > org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:61)
>>> > at
>>> > org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>>> > at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>> > ... 1 more
>>> > Caused by:
>>> > org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean
>>> > named 'DeleteDataFile,DeleteMetadataFile' is defined
>>> > at
>>> > org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:387)
>>> > at
>>> > org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:968)
>>> > at
>>> > org.springframework.beans.factory.support.AbstractBean

Re: crawler actionIds

2012-03-22 Thread Thomas Bennett 
Hey Brian,

> okay... got a patch up: https://issues.apache.org/jira/browse/OODT-428

Thanks! I'll try it out tomorrow morning :-)

> patch it into your crawler an let me know if it solves your problem

Will do!

> Also, on another note... that fix you did for *Metadata* to *MetadataFile* 
> CrawlerAction... was it for: MoveMetadataToFailureDir -> 
> MoveMetadataFileToFailureDir... or did i just find a new typo of mine again?

Sorry did not commit that change yet. But it is for MoveMetadataToFailureDir -> 
MoveMetadataFileToFailureDir

Cheers,
Tom

> -brian
> 
> On Mar 22, 2012, at 01:36 PM, Brian Foster  wrote:
> 
>> hey thomas,
>> 
>> i know what the problem is... working on the fix right now... i'll create an 
>> issue describing the problem and post the fix so you can patch it in as soon 
>> as i have something.
>> 
>> -brian
>> 
>> On Mar 22, 2012, at 07:07 AM, Thomas Bennett  wrote:
>> 
>>> Sorry forgot to mention that I'm using 0.4. ;-)
>>> 
>>> On 22 March 2012 16:02, Thomas Bennett  wrote:
>>> 
>>> > Hi,
>>> >
>>> > I can't seem to add multiple actionIds when I launch a crawler. Specifying
>>> > a single actionId seems to work.
>>> >
>>> > I'm just wondering if I need to specify the actionId's in a different way.
>>> > Note I'm not using the script + abs paths to configuration files.
>>> >
>>> > Cheers,
>>> > Tom
>>> >
>>> > Command executed:
>>> >
>>> > ~$ java -Dorg.apache.oodt.cas.cli.debug=true
>>> > -Djava.util.logging.config.file=/var/kat/katconfig/static/oodt/etc/logging.properties
>>> > -Djava.ext.dirs=/usr/local/oodt/cas-crawler/lib
>>> > -Dorg.apache.oodt.cas.crawl.bean.repo=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/crawler-config.xml
>>> > -Dorg.apache.oodt.cas.cli.action.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-actions.xml
>>> > -Dorg.apache.oodt.cas.cli.option.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-options.xml
>>> > org.apache.oodt.cas.crawl.CrawlerLauncher --operation --launchAutoCrawler
>>> > --productPath /var/kat/data --filemgrUrl 
>>> > http://127.0.0.1:9101--mimeExtractorRepo
>>> > /var/kat/katconfig/static/oodt/cas-crawler/policy/mime-extractor-map.xml
>>> > --clientTransferer
>>> > org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
>>> > --failureDir /var/kat/failed/ --actionIds DeleteDataFile 
>>> > DeleteMetadataFile
>>> >
>>> > Error message:
>>> > Setting property 'AutoDetectProductCrawler.mimeExtractorRepo'
>>> > Setting property 'StdProductCrawler.clientTransferer'
>>> > Setting property 'MetExtractorProductCrawler.clientTransferer'
>>> > Setting property 'AutoDetectProductCrawler.clientTransferer'
>>> > Setting property 'StdProductCrawler.filemgrUrl'
>>> > Setting property 'MetExtractorProductCrawler.filemgrUrl'
>>> > Setting property 'AutoDetectProductCrawler.filemgrUrl'
>>> > Setting property 'StdProductCrawler.actionIds'
>>> > Setting property 'MetExtractorProductCrawler.actionIds'
>>> > Setting property 'AutoDetectProductCrawler.actionIds'
>>> > Setting property 'StdProductCrawler.productPath'
>>> > Setting property 'MetExtractorProductCrawler.productPath'
>>> > Setting property 'AutoDetectProductCrawler.productPath'
>>> > Setting property 'MoveMetadataFileToFailureDir.toDir'
>>> > Setting property 'MovePushpullMetFileToFailureDir.toDir'
>>> > Setting property 'MoveDataFileToFailureDir.toDir'
>>> > Exception in thread "main" java.lang.RuntimeException:
>>> > org.apache.oodt.cas.cli.exception.CmdLineActionException: Failed to launch
>>> > crawler : No bean named 'DeleteDataFile,DeleteMetadataFile' is defined
>>> > at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:191)
>>> > at org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36)
>>> > Caused by: org.apache.oodt.cas.cli.exception.CmdLineActionException:
>>> > Failed to launch crawler : No bean named
>>> > 'DeleteDataFile,DeleteMetadataFile' is defined
>>> > at
>>> > org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:61)
>>> > at
>>> > org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>>> > at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>> > ... 1 more
>>> > Caused by:
>>> > org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean
>>> > named 'DeleteDataFile,DeleteMetadataFile' is defined
>>> > at
>>> > org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:387)
>>> > at
>>> > org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:968)
>>> > at
>>> > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:246)
>>> > at
>>> > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
>>> > at
>>> > org.springframework.beans.factory.support.AbstractBeanFactory.getBe

Re: crawler actionIds

2012-03-22 Thread Brian Foster 
hey tom,okay... got a patch up: https://issues.apache.org/jira/browse/OODT-428patch it into your crawler an let me know if it solves your problemAlso, on another note... that fix you did for *Metadata* to *MetadataFile* CrawlerAction... was it for: MoveMetadataToFailureDir -> MoveMetadataFileToFailureDir... or did i just find a new typo of mine again?-brianOn Mar 22, 2012, at 01:36 PM, Brian Foster  wrote:hey thomas,i know what the problem is... working on the fix right now... i'll create an issue describing the problem and post the fix so you can patch it in as soon as i have something.-brianOn Mar 22, 2012, at 07:07 AM, Thomas Bennett  wrote:Sorry forgot to mention that I'm using 0.4. ;-)  On 22 March 2012 16:02, Thomas Bennett  wrote:  > Hi, > > I can't seem to add multiple actionIds when I launch a crawler. Specifying > a single actionId seems to work. > > I'm just wondering if I need to specify the actionId's in a different way. > Note I'm not using the script + abs paths to configuration files. > > Cheers, > Tom > > Command executed: > > ~$ java -Dorg.apache.oodt.cas.cli.debug=true > -Djava.util.logging.config.file=/var/kat/katconfig/static/oodt/etc/logging.properties > -Djava.ext.dirs=/usr/local/oodt/cas-crawler/lib > -Dorg.apache.oodt.cas.crawl.bean.repo=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/crawler-config.xml > -Dorg.apache.oodt.cas.cli.action.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-actions.xml > -Dorg.apache.oodt.cas.cli.option.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-options.xml > org.apache.oodt.cas.crawl.CrawlerLauncher --operation --launchAutoCrawler > --productPath /var/kat/data --filemgrUrl http://127.0.0.1:9101--mimeExtractorRepo > /var/kat/katconfig/static/oodt/cas-crawler/policy/mime-extractor-map.xml > --clientTransferer > org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory > --failureDir /var/kat/failed/ --actionIds DeleteDataFile DeleteMetadataFile > > Error message: > Setting property 'AutoDetectProductCrawler.mimeExtractorRepo' > Setting property 'StdProductCrawler.clientTransferer' > Setting property 'MetExtractorProductCrawler.clientTransferer' > Setting property 'AutoDetectProductCrawler.clientTransferer' > Setting property 'StdProductCrawler.filemgrUrl' > Setting property 'MetExtractorProductCrawler.filemgrUrl' > Setting property 'AutoDetectProductCrawler.filemgrUrl' > Setting property 'StdProductCrawler.actionIds' > Setting property 'MetExtractorProductCrawler.actionIds' > Setting property 'AutoDetectProductCrawler.actionIds' > Setting property 'StdProductCrawler.productPath' > Setting property 'MetExtractorProductCrawler.productPath' > Setting property 'AutoDetectProductCrawler.productPath' > Setting property 'MoveMetadataFileToFailureDir.toDir' > Setting property 'MovePushpullMetFileToFailureDir.toDir' > Setting property 'MoveDataFileToFailureDir.toDir' > Exception in thread "main" java.lang.RuntimeException: > org.apache.oodt.cas.cli.exception.CmdLineActionException: Failed to launch > crawler : No bean named 'DeleteDataFile,DeleteMetadataFile' is defined > at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:191) > at org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36) > Caused by: org.apache.oodt.cas.cli.exception.CmdLineActionException: > Failed to launch crawler : No bean named > 'DeleteDataFile,DeleteMetadataFile' is defined > at > org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:61) > at > org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) > at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) > ... 1 more > Caused by: > org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean > named 'DeleteDataFile,DeleteMetadataFile' is defined > at > org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:387) > at > org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:968) > at > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:246) > at > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185) > at > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:168) > at > org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:885) > at > org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:71) > at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:84) > at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:76) > at > org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:58) > ... 3 more >

Re: crawler actionIds

2012-03-22 Thread Thomas Bennett 
Awesome!

I did not get a chance to look any deeper before I left work today, so thanks 
for looking into this.

Cheers,
Tom

Sent from my iPhone

On 22 Mar 2012, at 10:36 PM, Brian Foster  wrote:

> hey thomas,
> 
> i know what the problem is... working on the fix right now... i'll create an 
> issue describing the problem and post the fix so you can patch it in as soon 
> as i have something.
> 
> -brian
> 
> On Mar 22, 2012, at 07:07 AM, Thomas Bennett  wrote:
> 
>> Sorry forgot to mention that I'm using 0.4. ;-)
>> 
>> On 22 March 2012 16:02, Thomas Bennett  wrote:
>> 
>> > Hi,
>> >
>> > I can't seem to add multiple actionIds when I launch a crawler. Specifying
>> > a single actionId seems to work.
>> >
>> > I'm just wondering if I need to specify the actionId's in a different way.
>> > Note I'm not using the script + abs paths to configuration files.
>> >
>> > Cheers,
>> > Tom
>> >
>> > Command executed:
>> >
>> > ~$ java -Dorg.apache.oodt.cas.cli.debug=true
>> > -Djava.util.logging.config.file=/var/kat/katconfig/static/oodt/etc/logging.properties
>> > -Djava.ext.dirs=/usr/local/oodt/cas-crawler/lib
>> > -Dorg.apache.oodt.cas.crawl.bean.repo=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/crawler-config.xml
>> > -Dorg.apache.oodt.cas.cli.action.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-actions.xml
>> > -Dorg.apache.oodt.cas.cli.option.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-options.xml
>> > org.apache.oodt.cas.crawl.CrawlerLauncher --operation --launchAutoCrawler
>> > --productPath /var/kat/data --filemgrUrl 
>> > http://127.0.0.1:9101--mimeExtractorRepo
>> > /var/kat/katconfig/static/oodt/cas-crawler/policy/mime-extractor-map.xml
>> > --clientTransferer
>> > org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
>> > --failureDir /var/kat/failed/ --actionIds DeleteDataFile DeleteMetadataFile
>> >
>> > Error message:
>> > Setting property 'AutoDetectProductCrawler.mimeExtractorRepo'
>> > Setting property 'StdProductCrawler.clientTransferer'
>> > Setting property 'MetExtractorProductCrawler.clientTransferer'
>> > Setting property 'AutoDetectProductCrawler.clientTransferer'
>> > Setting property 'StdProductCrawler.filemgrUrl'
>> > Setting property 'MetExtractorProductCrawler.filemgrUrl'
>> > Setting property 'AutoDetectProductCrawler.filemgrUrl'
>> > Setting property 'StdProductCrawler.actionIds'
>> > Setting property 'MetExtractorProductCrawler.actionIds'
>> > Setting property 'AutoDetectProductCrawler.actionIds'
>> > Setting property 'StdProductCrawler.productPath'
>> > Setting property 'MetExtractorProductCrawler.productPath'
>> > Setting property 'AutoDetectProductCrawler.productPath'
>> > Setting property 'MoveMetadataFileToFailureDir.toDir'
>> > Setting property 'MovePushpullMetFileToFailureDir.toDir'
>> > Setting property 'MoveDataFileToFailureDir.toDir'
>> > Exception in thread "main" java.lang.RuntimeException:
>> > org.apache.oodt.cas.cli.exception.CmdLineActionException: Failed to launch
>> > crawler : No bean named 'DeleteDataFile,DeleteMetadataFile' is defined
>> > at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:191)
>> > at org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36)
>> > Caused by: org.apache.oodt.cas.cli.exception.CmdLineActionException:
>> > Failed to launch crawler : No bean named
>> > 'DeleteDataFile,DeleteMetadataFile' is defined
>> > at
>> > org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:61)
>> > at
>> > org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>> > at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>> > ... 1 more
>> > Caused by:
>> > org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean
>> > named 'DeleteDataFile,DeleteMetadataFile' is defined
>> > at
>> > org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:387)
>> > at
>> > org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:968)
>> > at
>> > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:246)
>> > at
>> > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
>> > at
>> > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:168)
>> > at
>> > org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:885)
>> > at
>> > org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:71)
>> > at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:84)
>> > at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:76)
>> > at
>> > org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(Craw

Re: crawler actionIds

2012-03-22 Thread Brian Foster 
hey thomas,i know what the problem is... working on the fix right now... i'll create an issue describing the problem and post the fix so you can patch it in as soon as i have something.-brianOn Mar 22, 2012, at 07:07 AM, Thomas Bennett  wrote:Sorry forgot to mention that I'm using 0.4. ;-)  On 22 March 2012 16:02, Thomas Bennett  wrote:  > Hi, > > I can't seem to add multiple actionIds when I launch a crawler. Specifying > a single actionId seems to work. > > I'm just wondering if I need to specify the actionId's in a different way. > Note I'm not using the script + abs paths to configuration files. > > Cheers, > Tom > > Command executed: > > ~$ java -Dorg.apache.oodt.cas.cli.debug=true > -Djava.util.logging.config.file=/var/kat/katconfig/static/oodt/etc/logging.properties > -Djava.ext.dirs=/usr/local/oodt/cas-crawler/lib > -Dorg.apache.oodt.cas.crawl.bean.repo=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/crawler-config.xml > -Dorg.apache.oodt.cas.cli.action.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-actions.xml > -Dorg.apache.oodt.cas.cli.option.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-options.xml > org.apache.oodt.cas.crawl.CrawlerLauncher --operation --launchAutoCrawler > --productPath /var/kat/data --filemgrUrl http://127.0.0.1:9101--mimeExtractorRepo > /var/kat/katconfig/static/oodt/cas-crawler/policy/mime-extractor-map.xml > --clientTransferer > org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory > --failureDir /var/kat/failed/ --actionIds DeleteDataFile DeleteMetadataFile > > Error message: > Setting property 'AutoDetectProductCrawler.mimeExtractorRepo' > Setting property 'StdProductCrawler.clientTransferer' > Setting property 'MetExtractorProductCrawler.clientTransferer' > Setting property 'AutoDetectProductCrawler.clientTransferer' > Setting property 'StdProductCrawler.filemgrUrl' > Setting property 'MetExtractorProductCrawler.filemgrUrl' > Setting property 'AutoDetectProductCrawler.filemgrUrl' > Setting property 'StdProductCrawler.actionIds' > Setting property 'MetExtractorProductCrawler.actionIds' > Setting property 'AutoDetectProductCrawler.actionIds' > Setting property 'StdProductCrawler.productPath' > Setting property 'MetExtractorProductCrawler.productPath' > Setting property 'AutoDetectProductCrawler.productPath' > Setting property 'MoveMetadataFileToFailureDir.toDir' > Setting property 'MovePushpullMetFileToFailureDir.toDir' > Setting property 'MoveDataFileToFailureDir.toDir' > Exception in thread "main" java.lang.RuntimeException: > org.apache.oodt.cas.cli.exception.CmdLineActionException: Failed to launch > crawler : No bean named 'DeleteDataFile,DeleteMetadataFile' is defined > at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:191) > at org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36) > Caused by: org.apache.oodt.cas.cli.exception.CmdLineActionException: > Failed to launch crawler : No bean named > 'DeleteDataFile,DeleteMetadataFile' is defined > at > org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:61) > at > org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) > at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) > ... 1 more > Caused by: > org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean > named 'DeleteDataFile,DeleteMetadataFile' is defined > at > org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:387) > at > org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:968) > at > org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:246) > at > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185) > at > org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:168) > at > org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:885) > at > org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:71) > at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:84) > at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:76) > at > org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:58) > ... 3 more >

Re: svn commit: r1302489 - in /oodt/trunk: CHANGES.txt filemgr/src/main/java/org/apache/oodt/cas/filemgr/catalog/DataSourceCatalog.java filemgr/src/test/org/apache/oodt/cas/filemgr/catalog/TestDataSou

2012-03-22 Thread Ramirez, Paul M (388J) 
Hey Chris,

Yep, you're right so I'll look into this further. I know that commit broke the 
build as Jenkins keeps telling the list. I wonder if the bug was there before. 
If I recall you just expanded the unit test. Anyhow when I have more time I'll 
try to track this down if someone doesn't get to it first. Essentially, the 
unit test is failing right now.

--Paul

On Mar 21, 2012, at 1:18 PM, Mattmann, Chris A (388J) wrote:

> Hi Paul,
> 
> On Mar 21, 2012, at 3:41 PM, Ramirez, Paul M (388J) wrote:
> 
>> Hey Guys,
>> 
>> This commit broke the unit test 
>> (org.apache.oodt.cas.filemgr.catalog.TestMappedDataSourceCatalog) with the 
>> addition of:
>> 
>> assertEquals("ShouldBeFirstForPage.txt", 
>> ((Product)myCat.getFirstPage(type).getPageProducts().get(0)).getProductName());
> 
> Hmmm, that commit was from me. Read on below.
> 
>> 
>> The data source catalog is sorted by product id so this test does not 
>> succeed (since 20 products were added before it) as it would with the lucene 
>> catalog which is naturally sorted by CAS.ProductReceivedTime. I can fix the 
>> test to look at the second page but wanted to make sure the intent was 
>> captured. Also this is sort of an oddity with our catalogs having one set 
>> sorted by default with product id and the other with time.
>> 
> 
> Sorting in the DataSourceCatalog by product ID should be equivalent to 
> sorting in the Lucene Catalog by ProductReceivedTime, right?
> 
> Cheers,
> Chris
> 
> ++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattm...@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++
>

Re: crawler actionIds

2012-03-22 Thread Thomas Bennett 
Sorry forgot to mention that I'm using 0.4. ;-)

On 22 March 2012 16:02, Thomas Bennett  wrote:

> Hi,
>
> I can't seem to add multiple actionIds when I launch a crawler. Specifying
> a single actionId seems to work.
>
> I'm just wondering if I need to specify the actionId's in a different way.
> Note I'm not using the script + abs paths to configuration files.
>
> Cheers,
> Tom
>
> Command executed:
>
> ~$ java -Dorg.apache.oodt.cas.cli.debug=true
> -Djava.util.logging.config.file=/var/kat/katconfig/static/oodt/etc/logging.properties
> -Djava.ext.dirs=/usr/local/oodt/cas-crawler/lib
> -Dorg.apache.oodt.cas.crawl.bean.repo=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/crawler-config.xml
>  
> -Dorg.apache.oodt.cas.cli.action.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-actions.xml
> -Dorg.apache.oodt.cas.cli.option.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-options.xml
> org.apache.oodt.cas.crawl.CrawlerLauncher --operation --launchAutoCrawler
> --productPath /var/kat/data --filemgrUrl 
> http://127.0.0.1:9101--mimeExtractorRepo
> /var/kat/katconfig/static/oodt/cas-crawler/policy/mime-extractor-map.xml
> --clientTransferer
> org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
> --failureDir /var/kat/failed/ --actionIds DeleteDataFile DeleteMetadataFile
>
> Error message:
> Setting property 'AutoDetectProductCrawler.mimeExtractorRepo'
> Setting property 'StdProductCrawler.clientTransferer'
> Setting property 'MetExtractorProductCrawler.clientTransferer'
> Setting property 'AutoDetectProductCrawler.clientTransferer'
> Setting property 'StdProductCrawler.filemgrUrl'
> Setting property 'MetExtractorProductCrawler.filemgrUrl'
> Setting property 'AutoDetectProductCrawler.filemgrUrl'
> Setting property 'StdProductCrawler.actionIds'
> Setting property 'MetExtractorProductCrawler.actionIds'
> Setting property 'AutoDetectProductCrawler.actionIds'
> Setting property 'StdProductCrawler.productPath'
> Setting property 'MetExtractorProductCrawler.productPath'
> Setting property 'AutoDetectProductCrawler.productPath'
> Setting property 'MoveMetadataFileToFailureDir.toDir'
> Setting property 'MovePushpullMetFileToFailureDir.toDir'
> Setting property 'MoveDataFileToFailureDir.toDir'
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.oodt.cas.cli.exception.CmdLineActionException: Failed to launch
> crawler : No bean named 'DeleteDataFile,DeleteMetadataFile' is defined
>  at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:191)
> at org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36)
> Caused by: org.apache.oodt.cas.cli.exception.CmdLineActionException:
> Failed to launch crawler : No bean named
> 'DeleteDataFile,DeleteMetadataFile' is defined
> at
> org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:61)
>  at
> org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
> at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>  ... 1 more
> Caused by:
> org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean
> named 'DeleteDataFile,DeleteMetadataFile' is defined
>  at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:387)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:968)
>  at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:246)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
>  at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:168)
> at
> org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:885)
>  at
> org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:71)
> at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:84)
>  at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:76)
> at
> org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:58)
>  ... 3 more
>

crawler actionIds

2012-03-22 Thread Thomas Bennett 
Hi,

I can't seem to add multiple actionIds when I launch a crawler. Specifying
a single actionId seems to work.

I'm just wondering if I need to specify the actionId's in a different way.
Note I'm not using the script + abs paths to configuration files.

Cheers,
Tom

Command executed:

~$ java -Dorg.apache.oodt.cas.cli.debug=true
-Djava.util.logging.config.file=/var/kat/katconfig/static/oodt/etc/logging.properties
-Djava.ext.dirs=/usr/local/oodt/cas-crawler/lib
-Dorg.apache.oodt.cas.crawl.bean.repo=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/crawler-config.xml
 
-Dorg.apache.oodt.cas.cli.action.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-actions.xml
-Dorg.apache.oodt.cas.cli.option.spring.config=file:///var/kat/katconfig/static/oodt/cas-crawler/policy/cmd-line-options.xml
org.apache.oodt.cas.crawl.CrawlerLauncher --operation --launchAutoCrawler
--productPath /var/kat/data --filemgrUrl
http://127.0.0.1:9101--mimeExtractorRepo
/var/kat/katconfig/static/oodt/cas-crawler/policy/mime-extractor-map.xml
--clientTransferer
org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
--failureDir /var/kat/failed/ --actionIds DeleteDataFile DeleteMetadataFile

Error message:
Setting property 'AutoDetectProductCrawler.mimeExtractorRepo'
Setting property 'StdProductCrawler.clientTransferer'
Setting property 'MetExtractorProductCrawler.clientTransferer'
Setting property 'AutoDetectProductCrawler.clientTransferer'
Setting property 'StdProductCrawler.filemgrUrl'
Setting property 'MetExtractorProductCrawler.filemgrUrl'
Setting property 'AutoDetectProductCrawler.filemgrUrl'
Setting property 'StdProductCrawler.actionIds'
Setting property 'MetExtractorProductCrawler.actionIds'
Setting property 'AutoDetectProductCrawler.actionIds'
Setting property 'StdProductCrawler.productPath'
Setting property 'MetExtractorProductCrawler.productPath'
Setting property 'AutoDetectProductCrawler.productPath'
Setting property 'MoveMetadataFileToFailureDir.toDir'
Setting property 'MovePushpullMetFileToFailureDir.toDir'
Setting property 'MoveDataFileToFailureDir.toDir'
Exception in thread "main" java.lang.RuntimeException:
org.apache.oodt.cas.cli.exception.CmdLineActionException: Failed to launch
crawler : No bean named 'DeleteDataFile,DeleteMetadataFile' is defined
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:191)
at org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36)
Caused by: org.apache.oodt.cas.cli.exception.CmdLineActionException: Failed
to launch crawler : No bean named 'DeleteDataFile,DeleteMetadataFile' is
defined
at
org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:61)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
... 1 more
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException:
No bean named 'DeleteDataFile,DeleteMetadataFile' is defined
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:387)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:968)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:246)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:168)
at
org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:885)
at
org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:71)
at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:84)
at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:76)
at
org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:58)
... 3 more

Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-Crawler changes)

2012-03-22 Thread Chris Mattmann 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r//#review6215
---


looking good happy to review unit tests when they are there. LGTM dude you rule!

- Chris


On 2012-03-22 06:09:52, brian Foster wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r//
> ---
> 
> (Updated 2012-03-22 06:09:52)
> 
> 
> Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and 
> Thomas Bennett.
> 
> 
> Summary
> ---
> 
> - Introduced NamingConvention support, which for MetExtractorProductCrawler 
> now takes an ID for the NamingConvention to use and AutoDetectProductCrawler 
> has a new element  in MimeExtractorRepo
> - Also cleaned up handleFile(File)... documented better and is now public and 
> returns the IngestResult for what happened when called.
> 
> 
> This addresses bug OODT-426.
> https://issues.apache.org/jira/browse/OODT-426
> 
> 
> Diffs
> -
> 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/AutoDetectProductCrawler.java
>  1302790 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/MetExtractorProductCrawler.java
>  1302790 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/ProductCrawler.java 
> 1302790 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/StdProductCrawler.java 
> 1302790 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigMetKeys.java
>  1302790 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigReader.java
>  1302790 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorRepo.java
>  1302790 
>   trunk/crawler/src/main/resources/cmd-line-options.xml 1302790 
>   trunk/crawler/src/main/resources/crawler-config.xml 1302790 
>   trunk/crawler/src/main/resources/examples/mime-extractor-map.xml 1302790 
>   trunk/crawler/src/main/resources/naming-beans.xml PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r//diff
> 
> 
> Testing
> ---
> 
> Still need to unit-test up cas-crawler
> 
> 
> Thanks,
> 
> brian
> 
>

Re: Review Request: Introduce a CAS-Metadata based renaming interface (CAS-Crawler changes)

2012-03-22 Thread Chris Mattmann 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r//#review6214
---



trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/MetExtractorProductCrawler.java


will this cause back incompat changes for peeps with default crawl scripts 
that use MetExtractorProductCrawler? If so, can we default it to something to 
appease that?



trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigReader.java


good job adding this!



trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigReader.java


does this make it so that Exception is never thrown and that this the 
default?



trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorRepo.java


same comment as above -- default naming convention ensures that exception 
is never thrown?


- Chris


On 2012-03-22 06:09:52, brian Foster wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r//
> ---
> 
> (Updated 2012-03-22 06:09:52)
> 
> 
> Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and 
> Thomas Bennett.
> 
> 
> Summary
> ---
> 
> - Introduced NamingConvention support, which for MetExtractorProductCrawler 
> now takes an ID for the NamingConvention to use and AutoDetectProductCrawler 
> has a new element  in MimeExtractorRepo
> - Also cleaned up handleFile(File)... documented better and is now public and 
> returns the IngestResult for what happened when called.
> 
> 
> This addresses bug OODT-426.
> https://issues.apache.org/jira/browse/OODT-426
> 
> 
> Diffs
> -
> 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/AutoDetectProductCrawler.java
>  1302790 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/MetExtractorProductCrawler.java
>  1302790 
>   trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/ProductCrawler.java 
> 1302790 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/StdProductCrawler.java 
> 1302790 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigMetKeys.java
>  1302790 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigReader.java
>  1302790 
>   
> trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorRepo.java
>  1302790 
>   trunk/crawler/src/main/resources/cmd-line-options.xml 1302790 
>   trunk/crawler/src/main/resources/crawler-config.xml 1302790 
>   trunk/crawler/src/main/resources/examples/mime-extractor-map.xml 1302790 
>   trunk/crawler/src/main/resources/naming-beans.xml PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r//diff
> 
> 
> Testing
> ---
> 
> Still need to unit-test up cas-crawler
> 
> 
> Thanks,
> 
> brian
> 
>

Review Request: Introduce a CAS-Metadata based renaming interface (CAS-Crawler changes)

2012-03-22 Thread brian Foster 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r//
---

Review request for oodt, Chris Mattmann, Ricky Nguyen, Paul Ramirez, and Thomas 
Bennett.


Summary
---

- Introduced NamingConvention support, which for MetExtractorProductCrawler now 
takes an ID for the NamingConvention to use and AutoDetectProductCrawler has a 
new element  in MimeExtractorRepo
- Also cleaned up handleFile(File)... documented better and is now public and 
returns the IngestResult for what happened when called.


This addresses bug OODT-426.
https://issues.apache.org/jira/browse/OODT-426


Diffs
-

  
trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/AutoDetectProductCrawler.java
 1302790 
  
trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/MetExtractorProductCrawler.java
 1302790 
  trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/ProductCrawler.java 
1302790 
  trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/StdProductCrawler.java 
1302790 
  
trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigMetKeys.java
 1302790 
  
trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorConfigReader.java
 1302790 
  
trunk/crawler/src/main/java/org/apache/oodt/cas/crawl/typedetection/MimeExtractorRepo.java
 1302790 
  trunk/crawler/src/main/resources/cmd-line-options.xml 1302790 
  trunk/crawler/src/main/resources/crawler-config.xml 1302790 
  trunk/crawler/src/main/resources/examples/mime-extractor-map.xml 1302790 
  trunk/crawler/src/main/resources/naming-beans.xml PRE-CREATION 

Diff: https://reviews.apache.org/r//diff


Testing
---

Still need to unit-test up cas-crawler


Thanks,

brian