Hi K, OK so I did a bit of searching here and located a bunch of files which are defined as legacy... you can check the search results out below https://github.com/apache/oodt/search?utf8=%E2%9C%93&q=AutoDetectProductCrawler&type=Code I would urge you to have a look at the AutoDetectProductCrawler Javadoc description included in master branch [0] as well to see if you've got everything required. Finally, I came across some documentation on the wiki which may guide you in the right direction [1]. It may also be outdated though so please let us know if that it the case. hth
[0] https://github.com/apache/oodt/blob/91d0bafe71124906bd94baad746189caf35fb39c/crawler/src/main/java/org/apache/oodt/cas/crawl/AutoDetectProductCrawler.java#L40-L64 [1] https://cwiki.apache.org/confluence/display/OODT/Mime+type+detection+with+the+AutoDetectProductCrawler On Mon, Apr 4, 2016 at 10:54 PM, Konstantinos Mavrommatis < [email protected]> wrote: > Hi, > It seems to be happening for a number of types of files that I have in the > mimetypes.xml. > A few things are puzzling to me: this file which is a .gz file is not > processed by the regular tika mimetypes which contains the gzip files > A file that has no extension, which defaults to txt is passed to the > MetExtractor.pl and processed. > > Any ideas I can find what are the preconditions that fail ? I tried to > change the log level to DEBUG for all components but I did not get much > more information. This must be something that changed in the OODT releases > >0.6 but could not find anything relevant in the release notes. > I also noticed in the documentation of the AutoDecectProductCrawler that > it uses the file met-extr-preconditions.xml which I could not find anywhere > in the deployed OODT or the src directories. Could that be a reason for the > problem I observe? > > Thanks > K > > -----Original Message----- > From: Lewis John Mcgibbney [mailto:[email protected]] > Sent: Monday, April 04, 2016 3:24 PM > To: [email protected] > Subject: Re: Transition from OODT 0.6 to 0.12 cannot find extractor > specifications > > Hi Konstantinos, > It appears to be happening with a tar.gz file as well right? > > WARNING: No extractor specs specified for > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fastq/cas-crawler-04-02-16.log.gz > > I wonder if it is the file names... However I would be extremely surprised > as I've seen some much more verbose file naming. > Lewis > > On Saturday, April 2, 2016, Konstantinos Mavrommatis < > [email protected]> wrote: > > > Hi, > > I am trying to replicate a fully functional service that I had setup > > long time ago using OODT 0.6 but I am having the following problem > > that does not allow me to ingest files. When I try to ingest files > > with the extension fastq.gz I get the line: > > WARNING: No extractor specs specified for > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/E837642_R1.fastq.gz Apr 02, 2016 10:12:14 PM > > org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > And of course the file is not ingested. This process works without > > problem with OODT 0.6 on a different server. > > > > The crawler command I am running is: > > ./crawler_launcher \ > > --operation \ > > --launchAutoCrawler \ > > --productPath $FILEPATH \ > > --filemgrUrl $OODT_FILEMGR_URL \ > > --clientTransferer > > org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferFactory \ > > --mimeExtractorRepo ../policy/mime-extractor-map.xml \ --noRecur \ > > --crawlForDirs 2>&1 > > > > > > > > I have setup OODT 0.12 on a server which runs FM listening to port 9000. > > From a client machine I have verified that I can use FM to ingest > products. > > I am now trying to use crawler to crawl and ingest all files in a > > directory. Since I have non standard MIME types in these directories I > > have done the following: > > 1. Added my own mime types in policy/mimetypes.xml eg > > <mime-type type="text/fastq"> > > <glob pattern="*.fastq"/> > > <glob pattern="*.fastq.gz"/> > > <glob pattern="*.fastq.bz"/> > > <glob pattern="*.fastq.bz2"/> > > <glob pattern="*.fastq.bzip"/> > > <glob pattern="*.fq"/> > > <glob pattern="*.fq.gz"/> > > <glob pattern="*.fq.bz"/> > > <glob pattern="*.fq.bz2"/> > > <glob pattern="*.fq.bzip"/> > > </mime-type> > > 2. created the file policy/mime-extractor-map.xml > > > > <mime type="text/fastq"> > > <extractor > > class="org.apache.oodt.cas.metadata.extractors.ExternMetExtractor"> > > <config > > file="/apache-oodt/crawler/bin/fastq.config"/> > > <preCondComparators> > > <preCondComparator > > id="CheckThatDataFileSizeIsGreaterThanZero"/> > > </preCondComparators> > > </extractor> > > </mime> > > > > 3. created the file fastq.config > > <?xml version="1.0" encoding="UTF-8"?> <cas:externextractor > > xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas";> > > <exec workingDir=""> > > > > > <extractorBinPath>/apache-oodt/crawler/bin/MetExtractorNGS.pl</extractorBinPath> > > <args> > > <arg isDataFile="true"></arg> > > <arg>fastq</arg> > > </args> > > </exec> > > </cas:externextractor> > > > > > > > > The MetExtractorNGS.pl is a small perl script that opens the file to > > be ingested, gets some information and stores it in the .met file that > > corresponds to the file to be ingested and have manually verified that > > works as expected producing the correct met file. > > > > What am I missing here? Any ideas comments suggestions will be greatly > > appreciated. > > Thanks in advance for any help > > Kostas > > > > > > > > PS1 The full output from running the crawler command follows: > > > > > > Setting property 'StdProductCrawler.filemgrUrl' > > Setting property 'MetExtractorProductCrawler.filemgrUrl' > > Setting property 'AutoDetectProductCrawler.filemgrUrl' > > Setting property 'StdProductCrawler.clientTransferer' > > Setting property 'MetExtractorProductCrawler.clientTransferer' > > Setting property 'AutoDetectProductCrawler.clientTransferer' > > Setting property 'StdProductCrawler.noRecur' > > Setting property 'MetExtractorProductCrawler.noRecur' > > Setting property 'AutoDetectProductCrawler.noRecur' > > Setting property 'AutoDetectProductCrawler.mimeExtractorRepo' > > Setting property 'StdProductCrawler.productPath' > > Setting property 'MetExtractorProductCrawler.productPath' > > Setting property 'AutoDetectProductCrawler.productPath' > > Apr 02, 2016 10:12:13 PM > > org.springframework.beans.factory.config.PropertyOverrideConfigurer > > processKey > > FINE: Property 'AutoDetectProductCrawler.noRecur' set to value [true] > > Apr 02, 2016 10:12:13 PM > > org.springframework.beans.factory.config.PropertyOverrideConfigurer > > processKey > > FINE: Property 'StdProductCrawler.productPath' set to value > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq] > > Apr 02, 2016 10:12:13 PM > > org.springframework.beans.factory.config.PropertyOverrideConfigurer > > processKey > > FINE: Property 'MetExtractorProductCrawler.noRecur' set to value > > [true] Apr 02, 2016 10:12:13 PM > > org.springframework.beans.factory.config.PropertyOverrideConfigurer > > processKey > > FINE: Property 'AutoDetectProductCrawler.mimeExtractorRepo' set to > > value [../policy/mime-extractor-map.xml] > > Apr 02, 2016 10:12:13 PM > > org.springframework.beans.factory.config.PropertyOverrideConfigurer > > processKey > > FINE: Property 'MetExtractorProductCrawler.clientTransferer' set to > > value > > [org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferFactory] > > Apr 02, 2016 10:12:13 PM > > org.springframework.beans.factory.config.PropertyOverrideConfigurer > > processKey > > FINE: Property 'AutoDetectProductCrawler.filemgrUrl' set to value [ > > https://urldefense.proofpoint.com/v2/url?u=http-3A__192.168.8.44-3A900 > > 0&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs- > > T2gHY95y7ZA&m=Qaz0eKz7FHe35NMF43A17ey59ANhAqJD5ZfwZQC0VRo&s=OvpwZVR1Xq > > gKclL83VXAWh__c7nz87xK_nS-O7hIXqc&e= ] Apr 02, 2016 10:12:13 PM > > org.springframework.beans.factory.config.PropertyOverrideConfigurer > > processKey > > FINE: Property 'AutoDetectProductCrawler.clientTransferer' set to > > value > > [org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferFactory] > > Apr 02, 2016 10:12:13 PM > > org.springframework.beans.factory.config.PropertyOverrideConfigurer > > processKey > > FINE: Property 'StdProductCrawler.noRecur' set to value [true] Apr 02, > > 2016 10:12:13 PM > > org.springframework.beans.factory.config.PropertyOverrideConfigurer > > processKey > > FINE: Property 'StdProductCrawler.filemgrUrl' set to value [ > > https://urldefense.proofpoint.com/v2/url?u=http-3A__192.168.8.44-3A900 > > 0&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs- > > T2gHY95y7ZA&m=Qaz0eKz7FHe35NMF43A17ey59ANhAqJD5ZfwZQC0VRo&s=OvpwZVR1Xq > > gKclL83VXAWh__c7nz87xK_nS-O7hIXqc&e= ] Apr 02, 2016 10:12:13 PM > > org.springframework.beans.factory.config.PropertyOverrideConfigurer > > processKey > > FINE: Property 'AutoDetectProductCrawler.productPath' set to value > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq] > > Apr 02, 2016 10:12:13 PM > > org.springframework.beans.factory.config.PropertyOverrideConfigurer > > processKey > > FINE: Property 'MetExtractorProductCrawler.filemgrUrl' set to value [ > > https://urldefense.proofpoint.com/v2/url?u=http-3A__192.168.8.44-3A900 > > 0&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs- > > T2gHY95y7ZA&m=Qaz0eKz7FHe35NMF43A17ey59ANhAqJD5ZfwZQC0VRo&s=OvpwZVR1Xq > > gKclL83VXAWh__c7nz87xK_nS-O7hIXqc&e= ] Apr 02, 2016 10:12:13 PM > > org.springframework.beans.factory.config.PropertyOverrideConfigurer > > processKey > > FINE: Property 'StdProductCrawler.clientTransferer' set to value > > [org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferFactory] > > Apr 02, 2016 10:12:13 PM > > org.springframework.beans.factory.config.PropertyOverrideConfigurer > > processKey > > FINE: Property 'MetExtractorProductCrawler.productPath' set to value > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq] Apr 02, 2016 10:12:13 PM org.apache.oodt.cas.crawl.ProductCrawler > > crawl > > INFO: Crawling > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q Apr 02, 2016 10:12:13 PM org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > INFO: Handling file > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/E837642_R1.fastq.gz > > Apr 02, 2016 10:12:14 PM > > org.apache.oodt.cas.crawl.AutoDetectProductCrawler passesPreconditions > > WARNING: No extractor specs specified for > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/E837642_R1.fastq.gz Apr 02, 2016 10:12:14 PM > > org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > WARNING: Failed to pass preconditions for ingest of product: > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq/E837642_R1.fastq.gz] Apr 02, 2016 10:12:14 PM > > org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > INFO: Handling file > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/E837642_R1.fastq.gz.met > > Apr 02, 2016 10:12:14 PM > > org.apache.oodt.cas.metadata.preconditions.PreCondEvalUtils eval > > INFO: Passed precondition comparator id > > CheckThatDataFileSizeIsGreaterThanZero > > Apr 02, 2016 10:12:14 PM > > org.apache.oodt.cas.metadata.extractors.ExternMetExtractor > > extrMetadata > > INFO: Generating met file for product file: > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq/E837642_R1.fastq.gz.met] > > Apr 02, 2016 10:12:14 PM > > org.apache.oodt.cas.metadata.extractors.ExternMetExtractor > > extrMetadata > > INFO: Executing command line: > > [/celgene/software/apache-oodt/crawler/bin/MetExtractorNGS.pl > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/E837642_R1.fastq.gz.met > > text ] with workingDir: > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq] > > to extract metadata > > OUTPUT: [WARN : MetExtractorNGS - 2016/04/02 22:12:15] - Input file > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/E837642_R1.fastq.gz.met will be ignored. .met files are not > > processed ! > > Apr 02, 2016 10:12:15 PM org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > SEVERE: Failed to get metadata for product : Met extractor failed to > > create metadata file > > org.apache.oodt.cas.metadata.exceptions.MetExtractionException: Met > > extractor failed to create metadata file > > at > > > org.apache.oodt.cas.metadata.extractors.ExternMetExtractor.extrMetadata(ExternMetExtractor.java:120) > > at > > > org.apache.oodt.cas.metadata.AbstractMetExtractor.extractMetadata(AbstractMetExtractor.java:74) > > at > > > org.apache.oodt.cas.crawl.AutoDetectProductCrawler.getMetadataForProduct(AutoDetectProductCrawler.java:84) > > at > > > org.apache.oodt.cas.crawl.ProductCrawler.handleFile(ProductCrawler.java:136) > > at > > org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:104) > > at > > org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:74) > > at > > > org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:58) > > at > > org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) > > at > > org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:188) > > at > > org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36 > > ) > > > > Apr 02, 2016 10:12:15 PM org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > INFO: Handling file > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/E837642_R2.fastq.gz > > Apr 02, 2016 10:12:15 PM > > org.apache.oodt.cas.crawl.AutoDetectProductCrawler passesPreconditions > > WARNING: No extractor specs specified for > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/E837642_R2.fastq.gz Apr 02, 2016 10:12:15 PM > > org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > WARNING: Failed to pass preconditions for ingest of product: > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq/E837642_R2.fastq.gz] Apr 02, 2016 10:12:15 PM > > org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > INFO: Handling file > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/E837642_R2.fastq.gz.met > > Apr 02, 2016 10:12:15 PM > > org.apache.oodt.cas.metadata.preconditions.PreCondEvalUtils eval > > INFO: Passed precondition comparator id > > CheckThatDataFileSizeIsGreaterThanZero > > Apr 02, 2016 10:12:16 PM > > org.apache.oodt.cas.metadata.extractors.ExternMetExtractor > > extrMetadata > > INFO: Generating met file for product file: > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq/E837642_R2.fastq.gz.met] > > Apr 02, 2016 10:12:16 PM > > org.apache.oodt.cas.metadata.extractors.ExternMetExtractor > > extrMetadata > > INFO: Executing command line: > > [/celgene/software/apache-oodt/crawler/bin/MetExtractorNGS.pl > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/E837642_R2.fastq.gz.met > > text ] with workingDir: > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq] > > to extract metadata > > OUTPUT: [WARN : MetExtractorNGS - 2016/04/02 22:12:16] - Input file > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/E837642_R2.fastq.gz.met will be ignored. .met files are not > > processed ! > > Apr 02, 2016 10:12:17 PM org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > SEVERE: Failed to get metadata for product : Met extractor failed to > > create metadata file > > org.apache.oodt.cas.metadata.exceptions.MetExtractionException: Met > > extractor failed to create metadata file > > at > > > org.apache.oodt.cas.metadata.extractors.ExternMetExtractor.extrMetadata(ExternMetExtractor.java:120) > > at > > > org.apache.oodt.cas.metadata.AbstractMetExtractor.extractMetadata(AbstractMetExtractor.java:74) > > at > > > org.apache.oodt.cas.crawl.AutoDetectProductCrawler.getMetadataForProduct(AutoDetectProductCrawler.java:84) > > at > > > org.apache.oodt.cas.crawl.ProductCrawler.handleFile(ProductCrawler.java:136) > > at > > org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:104) > > at > > org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:74) > > at > > > org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:58) > > at > > org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) > > at > > org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:188) > > at > > org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36 > > ) > > > > Apr 02, 2016 10:12:17 PM org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > INFO: Handling file > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/cas-crawler-04-02-16.log.gz > > Apr 02, 2016 10:12:17 PM > > org.apache.oodt.cas.crawl.AutoDetectProductCrawler passesPreconditions > > WARNING: No extractor specs specified for > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/cas-crawler-04-02-16.log.gz Apr 02, 2016 10:12:17 PM > > org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > WARNING: Failed to pass preconditions for ingest of product: > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq/cas-crawler-04-02-16.log.gz] Apr 02, 2016 10:12:17 PM > > org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > INFO: Handling file > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/cas-crawler-04-02-16.tar.gz > > Apr 02, 2016 10:12:17 PM > > org.apache.oodt.cas.crawl.AutoDetectProductCrawler passesPreconditions > > WARNING: No extractor specs specified for > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/cas-crawler-04-02-16.tar.gz Apr 02, 2016 10:12:17 PM > > org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > WARNING: Failed to pass preconditions for ingest of product: > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq/cas-crawler-04-02-16.tar.gz] Apr 02, 2016 10:12:17 PM > > org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > INFO: Handling file > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/cas-crawler-mnt-celgene.rnd.combio.mmgp.external-TestSeqData-RNA-Seq > > -RawData-fastq-04-02-16.tar.gz > > Apr 02, 2016 10:12:17 PM > > org.apache.oodt.cas.crawl.AutoDetectProductCrawler passesPreconditions > > WARNING: No extractor specs specified for > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/cas-crawler-mnt-celgene.rnd.combio.mmgp.external-TestSeqData-RNA-Seq > > -RawData-fastq-04-02-16.tar.gz Apr 02, 2016 10:12:17 PM > > org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > WARNING: Failed to pass preconditions for ingest of product: > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq/cas-crawler-mnt-celgene.rnd.combio.mmgp.external-TestSeqData-RNA-Se > > q-RawData-fastq-04-02-16.tar.gz] Apr 02, 2016 10:12:17 PM > > org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > INFO: Handling file > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/test > > Apr 02, 2016 10:12:17 PM > > org.apache.oodt.cas.metadata.preconditions.PreCondEvalUtils eval > > INFO: Passed precondition comparator id > > CheckThatDataFileSizeIsGreaterThanZero > > Apr 02, 2016 10:12:17 PM > > org.apache.oodt.cas.metadata.extractors.ExternMetExtractor > > extrMetadata > > INFO: Generating met file for product file: > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq/test] > > Apr 02, 2016 10:12:17 PM > > org.apache.oodt.cas.metadata.extractors.ExternMetExtractor > > extrMetadata > > INFO: Executing command line: > > [/celgene/software/apache-oodt/crawler/bin/MetExtractorNGS.pl > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/test > > text ] with workingDir: > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq] > > to extract metadata > > OUTPUT: [DEBUG : MetExtractorNGS - 2016/04/02 22:12:18] - Accessing > > NGS server at > > https://urldefense.proofpoint.com/v2/url?u=http-3A__192.168.8.44-3A808 > > 2_RPC2&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yv > > Z1Cs-T2gHY95y7ZA&m=Qaz0eKz7FHe35NMF43A17ey59ANhAqJD5ZfwZQC0VRo&s=tSci2 > > Q1bJj0cQnBHjjOwtZjjx9uNMoN5Bi-ABG0Q7Y4&e= > > OUTPUT: [DEBUG : metadataPrepare - 2016/04/02 22:12:18] - addMetadata: > > metadata for file_host are not in array format.Converting.. > > OUTPUT: [DEBUG : metadataPrepare - 2016/04/02 22:12:18] - addMetadata: > > adding key/value [file_host]/[ip-192-168-8-66] > > OUTPUT: [DEBUG : metadataPrepare - 2016/04/02 22:12:18] - addMetadata: > > metadata for ProductType are not in array format.Converting.. > > OUTPUT: [DEBUG : metadataPrepare - 2016/04/02 22:12:18] - addMetadata: > > adding key/value [ProductType]/[GenericFile] > > OUTPUT: [DEBUG : metadataPrepare - 2016/04/02 22:12:18] - addMetadata: > > metadata for ingest_user are not in array format.Converting.. > > OUTPUT: [DEBUG : metadataPrepare - 2016/04/02 22:12:18] - addMetadata: > > adding key/value [ingest_user]/[kmavrommatis] > > OUTPUT: [DEBUG : MetExtractorNGS - 2016/04/02 22:12:18] - The file > > path is ARRAY(0x22d3f48). It will be added under the FilePath metadata > > field > > OUTPUT: [DEBUG : metadataPrepare - 2016/04/02 22:12:18] - addMetadata: > > metadata for FilePath are not in array format.Converting.. > > OUTPUT: [DEBUG : metadataPrepare - 2016/04/02 22:12:18] - addMetadata: > > adding key/value > > [FilePath]/[/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/ > > RawData/fastq/test] > > OUTPUT: [DEBUG : MetExtractorNGS - 2016/04/02 22:12:18] - This file is > > of type text > > OUTPUT: [DEBUG : MetExtractorNGS - 2016/04/02 22:12:18] - Storing > > metadata in file > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/test.met > > OUTPUT: [DEBUG : metadataPrepare - 2016/04/02 22:12:18] - Changing > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/test > > to > > OUTPUT: [DEBUG : metadataPrepare - 2016/04/02 22:12:18] - > > /mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fast > > q/test > > OUTPUT: [DEBUG : metadataPrepare - 2016/04/02 22:12:18] - Changing > > kmavrommatis to > > OUTPUT: [DEBUG : metadataPrepare - 2016/04/02 22:12:18] - kmavrommatis > > OUTPUT: [DEBUG : metadataPrepare - 2016/04/02 22:12:18] - Changing > > GenericFile to > > OUTPUT: [DEBUG : metadataPrepare - 2016/04/02 22:12:18] - GenericFile > > OUTPUT: [DEBUG : metadataPrepare - 2016/04/02 22:12:18] - Changing > > ip-192-168-8-66 to > > OUTPUT: [DEBUG : metadataPrepare - 2016/04/02 22:12:18] - > > ip-192-168-8-66 > > OUTPUT: [DEBUG : MetExtractorNGS - 2016/04/02 22:12:19] - Process > > finished SUCCESSFULLY Apr 02, 2016 10:12:19 PM > > org.apache.oodt.cas.metadata.extractors.ExternMetExtractor > > extrMetadata > > INFO: Met extraction successful for product file: > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq/test] Apr 02, 2016 10:12:19 PM > > org.apache.oodt.cas.crawl.ProductCrawler ingest > > INFO: ProductCrawler: Ready to ingest product: > > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fastq/test]: > > ProductType: [GenericFile] > > Apr 02, 2016 10:12:19 PM > > org.apache.oodt.cas.filemgr.ingest.StdIngester > > setFileManager > > INFO: StdIngester: connected to file manager: > > [https://urldefense.proofpoint.com/v2/url?u=http-3A__192.168.8.44-3A90 > > 00&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs > > -T2gHY95y7ZA&m=Qaz0eKz7FHe35NMF43A17ey59ANhAqJD5ZfwZQC0VRo&s=OvpwZVR1X > > qgKclL83VXAWh__c7nz87xK_nS-O7hIXqc&e= ] Apr 02, 2016 10:12:19 PM > > org.apache.oodt.cas.filemgr.datatransfer.InPlaceDataTransferer > > setFileManagerUrl > > INFO: In Place Data Transfer to: > > [https://urldefense.proofpoint.com/v2/url?u=http-3A__192.168.8.44-3A90 > > 00&d=CwIBaQ&c=CZZujK3G2KuXGEKnzq-Hrg&r=wndYZ4MLMT9l3Zb2WZv2hq2O6yvZ1Cs > > -T2gHY95y7ZA&m=Qaz0eKz7FHe35NMF43A17ey59ANhAqJD5ZfwZQC0VRo&s=OvpwZVR1X > > qgKclL83VXAWh__c7nz87xK_nS-O7hIXqc&e= ] enabled Apr 02, 2016 10:12:19 > > PM org.apache.oodt.cas.filemgr.ingest.StdIngester > > ingest > > INFO: StdIngester: ingesting product: ProductName: [test]: ProductType: > > [GenericFile]: FileLocation: > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq/] > > Apr 02, 2016 10:12:19 PM > > org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient > > ingestProduct > > FINEST: File Manager Client: clientTransfer enabled: transfering > > product [test] Apr 02, 2016 10:12:19 PM > > org.apache.oodt.cas.filemgr.versioning.VersioningUtils > > createBasicDataStoreRefsFlat > > FINE: VersioningUtils: Generated data store ref: > > file:/opt/oodt/data/archive/test/test from origRef: > > file:/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData > > /fastq/test Apr 02, 2016 10:12:19 PM > > org.apache.oodt.cas.crawl.ProductCrawler ingest > > INFO: Successfully ingested product: > > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fastq/test]: > > product id: 4c8de2da-265a-48c4-8380-3f1103dfecfc > > Apr 02, 2016 10:12:19 PM org.apache.oodt.cas.crawl.ProductCrawler > > handleFile > > INFO: Successful ingest of product: > > [/mnt/celgene.rnd.combio.mmgp.external/TestSeqData/RNA-Seq/RawData/fas > > tq/test] > > > > > > ********************************************************* > > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS CONFIDENTIAL AND > > MAY CONTAIN LEGALLY PRIVILEGED INFORMATION INTENDED ONLY FOR THE USE > > OF THE INDIVIDUAL OR INDIVIDUALS NAMED ABOVE. > > If the reader is not the intended recipient, or the employee or agent > > responsible to deliver it to the intended recipient, you are hereby > > notified that any dissemination, distribution or copying of this > > communication is strictly prohibited. If you have received this > > communication in error, please reply to the sender to notify us of the > > error and delete the original message. Thank You. > > > > > -- > *Lewis* > > ********************************************************* > THIS ELECTRONIC MAIL MESSAGE AND ANY ATTACHMENT IS > CONFIDENTIAL AND MAY CONTAIN LEGALLY PRIVILEGED > INFORMATION INTENDED ONLY FOR THE USE OF THE INDIVIDUAL > OR INDIVIDUALS NAMED ABOVE. > If the reader is not the intended recipient, or the > employee or agent responsible to deliver it to the > intended recipient, you are hereby notified that any > dissemination, distribution or copying of this > communication is strictly prohibited. If you have > received this communication in error, please reply to the > sender to notify us of the error and delete the original > message. Thank You. > -- *Lewis*
