Hi All, I am a newbie here, so please be patient if I ask dumb questions. I am trying to work my way through the process of setting up a small system to do some basic crawling and ingesting. I have science, meta data, and engineering files that are dropped off into a subdirectory of my staging directory where the name of the subdirectory is based on the file type combined with the UTC year and day of year of when the files are received.
For example: All of the engineering data files (*.ecsv, *.sfdu) are dropped off in: staging/ops/eng/[year]/[day of year] And, all of the raw science data files and their associated meta data files (*.out, *.dtl, *.lbl) are dropped off in in: staging/ops/sci/[year]/[day of year] Each of these mime types have been added in filemgr/etc/mime-types.xml The engineering files require only simple handling, so I am starting with those first. I am using the AutoDetectProductCrawler. I have defined a ProductType of "EngineeringFile" in filemgr/policy/oodt/product-types.xml. I added the "EngineeringFile" (which is just a copy of the GenericFile product type for now) to filemgr/policy/oodt/product-type-element-map.xml, and gave it the same elements the "GenericFile" product type has. I added the engineering files mime types to extensions/policy/mime-extractor-map.xml, and I wrote a simple ExternExtractor in perl that creates a .met file for each engineering file in the directory, sets the same keys/values that are set in the example met file "blah.txt.met", and returns a 0 upon success. I made my own copy of the oodt script and added statements to start crawler_launcher for me. When I run it all of the processes come up and run, the directory is crawled, the extractor is called and the met files are created. Yay! This in itself was a big accomplishment. But, now, the ingestion is failing, and I can't figure out why. Can anyone give me ideas on how to troubleshoot this? The error log is below, if you search for the first occurrence of "WARNING" you will see the start of the problems it is having. Thanks very much. Valerie ---- contents of runoodt.log Using OODT_BASE: /homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy Using OODT_HOME: /homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy Using OODT_TMPDIR: /homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/temp Using JRE_HOME: /project/jedi/users/jedi-pipeline/jdk1.7.0_55 Using CLASSPATH: started filemgr Setting property 'StdProductCrawler.clientTransferer' Setting property 'MetExtractorProductCrawler.clientTransferer' Setting property 'AutoDetectProductCrawler.clientTransferer' Setting property 'StdProductCrawler.filemgrUrl' Setting property 'MetExtractorProductCrawler.filemgrUrl' Setting property 'AutoDetectProductCrawler.filemgrUrl' Setting property 'StdProductCrawler.productPath' Setting property 'MetExtractorProductCrawler.productPath' Setting property 'AutoDetectProductCrawler.productPath' Setting property 'AutoDetectProductCrawler.mimeExtractorRepo' Oct 01, 2014 12:40:07 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey FINE: Property 'StdProductCrawler.productPath' set to value [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng] Oct 01, 2014 12:40:07 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey FINE: Property 'AutoDetectProductCrawler.mimeExtractorRepo' set to value [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/extensions/policy/mime-extractor-map.xml] Oct 01, 2014 12:40:07 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey FINE: Property 'MetExtractorProductCrawler.clientTransferer' set to value [org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory] Oct 01, 2014 12:40:07 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey FINE: Property 'AutoDetectProductCrawler.filemgrUrl' set to value [http://localhost:9000] Oct 01, 2014 12:40:07 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey FINE: Property 'AutoDetectProductCrawler.clientTransferer' set to value [org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory] Oct 01, 2014 12:40:07 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey FINE: Property 'AutoDetectProductCrawler.productPath' set to value [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng] Oct 01, 2014 12:40:07 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey FINE: Property 'StdProductCrawler.filemgrUrl' set to value [http://localhost:9000] Oct 01, 2014 12:40:07 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey FINE: Property 'MetExtractorProductCrawler.filemgrUrl' set to value [http://localhost:9000] Oct 01, 2014 12:40:07 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey FINE: Property 'MetExtractorProductCrawler.productPath' set to value [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng] Oct 01, 2014 12:40:07 PM org.springframework.beans.factory.config.PropertyOverrideConfigurer processKey FINE: Property 'StdProductCrawler.clientTransferer' set to value [org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory] Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.crawl.ProductCrawler crawl INFO: Crawling /homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.crawl.ProductCrawler crawl INFO: Crawling /homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng/2014 Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.crawl.ProductCrawler crawl INFO: Crawling /homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng/2014/002 Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.crawl.ProductCrawler handleFile INFO: Handling file /homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng/2014/002/JEDI_2014002183121000_2014002190215999.sfdu Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.metadata.preconditions.PreCondEvalUtils eval INFO: Passed precondition comparator id CheckThatDataFileSizeIsGreaterThanZero Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.metadata.extractors.ExternMetExtractor extrMetadata INFO: Generating met file for product file: [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng/2014/002/JEDI_2014002183121000_2014002190215999.sfdu] Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.metadata.extractors.ExternMetExtractor extrMetadata INFO: Executing command line: [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/extensions/extractors/jediEngineeringFileExtractor.pl /homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng/2014/002/JEDI_2014002183121000_2014002190215999.sfdu ] with workingDir: [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng/2014/002] to extract metadata OUTPUT: processing file: [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng/2014/002/JEDI_2014002183121000_2014002190215999.sfdu], num_args = 0 OUTPUT: /homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng/2014/002/JEDI_2014002183121000_2014002190215999.sfdu is a sfdu engineering data file Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.metadata.extractors.ExternMetExtractor extrMetadata INFO: Met extraction successful for product file: [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng/2014/002/JEDI_2014002183121000_2014002190215999.sfdu] Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.crawl.ProductCrawler ingest INFO: ProductCrawler: Ready to ingest product: [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng/2014/002/JEDI_2014002183121000_2014002190215999.sfdu]: ProductType: [EngineeringFile] Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.filemgr.ingest.StdIngester setFileManager INFO: StdIngester: connected to file manager: [http://localhost:9000] Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferer setFileManagerUrl INFO: Local Data Transfer to: [http://localhost:9000] enabled Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.filemgr.ingest.StdIngester getProductType WARNING: Unable to obtain product type: [EngineeringFile] from File Manager at: [http://localhost:9000]: Message: java.lang.Exception: java.lang.NullPointerException Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.filemgr.ingest.StdIngester ingest INFO: StdIngester: ingesting product: ProductName: [JEDI_2014002183121000_2014002190215999.sfdu]: ProductType: [EngineeringFile]: FileLocation: [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng/2014/002/] java.lang.NullPointerException at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.runExtractors(XmlRpcFileManager.java:1090) at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.addMetadata(XmlRpcFileManager.java:1065) at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.ingestProduct(XmlRpcFileManager.java:722) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.xmlrpc.Invoker.execute(Invoker.java:130) at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84) at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146) at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139) at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125) at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761) at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642) at java.lang.Thread.run(Thread.java:745) org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@66e0e0e2] : null at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104) at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71) at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73) at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194) at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185) at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178) at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1198) at org.apache.oodt.cas.filemgr.ingest.StdIngester.ingest(StdIngester.java:199) at org.apache.oodt.cas.crawl.ProductCrawler.ingest(ProductCrawler.java:304) at org.apache.oodt.cas.crawl.ProductCrawler.handleFile(ProductCrawler.java:188) at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:108) at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:75) at org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:58) at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) at org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36) Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient ingestProduct SEVERE: Failed to ingest product [org.apache.oodt.cas.filemgr.structs.Product@281de7b2] : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@66e0e0e2] : null -- rolling back ingest Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient ingestProduct SEVERE: Failed to rollback ingest of product [org.apache.oodt.cas.filemgr.structs.Product@281de7b2] : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@66e0e0e2] : null java.lang.Exception: Failed to ingest product [org.apache.oodt.cas.filemgr.structs.Product@281de7b2] : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@66e0e0e2] : null at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.ingestProduct(XmlRpcFileManagerClient.java:1303) at org.apache.oodt.cas.filemgr.ingest.StdIngester.ingest(StdIngester.java:199) at org.apache.oodt.cas.crawl.ProductCrawler.ingest(ProductCrawler.java:304) at org.apache.oodt.cas.crawl.ProductCrawler.handleFile(ProductCrawler.java:188) at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:108) at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:75) at org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:58) at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) at org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36) Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.filemgr.ingest.StdIngester ingest WARNING: exception ingesting product: [JEDI_2014002183121000_2014002190215999.sfdu]: Message: Failed to ingest product [org.apache.oodt.cas.filemgr.structs.Product@281de7b2] : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@66e0e0e2] : null Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.crawl.ProductCrawler ingest WARNING: ProductCrawler: Exception ingesting product: [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng/2014/002/JEDI_2014002183121000_2014002190215999.sfdu]: Message: exception ingesting product: [JEDI_2014002183121000_2014002190215999.sfdu]: Message: Failed to ingest product [org.apache.oodt.cas.filemgr.structs.Product@281de7b2] : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@66e0e0e2] : null: attempting to continue crawling org.apache.oodt.cas.filemgr.structs.exceptions.IngestException: exception ingesting product: [JEDI_2014002183121000_2014002190215999.sfdu]: Message: Failed to ingest product [org.apache.oodt.cas.filemgr.structs.Product@281de7b2] : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Error ingesting product [org.apache.oodt.cas.filemgr.structs.Product@66e0e0e2] : null at org.apache.oodt.cas.filemgr.ingest.StdIngester.ingest(StdIngester.java:204) at org.apache.oodt.cas.crawl.ProductCrawler.ingest(ProductCrawler.java:304) at org.apache.oodt.cas.crawl.ProductCrawler.handleFile(ProductCrawler.java:188) at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:108) at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:75) at org.apache.oodt.cas.crawl.cli.action.CrawlerLauncherCliAction.execute(CrawlerLauncherCliAction.java:58) at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331) at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187) at org.apache.oodt.cas.crawl.CrawlerLauncher.main(CrawlerLauncher.java:36) Oct 01, 2014 12:40:08 PM org.apache.oodt.cas.crawl.ProductCrawler handleFile WARNING: Failed to ingest product: [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging/ops/eng/2014/002/JEDI_2014002183121000_2014002190215999.sfdu]: performing postIngestFail actions Valerie A. Mallder New Horizons Deputy Mission System Engineer The Johns Hopkins University/Applied Physics Laboratory 11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723 240-228-7846 (Office) 410-504-2233 (Blackberry)
