Hi, I’m in the process of moving from v1.6 to v3.1 – I have a v3.1 DEV system up and running that contains an up to date/upgraded copy of my v1.6 PROD database/assetstore and I am now starting to look at SOLR statistics on my v3.1 DEV system.
SOLR stats appear to be working on the DEV system (I can see stats for various items that I’ve been accessing during the implementation of the DEV system) and now I want to investigate importing historical logs. I have studiously kept Production log files from way back when (from pre and post v1.6) for the day when I could use these for generating stats, and I’m now trying to import some of these into the SOLR statistics on my DEV system (having first deleted /usr/local/dspace/solr/statistics/data in order to start with a clean slate). I started with some pre v1.6 “old format” files (with the naming style dspace.log.1, dspace.log.2 etc) – I just took the last 101 files (i.e. dspace.log up to dspace.log.100) as a test – they seem to convert OK using: /usr/local/dspace/bin/dspace stats-log-converter -i /usr/local/dspace/import-logs/dspace.log -o /usr/local/dspace/converted-logs/dspace-log.out -m - but when I then try and import these using: /usr/local/dspace/bin/dspace stats-log-importer -i /usr/local/dspace/converted-logs/dspace-log.out -m - I get a NullPointerException (running with –v as user tomcat7 which DSpace runs as and which appears to own all relevant files/directories): -------------------------------------------------------------------------------- tomcat7@dspace3dev:/usr/local/dspace$ /usr/local/dspace/bin/dspace stats-log-importer -i /usr/local/dspace/converted-logs/dspace-log.out -m -v INFO [main] (DSpaceKernelInit.java:52) - Created new kernel: DSpaceKernel:org.dspace:name=1ddd555d-dc42-4826-9f9f-d411894a155a,type=DSpaceKernel:lastLoad=null:loadTime=0:running=false:kernel=null INFO [main] (ConfigurationManager.java:1224) - Loading from classloader: file:/usr/local/dspace/config/dspace.cfg INFO [main] (ConfigurationManager.java:1224) - Using dspace provided log configuration (log.init.config) INFO [main] (ConfigurationManager.java:1224) - Loading: /usr/local/dspace/config/log4j.properties Writing to solr server at: http://localhost:8080/solr/statistics dspace-log.out.46 Processing file: /usr/local/dspace/converted-logs/dspace-log.out.46 Line:20101218052459641,view_community,7,2010-12-18T05:24:59,anonymous,77.88.25.28 ip addr = 77.88.25.28, dns name = 77.88.25.28, country = Russian Federation, city = null Line:20101218052540724,view_item,12598,2010-12-18T05:25:40,anonymous,207.46.13.143 ip addr = 207.46.13.143, dns name = msnbot-207-46-13-143.search.msn.com., IGNORE (search engine) Line:20101218052542281,view_bitstream,5602,2010-12-18T05:25:42,anonymous,66.249.72.131 ip addr = 66.249.72.131, dns name = crawl-66-249-72-131.googlebot.com., IGNORE (search engine) Line:20101218052543694,view_item,2043,2010-12-18T05:25:43,anonymous,207.46.13.138 ip addr = 207.46.13.138, dns name = msnbot-207-46-13-138.search.msn.com., IGNORE (search engine) Line:20101218052607357,view_item,6491,2010-12-18T05:26:07,anonymous,207.46.204.218 ip addr = 207.46.204.218, dns name = msnbot-207-46-204-218.search.msn.com., IGNORE (search engine) Line:20101218052608909,view_item,310,2010-12-18T05:26:08,anonymous,207.46.204.241 ip addr = 207.46.204.241, dns name = msnbot-207-46-204-241.search.msn.com., IGNORE (search engine) Line:20101218052709696,view_community,10,2010-12-18T05:27:09,anonymous,207.46.13.88 ip addr = 207.46.13.88, dns name = msnbot-207-46-13-88.search.msn.com., IGNORE (search engine) Line:20101218052709742,view_bitstream,722,2010-12-18T05:27:09,anonymous,207.46.13.88 ip addr = 207.46.13.88, dns name = msnbot-207-46-13-88.search.msn.com., IGNORE (search engine) Line:20101218052712453,view_bitstream,3065,2010-12-18T05:27:12,anonymous,207.46.13.136 ip addr = 207.46.13.136, dns name = msnbot-207-46-13-136.search.msn.com., IGNORE (search engine) Line:20101218052737888,view_bitstream,5113,2010-12-18T05:27:37,anonymous,66.249.72.131 ip addr = 66.249.72.131, dns name = crawl-66-249-72-131.googlebot.com., IGNORE (search engine) Line:20101218052739293,view_item,6480,2010-12-18T05:27:39,anonymous,66.249.72.131 ip addr = 66.249.72.131, dns name = crawl-66-249-72-131.googlebot.com., IGNORE (search engine) Line:20101218052813431,view_item,16863,2010-12-18T05:28:13,anonymous,207.46.195.233 ip addr = 207.46.195.233, dns name = msnbot-207-46-195-233.search.msn.com., IGNORE (search engine) Line:20101218052835476,view_bitstream,6156,2010-12-18T05:28:35,anonymous,66.249.72.131 ip addr = 66.249.72.131, dns name = crawl-66-249-72-131.googlebot.com., IGNORE (search engine) Line:20101218052916973,view_bitstream,6654,2010-12-18T05:29:16,anonymous,208.68.143.203 ip addr = 208.68.143.203, dns name = 208.68.143.203, country = United States, city = Needham Heights - DSO with ID '6654' is no longer in the system Line:20101218052933501,view_bitstream,4502,2010-12-18T05:29:33,anonymous,66.249.72.131 ip addr = 66.249.72.131, dns name = crawl-66-249-72-131.googlebot.com., IGNORE (search engine) Line:20101218053013210,view_item,66,2010-12-18T05:30:13,anonymous,207.46.13.99 ip addr = 207.46.13.99, dns name = msnbot-207-46-13-99.search.msn.com., IGNORE (search engine) Line:20101218053031328,view_bitstream,3698,2010-12-18T05:30:31,anonymous,66.249.72.131 ip addr = 66.249.72.131, dns name = crawl-66-249-72-131.googlebot.com., IGNORE (search engine) Line:20101218053109278,view_item,3208,2010-12-18T05:31:09,anonymous,207.46.195.240 ip addr = 207.46.195.240, dns name = msnbot-207-46-195-240.search.msn.com., IGNORE (search engine) Line:20101218053110356,view_bitstream,1553,2010-12-18T05:31:10,anonymous,77.88.25.28 ip addr = 77.88.25.28, dns name = 77.88.25.28, country = Russian Federation, city = null - DSO with ID '1553' is no longer in the system Line:20101218053129209,view_bitstream,5434,2010-12-18T05:31:29,anonymous,66.249.72.131 ip addr = 66.249.72.131, dns name = crawl-66-249-72-131.googlebot.com., IGNORE (search engine) Line:20101218053131785,view_bitstream,2502,2010-12-18T05:31:31,anonymous,173.212.255.198 ip addr = 173.212.255.198, dns name = secureserver.regencyhost.net., country = United States, city = Scranton - DSO with ID '2502' is no longer in the system Line:20101218053218575,view_item,2551,2010-12-18T05:32:18,anonymous,66.249.72.131 ip addr = 66.249.72.131, dns name = crawl-66-249-72-131.googlebot.com., IGNORE (search engine) Line:20101218053220348,view_item,7227,2010-12-18T05:32:20,anonymous,66.249.72.131 ip addr = 66.249.72.131, dns name = crawl-66-249-72-131.googlebot.com., IGNORE (search engine) Line:20101218053221250,view_item,1249,2010-12-18T05:32:21,anonymous,66.249.72.131 ip addr = 66.249.72.131, dns name = crawl-66-249-72-131.googlebot.com., IGNORE (search engine) Line:20101218053222148,view_item,9963,2010-12-18T05:32:22,anonymous,66.249.72.131 ip addr = 66.249.72.131, dns name = crawl-66-249-72-131.googlebot.com., IGNORE (search engine) Line:20101218053223040,view_item,5869,2010-12-18T05:32:23,anonymous,66.249.72.131 ip addr = 66.249.72.131, dns name = crawl-66-249-72-131.googlebot.com., IGNORE (search engine) Line:20101218053226421,view_bitstream,6006,2010-12-18T05:32:26,anonymous,66.249.72.131 ip addr = 66.249.72.131, dns name = crawl-66-249-72-131.googlebot.com., IGNORE (search engine) Line:20101218053231010,view_bitstream,137,2010-12-18T05:32:31,anonymous,137.189.89.53 ip addr = 137.189.89.53, dns name = pc89053.cse.cuhk.edu.hk., country = Hong Kong, city = Central District Line:20101218053313952,view_bitstream,171,2010-12-18T05:33:13,anonymous,122.174.134.164 ip addr = 122.174.134.164, dns name = ABTS-TN-dynamic-164.134.174.122.airtelbroadband.in., country = India, city = Chennai Line:20101218053314559,view_item,11506,2010-12-18T05:33:14,anonymous,77.88.25.28 ip addr = 77.88.25.28, dns name = 77.88.25.28, country = Russian Federation, city = null Exception: null java.lang.NullPointerException at org.dspace.statistics.util.StatisticsImporter.load(StatisticsImporter.java:343) at org.dspace.statistics.util.StatisticsImporter.main(StatisticsImporter.java:505) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:183) -------------------------------------------------------------------------------- Looking at the “item” that is causing the exception (11506), there does appear to be an item with this value as an Item ID and Handle (not sure which is being referred to in the logs), so the Item itself appears to be there . . . My other thought is that the importer processes a number of lines and then commits them in a batch, and it is when it is getting to this "commit" stage that it is falling over? (but there does appear to be "stuff" appearing in /usr/local/dspace/solr/statistics/data?) I also tried turning up the log level to DEBUG, but not seeing anything that looks relevant to me (these are the last few lines and everything above looks pretty similar!): -------------------------------------------------------------------------------- 2013-09-23 12:01:14,718 DEBUG org.dspace.storage.rdbms.DatabaseManager @ Running query "select * from item where item_id = ? " with parameters: 11506 2013-09-23 12:01:14,720 DEBUG org.dspace.content.Item @ anonymous::find_item:item_id=11506 2013-09-23 12:01:14,720 DEBUG org.dspace.storage.rdbms.DatabaseManager @ Running query "SELECT * FROM Handle WHERE resource_type_id = ? AND resource_id = ?" with parameters: 2,11506 2013-09-23 12:01:14,720 DEBUG org.dspace.storage.rdbms.DatabaseManager @ Running query "select * from eperson where email = ? " with parameters: anonymous -------------------------------------------------------------------------------- Anyone have any ideas as to why this isn’t working? Could it be because I’m importing PROD logs into a DEV system (even though it is an up to date copy of PROD)? Am I doing anything obviously wrong/stupid? Any pointers, suggestions, insights etc more than welcome as I’m a bit stuck just now :-( Cheers, Mike Michael White eLearning Liaison and Development (eLD) Information Services S8, Library University of Stirling Stirling SCOTLAND FK9 4LA Email: [email protected] Tel: +44 (0) 1786 466877 Fax: +44 (0) 1786 466880 http://www.stir.ac.uk/is/staff/about/teams/aldt/#eld -- The University of Stirling has been ranked in the top 12 of UK universities for graduate employment*. 94% of our 2012 graduates were in work and/or further study within six months of graduation. *The Telegraph The University of Stirling is a charity registered in Scotland, number SC 011159. ------------------------------------------------------------------------------ LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99! 1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint 2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13. http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk _______________________________________________ DSpace-tech mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
