On Thu, Feb 06, 2020 at 02:50:43PM +0200, Alan Orth wrote: > Our yearly Solr statistics sharding (stats-util -s) failed this year > because our core is very large (43GiB) and apparently timed out somewhere. > It failed again when I tried to run it manually: > > Moving: 51633080 into core statistics-2019 > ... > Exception: Read timed out > java.net.SocketTimeoutException: Read timed out > > As a test I used this really great tool called solr-import-export-json to > export some of my 2019 statistics to JSON on the production server, then > import them into a new core in my development instance: > > $ ./run.sh -s http://localhost:8081/solr/statistics -a export -o > /tmp/statistics-2019-01.json -f 'dateYearMonth:2019-01' -k uid > $ curl ' > http://localhost:8080/solr/admin/cores?action=CREATE&name=statistics-2019&instanceDir=/home/aorth/dspace/solr/statistics&dataDir=/home/aorth/dspace/solr/statistics-2019/data > ' > $ ./run.sh -s http://localhost:8080/solr/statistics-2019 -a import -o > /tmp/statistics-2019-01.json -k uid > > This worked brilliantly... in fact I am very impressed with this tool and > recommend it to people! > > The problem is, this core does not get enumerated automatically by Solr > after I restart the servlet container. I got it to load by hard-coding the > core into dspace/solr/solr.xml config² but it seems hacky. How are these > core shards enumerated by DSpace's Solr application? What would cause > shards to not be loaded automatically? > > My environment is DSpace 5.8 with Tomcat 7.0.99 and OpenJDK 8.
I think that a good place to look is 'dspace-api/src/main/java/org/dspace/statistics/SolrLoggerServiceImpl#initSolrYearCores'. Also #createCore in the same class. This is where DSpace enumerates the cores that it will use for statistics. It seems to be looking for directories 'solr/statistics-YYYY'. It will call CREATE in Solr's CoreAdmin API, which would seem to register a core if it already exists. You seem to be doing the same thing, but there must be something slightly different about your actions. Or perhaps the way you are testing -- it looks to me as though Solr is unaware of the additional cores at startup and is told of them by DSpace when *it* starts up. But I think it is actually DSpace that is doing something hacky: using the same InstanceDir for multiple cores. I have no idea why that works. Sadly, SolrJ is almost entirely undocumented, at least in this area. I have had to puzzle out a lot of its working by reference to the web API documentation in the Solr Ref Guide. -- Mark H. Wood Lead Technology Analyst University Library Indiana University - Purdue University Indianapolis 755 W. Michigan Street Indianapolis, IN 46202 317-274-0749 www.ulib.iupui.edu -- All messages to this mailing list should adhere to the DuraSpace Code of Conduct: https://duraspace.org/about/policies/code-of-conduct/ --- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/20200206150143.GF11530%40IUPUI.Edu.
signature.asc
Description: PGP signature
