Thanks for the detailed explanation! Things make a lot of sense now. Thanks, Xiaoyan
On Mon, Feb 18, 2013 at 2:10 AM, Rupert Westenthaler < [email protected]> wrote: > Hi > > On Fri, Feb 15, 2013 at 4:24 AM, xiaoyan yu <[email protected]> wrote: > > I realized that it would take a while for the index initialization. Now > the > > entityhub works with the big dbpedia. > > However, I don't understand why the extracted index under > > stanbol\indexes\default is only 20G. I extracted the dbpedia.index.zip by > > unzip, resulting 40G data. And I did observe more than 20G index during > the > > initialization. > > I never had a detailed look at what exactly happens during the Index > initialization. I can tell you that the initialization first extracts > all files from the provided solrindex.zip file to the target directory > and second uses the Solr Java API to manually add a new SolrCore to > the Solr CoreContainer. While the later will add some new files (e.g. > the lock file) I would not expect that the files size does change a > lot. > > The Entityhub Indexing Tool uses the Java ZIP implementation for > creating the ZIP archive. Last week I failed to extract a > dbpedia.solrindex.zip file by using unzip on an Ubuntu box even so - > based on the documentation - the installed version should have > supported archives with that size. In my case the files created by > unzip where corrupted (only ~20GByte of the expected 30G). But as the > index was successfully initialized by Apache Stanbol I did not further > investigate the reason for that. So this might be a reason for the > discrepancy in sizes. > > An other possibility is that the Solr Index optimization at the end of > the indexing process failed. During optimization the size of the index > grows up to ~200% of the original size. If the optimization fails the > resulting solrindex.zip file may also contain the temporary files of > the failed optimization. When you install such an index than Solr will > complete the unfinished optimization. If successful the size of the > index could shrink to about half the size. You can easily check of a > solrindex.zip file contains an unoptimized index by counting the > number of files in the index directory. Any number much greater than > 20 files indicates that the index was not optimized. > > > Please shed light on this. > > > > Please also help on the configuration exception as listed in step 4 in > the > > last email. > > Never seen an Exception like that. However It looks like as the > configuration for the Apache Sling Logging Services that ensures only > WARN level loggings for all "org.apache.solr" packages got somehow > duplicated. You can edit such configuration by using the Configuration > tab of the Apache Felix Webconsole. > > best > Rupert > > > > > Thank > > Xiaoyan > > > > On Thursday, February 14, 2013, xiaoyan yu wrote: > > > >> Thanks for your help. > >> Here is what I did. > >> 1) shutdown stanbol > >> 2) remove the stanbol/indexes/default/dbpedia-{today} folder > >> 3) create a symlink to dbpedia.solrindex.zip under stanbol/datafiles > >> 3) find the dbpedia.solrindex.ref file under the stanbol working > directory > >> and delete it. > >> 4) start stanbol and got errors though got "startup completed" at last > >> *ERROR* [org.osgi.service.cm.ManagedServiceFactory, id=32, bundle=93]: > >> Updating configuration property org.apache.sling.commons.log.names > caused a > >> problem: Category org.apache.solr already defined by configuration > >> > org.apache.sling.commons.log.LogManager.factory.config.b1f5ba91-406a-4adf-8d8c-4e22f8b202e3 > >> org.osgi.service.cm.ConfigurationException: > >> org.apache.sling.commons.log.names : Category org.apache.solr already > >> defined by configuration > >> > org.apache.sling.commons.log.LogManager.factory.config.b1f5ba91-406a-4adf-8d8c-4e22f8b202e3 > >> at > >> > org.apache.sling.commons.log.internal.config.LoggerManagedServiceFactory.updated(LoggerManagedServiceFactory.java:39) > >> at > >> > org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.run(ConfigurationManager.java:1366) > >> at > org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:104) > >> at java.lang.Thread.run(Thread.java:662) > >> Caused by: > >> org.apache.sling.commons.log.internal.config.ConfigurationException: > >> at > >> > org.apache.sling.commons.log.internal.slf4j.LogConfigManager.updateLoggerConfiguration(LogConfigManager.java:453) > >> at > >> > org.apache.sling.commons.log.internal.config.LoggerManagedServiceFactory.updated(LoggerManagedServiceFactory.java:37) > >> ... 3 more > >> 5) a small index (less than 1G) is generated under the > >> stanbol/indexes/default/dbpedia-{today} folder > >> 6) when I did > >> curl " > >> > http://localhost:8080/entityhub/site/dbpedia/entity?id=http://dbpedia.org/resource/Paris > ", > >> once again I got the exception > >> "org.apache.stanbol.entityhub.servicesapi.yard.YardException: The > >> SolrIndex 'dbpedia' for SolrYard 'dbpedia default data index' is > currently > >> not active!" > >> 7) search for the dbpedia.solrindex.ref file again and the content is > like > >> " > >> #Thu Feb 14 08:44:48 PST 2013 > >> Index-Name=dbpedia > >> Archive=dbpedia.solrindex.zip > >> State=UNINITIALISED > >> Index-Archive=dbpedia.solrindex.zip > >> Server-Name=default > >> " > >> > >> Please help again. > >> > >> Thanks, > >> Xiaoyan > >> > >> > >> On Tue, Feb 12, 2013 at 12:05 AM, Rupert Westenthaler < > >> [email protected] <javascript:_e({}, 'cvml', > >> '[email protected]');>> wrote: > >> > >>> HI > >>> > >>> > So my question is whether I MUST keep the ~12G zip file under > datafiles > >>> > folder. > >>> > >>> You can delete it as soon as the initialization has completed - what > >>> was not the case because of the "IOException: No space left on > >>> device". > >>> > >>> I usually do only create a symlink for the {name}.solrindex.zip file > >>> in the stanbol/datafiles directory instead of physically copying the > >>> file in that directory. You can even (1) start Stanbol go init the > >>> default index (2) shutdown Stanbol (3) delete the > >>> /stanbol/indexes/default/dbpedia-{date} folder and (4) create a > >>> symlink to an other location with the already extracted > >>> {name}.solrindex.zip. > >>> > >>> > >>> BTW: to force the reinitialization after an error like that you might > >>> need to delete the "dbpedia.solrindex.ref" file (just use 'find . > >>> -name "dbpedia.solrindex.ref"' in the stanbol working dir to find it). > >>> Make sure that Stanbol is not running when you do this. > >>> > >>> best > >>> Rupert > >>> > >>> On Tue, Feb 12, 2013 at 12:59 AM, xiaoyan yu > >>> <[email protected]<javascript:_e({}, > 'cvml', '[email protected]');>> > >>> wrote: > >>> > I'd like to use a large dbpedia dataset with entityHub in my vm with > >>> ~50G > >>> > disk space. > >>> > I created one (~12G) by following > >>> > > >>> > https://github.com/apache/stanbol/tree/trunk/entityhub/indexing/dbpedia#entity-typesand > >>> > put it under the datafiles folder. > >>> > After updating the dbpedia bundle from the admin console, I am > informed > >>> of > >>> > no enough disk space by the system. > >>> > It seems that another ~26G index is automatically created under my > >>> launcher > >>> > folder stanbol/indexes/default. > >>> > > >>> > Now I deleted the ~12G zip file from the datafiles folder in hopes > that > >>> > stanbol won't use it once ~26G index is generated. > >>> > > >>> > However, when I did > >>> > curl " > >>> > > >>> > http://localhost:8080/entityhub/site/dbpedia/entity?id=http://dbpedia.org/resource/Paris > >>> > ", > >>> > I got the exception > >>> > "org.apache.stanbol.entityhub.servicesapi.yard.YardException: The > >>> SolrIndex > >>> > 'dbpedia' for SolrYard 'dbpedia default data index' is currently not > >>> > active!" > >>> > > >>> > So my question is whether I MUST keep the ~12G zip file under > datafiles > >>> > folder. > >>> > > >>> > > >>> > Thanks, > >>> > Xiaoyan > >>> > >>> > >>> > >>> -- > >>> | Rupert Westenthaler > >>> [email protected]<javascript:_e({}, > 'cvml', '[email protected]');> > >>> | Bodenlehenstraße 11 ++43-699-11108907 > >>> | A-5500 Bischofshofen > >>> > >> > >> > > > > -- > | Rupert Westenthaler [email protected] > | Bodenlehenstraße 11 ++43-699-11108907 > | A-5500 Bischofshofen >
