Hi
On Fri, Feb 15, 2013 at 4:24 AM, xiaoyan yu <[email protected]> wrote:
> I realized that it would take a while for the index initialization. Now the
> entityhub works with the big dbpedia.
> However, I don't understand why the extracted index under
> stanbol\indexes\default is only 20G. I extracted the dbpedia.index.zip by
> unzip, resulting 40G data. And I did observe more than 20G index during the
> initialization.
I never had a detailed look at what exactly happens during the Index
initialization. I can tell you that the initialization first extracts
all files from the provided solrindex.zip file to the target directory
and second uses the Solr Java API to manually add a new SolrCore to
the Solr CoreContainer. While the later will add some new files (e.g.
the lock file) I would not expect that the files size does change a
lot.
The Entityhub Indexing Tool uses the Java ZIP implementation for
creating the ZIP archive. Last week I failed to extract a
dbpedia.solrindex.zip file by using unzip on an Ubuntu box even so -
based on the documentation - the installed version should have
supported archives with that size. In my case the files created by
unzip where corrupted (only ~20GByte of the expected 30G). But as the
index was successfully initialized by Apache Stanbol I did not further
investigate the reason for that. So this might be a reason for the
discrepancy in sizes.
An other possibility is that the Solr Index optimization at the end of
the indexing process failed. During optimization the size of the index
grows up to ~200% of the original size. If the optimization fails the
resulting solrindex.zip file may also contain the temporary files of
the failed optimization. When you install such an index than Solr will
complete the unfinished optimization. If successful the size of the
index could shrink to about half the size. You can easily check of a
solrindex.zip file contains an unoptimized index by counting the
number of files in the index directory. Any number much greater than
20 files indicates that the index was not optimized.
> Please shed light on this.
>
> Please also help on the configuration exception as listed in step 4 in the
> last email.
Never seen an Exception like that. However It looks like as the
configuration for the Apache Sling Logging Services that ensures only
WARN level loggings for all "org.apache.solr" packages got somehow
duplicated. You can edit such configuration by using the Configuration
tab of the Apache Felix Webconsole.
best
Rupert
>
> Thank
> Xiaoyan
>
> On Thursday, February 14, 2013, xiaoyan yu wrote:
>
>> Thanks for your help.
>> Here is what I did.
>> 1) shutdown stanbol
>> 2) remove the stanbol/indexes/default/dbpedia-{today} folder
>> 3) create a symlink to dbpedia.solrindex.zip under stanbol/datafiles
>> 3) find the dbpedia.solrindex.ref file under the stanbol working directory
>> and delete it.
>> 4) start stanbol and got errors though got "startup completed" at last
>> *ERROR* [org.osgi.service.cm.ManagedServiceFactory, id=32, bundle=93]:
>> Updating configuration property org.apache.sling.commons.log.names caused a
>> problem: Category org.apache.solr already defined by configuration
>> org.apache.sling.commons.log.LogManager.factory.config.b1f5ba91-406a-4adf-8d8c-4e22f8b202e3
>> org.osgi.service.cm.ConfigurationException:
>> org.apache.sling.commons.log.names : Category org.apache.solr already
>> defined by configuration
>> org.apache.sling.commons.log.LogManager.factory.config.b1f5ba91-406a-4adf-8d8c-4e22f8b202e3
>> at
>> org.apache.sling.commons.log.internal.config.LoggerManagedServiceFactory.updated(LoggerManagedServiceFactory.java:39)
>> at
>> org.apache.felix.cm.impl.ConfigurationManager$ManagedServiceFactoryUpdate.run(ConfigurationManager.java:1366)
>> at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:104)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by:
>> org.apache.sling.commons.log.internal.config.ConfigurationException:
>> at
>> org.apache.sling.commons.log.internal.slf4j.LogConfigManager.updateLoggerConfiguration(LogConfigManager.java:453)
>> at
>> org.apache.sling.commons.log.internal.config.LoggerManagedServiceFactory.updated(LoggerManagedServiceFactory.java:37)
>> ... 3 more
>> 5) a small index (less than 1G) is generated under the
>> stanbol/indexes/default/dbpedia-{today} folder
>> 6) when I did
>> curl "
>> http://localhost:8080/entityhub/site/dbpedia/entity?id=http://dbpedia.org/resource/Paris";,
>> once again I got the exception
>> "org.apache.stanbol.entityhub.servicesapi.yard.YardException: The
>> SolrIndex 'dbpedia' for SolrYard 'dbpedia default data index' is currently
>> not active!"
>> 7) search for the dbpedia.solrindex.ref file again and the content is like
>> "
>> #Thu Feb 14 08:44:48 PST 2013
>> Index-Name=dbpedia
>> Archive=dbpedia.solrindex.zip
>> State=UNINITIALISED
>> Index-Archive=dbpedia.solrindex.zip
>> Server-Name=default
>> "
>>
>> Please help again.
>>
>> Thanks,
>> Xiaoyan
>>
>>
>> On Tue, Feb 12, 2013 at 12:05 AM, Rupert Westenthaler <
>> [email protected] <javascript:_e({}, 'cvml',
>> '[email protected]');>> wrote:
>>
>>> HI
>>>
>>> > So my question is whether I MUST keep the ~12G zip file under datafiles
>>> > folder.
>>>
>>> You can delete it as soon as the initialization has completed - what
>>> was not the case because of the "IOException: No space left on
>>> device".
>>>
>>> I usually do only create a symlink for the {name}.solrindex.zip file
>>> in the stanbol/datafiles directory instead of physically copying the
>>> file in that directory. You can even (1) start Stanbol go init the
>>> default index (2) shutdown Stanbol (3) delete the
>>> /stanbol/indexes/default/dbpedia-{date} folder and (4) create a
>>> symlink to an other location with the already extracted
>>> {name}.solrindex.zip.
>>>
>>>
>>> BTW: to force the reinitialization after an error like that you might
>>> need to delete the "dbpedia.solrindex.ref" file (just use 'find .
>>> -name "dbpedia.solrindex.ref"' in the stanbol working dir to find it).
>>> Make sure that Stanbol is not running when you do this.
>>>
>>> best
>>> Rupert
>>>
>>> On Tue, Feb 12, 2013 at 12:59 AM, xiaoyan yu
>>> <[email protected]<javascript:_e({}, 'cvml', '[email protected]');>>
>>> wrote:
>>> > I'd like to use a large dbpedia dataset with entityHub in my vm with
>>> ~50G
>>> > disk space.
>>> > I created one (~12G) by following
>>> >
>>> https://github.com/apache/stanbol/tree/trunk/entityhub/indexing/dbpedia#entity-typesand
>>> > put it under the datafiles folder.
>>> > After updating the dbpedia bundle from the admin console, I am informed
>>> of
>>> > no enough disk space by the system.
>>> > It seems that another ~26G index is automatically created under my
>>> launcher
>>> > folder stanbol/indexes/default.
>>> >
>>> > Now I deleted the ~12G zip file from the datafiles folder in hopes that
>>> > stanbol won't use it once ~26G index is generated.
>>> >
>>> > However, when I did
>>> > curl "
>>> >
>>> http://localhost:8080/entityhub/site/dbpedia/entity?id=http://dbpedia.org/resource/Paris
>>> > ",
>>> > I got the exception
>>> > "org.apache.stanbol.entityhub.servicesapi.yard.YardException: The
>>> SolrIndex
>>> > 'dbpedia' for SolrYard 'dbpedia default data index' is currently not
>>> > active!"
>>> >
>>> > So my question is whether I MUST keep the ~12G zip file under datafiles
>>> > folder.
>>> >
>>> >
>>> > Thanks,
>>> > Xiaoyan
>>>
>>>
>>>
>>> --
>>> | Rupert Westenthaler
>>> [email protected]<javascript:_e({}, 'cvml',
>>> '[email protected]');>
>>> | Bodenlehenstraße 11 ++43-699-11108907
>>> | A-5500 Bischofshofen
>>>
>>
>>
--
| Rupert Westenthaler [email protected]
| Bodenlehenstraße 11 ++43-699-11108907
| A-5500 Bischofshofen
