Thanks Rupert! The genericrfd reindexing worked. Just one thing : it seems kind of odd that my solrindex.zip got from 796MB (after dbpedia indexing) to 1,5GB (after genericrdf indexing based on dbpedia index) but my yago_class_labels.nt file contains around 100,000 entries. The only thing I changed in config was the name of the site as you suggested and in mappings.txt file I removed everything except "rdfs:label".
2014-05-26 16:26 GMT+03:00 Rupert Westenthaler < rupert.westentha...@gmail.com>: > Hi Cristian, > > On Mon, May 26, 2014 at 2:33 PM, Cristian Petroaca > <cristian.petro...@gmail.com> wrote: > > I just found out that according to > > > http://svn.apache.org/repos/asf/stanbol/trunk/entityhub/indexing/dbpedia/README.mdthe > > min-score can actually be set to 0 and all entities will be indexed > > :). > > So, I'll give that a go ( hopefully my dbpedia index won't become > gigantic > > in size). > > > > Even if you set the value to zero it will still only index entities > listed in the incoming_links.txt file. So you will need to append the > Yago types to that file. > > An other possibility would be to first create the dbpedia index and > after that append the Yago classes by using the generic rdf indexing > tool. For that you can > > 1) take the destination folder of the dbpedia indexing tool and link > (or move) it to the destination of the generic indexing tool. > 2) make sure to configure the same site name as for the dbpedia index > tool to the generic indexing tool > 3) add the RDF data of the Yago classes to the rdf data folder of the > generic indexing tool > 4) adapt all the other configurations as needed > 5) start the indexing process. > > The generic indexing tool will check if the target solr index does > already exist. As it is present it will just add the additional > entities to the solr core. > > When the process completes you can use the "solrindex.zip" file > generated by the generic RDF indexing tool together with the OSGI > bunlde (the jar file) generated by the dbpedia indexing tool. > > Especially if you have already created an dbpedia index I would > recommend you to try this out as it would avoid re-indexing the whole > dbpedia data again. > > best > Rupert > > > > > > > 2014-05-25 16:58 GMT+03:00 Cristian Petroaca < > cristian.petro...@gmail.com>: > > > >> Hi Rupert. > >> > >> I'm answering to your suggestions on integrating the yago class labels > in > >> the dbpedia index in this thread since it's a lot shorter than the other > >> one. > >> > >> For clarity, your suggestions were : > >> > >> "1. The indexing tool does support LDPath. That means you can import > >> all the required RDF files and use LDPath to append the labels of the > Yago > >> Types directly to the dbpedia entities. This would prevent additional > >> lookups to retrieve the types, but also increase the size of the index a > >> lot. 2. You could also index the Yago Types and use an additional > Entityhub > >> lookup to retrieve them. In this case you should first collect all types > >> referenced by Entities in the processed text and in a second step > retrieve > >> the labels. While this means additional lookups it will only load the > >> labels for an type once. In addition you could use a cache for types. 3. > >> Your engine could use LDPath to retrieve the types. This would require > to > >> index the data like with option (2) and use a LDPath statement similar > to > >> (1). It would be the slowest solution (as it requires an additional > lookup > >> for every extracted entity) but require the least code." > >> > >> It seems that the best solution would be no 2, so I took that path. But > >> I'm having some issues with building the dbpedia index with the yago > class > >> labels. > >> > >> I managed to create an .nt file from the data files on the yago site > which > >> contains the yago class labels. The file has this format : > >> <http://dbpedia.org/class/yago/Floret111669786> < > >> http://www.w3.org/2000/01/rdf-schema#label> "floret"@en . > >> <http://dbpedia.org/class/yago/Servant110582154> < > >> http://www.w3.org/2000/01/rdf-schema#label> "retainer"@en . > >> <http://dbpedia.org/class/yago/Varietal107900225> < > >> http://www.w3.org/2000/01/rdf-schema#label> "varietal"@en . > >> > >> I compressed this to a .bz2 archive and put it in the > >> indexing/resources/rdfdata folder with the rest of them. > >> > >> After running the indexer I got my dbpedia index but it seems the yago > >> class labels are not present in the index. The first clue was that they > >> were missing from the indexing/destination/indexed-entities-ids archive. > >> Second confirmation came when I tried to retrieve a yago class label by > >> calling site.getEntity(yago_class_uri) and the return was null. I should > >> mention that the same call works if I want to get a > >> http://dbpedia.org/resource/[id] entity. > >> > >> From what I saw, the indexing process indexes entities only if they are > in > >> the incoming_links.txt file and only if their score is higher than 2 so > I > >> guess that's the point where the yago classes were not inserted. From > >> looking at the code, the min-score parameter from the minincoming.config > >> file cannot be set to 0, or something that would ignore the > >> incoming_links.txt ranking and just index everything. So, in this > >> situation, is there a solution for getting these yago classes as > entities > >> in the index? > >> > >> I'd like to mention that the indexing process did correctly read the > >> yago_class_labels.nt file and started to index the entities into Jena. > >> > >> Thanks, > >> Cristian > >> > >> > >> > >> 2014-05-07 14:54 GMT+03:00 Cristian Petroaca < > cristian.petro...@gmail.com> > >> : > >> > >> Hi Rupert, > >>> > >>> Ok, I'll resend this mail in this thread. Again, out of habit I sent it > >>> in the gigantic "Named entities coreference" thread instead. > >>> > >>> So, I managed to create a dbpedia index with the yago class information > >>> but looking into the yago_types.nt file which assigns yago classes to > >>> dbpedia entities I realized that there are no yago class labels > present, I > >>> just have the class uri like : < > >>> http://dbpedia/..something../President1829302/. I also need the class > >>> labels so that I can compare them to the noun token's string from the > text. > >>> > >>> I can get the labels from one of the yago downloads here : > >>> > http://www.mpi-inf.mpg.de/yago-naga/yago/download/yago/yagoMultilingualClassLabels.txt > . > >>> I'll need another yago download file to map the yago wordnet classes to > >>> dbpedia uris. That could be done via a script maybe. > >>> > >>> Once I have the dbpedia_yago_class_uri -> label file is it possible to > >>> integrate this data in the dbpedia index and later be able to query the > >>> labels from the 'dbpedia' Site? How would that work in the dbpedia > indexing > >>> process? What should I change in the mappings.txt file? At first > glance it > >>> seems that the indexing is done based on the incoming_links.txt entity > >>> scoring and in my case I don't want to include triples involving the > actual > >>> entity but triples invloving a property of the entity (its yago class). > >>> > >>> Other than that, I saw that someone will be working on integrating YAGO > >>> as part of Gsoc 2014. So maybe waiting for that is an option too but I > >>> don't know what the extent of the integration will be. > >>> > >>> Thanks, > >>> Cristi > >>> > >>> > >>> 2014-04-30 12:04 GMT+03:00 Rupert Westenthaler < > >>> rupert.westentha...@gmail.com>: > >>> > >>> On Wed, Apr 30, 2014 at 10:37 AM, Cristian Petroaca > >>>> <cristian.petro...@gmail.com> wrote: > >>>> > Hi All, > >>>> > > >>>> > I'm currently working on > >>>> https://issues.apache.org/jira/browse/STANBOL-1279. > >>>> > > >>>> > I am using the SiteManager to get a Site with referenceId = > "dbpedia" > >>>> and > >>>> > am querying data related to some NERs (querying by NER label and > type). > >>>> > This works and I do get results from the dbpedia index. > >>>> > > >>>> > What I want to do is this : > >>>> > > >>>> > 1. I want to be able to store and get yago class types in the > dbpedia > >>>> data. > >>>> > This data is stored in the yago-types.nt file from the dbpedia 3.9 > >>>> > downloads. Is it possible to create a new dbpedia index with the 3.9 > >>>> files > >>>> > using this script > >>>> > > >>>> > https://svn.apache.org/repos/asf/stanbol/trunk/entityhub/indexing/dbpedia/dbpedia-3.8/fetch_data_en_int.sh > >>>> > ? > >>>> > >>>> yep. Just make suer you change > >>>> > >>>> DBPEDIA=http://downloads.dbpedia.org/3.8 > >>>> > >>>> to dbpedia 3.9 > >>>> > >>>> BTW: you can also remove > >>>> > >>>> #corrects encoding and recompress using gz > >>>> bzcat ${filename}.bz2 \ > >>>> | sed 's/\\\\/\\u005c\\u005c/g;s/\\\([^u"]\)/\\u005c\1/g' > \ > >>>> | gzip -c > ${filename}.gz > >>>> rm -f ${filename}.bz2 > >>>> > >>>> as this is no longer necessary. > >>>> > >>>> > > >>>> > 2. I want to access some specific dbpedia properties such as > >>>> > dbpedia-owl:locationCity and others. These are already present in > the > >>>> > mappingbased_properties_en.nt > >>>> > file which is in the fetch_data_en_int.sh script but are not in the > >>>> > > >>>> > https://svn.apache.org/repos/asf/stanbol/trunk/entityhub/indexing/dbpedia/src/main/resources/indexing/config/mappings.txt > >>>> > file. > >>>> > Should I include them there and do a dbpedia index rebuild? > >>>> > >>>> Exactly. If the size of the created SolrIndex is an issue I recommend > >>>> also that you remove properties you do not need. > >>>> > >>>> > > >>>> > I've already described this in the "Named entity coref resolution > >>>> based on > >>>> > dbpedia" mail thread but I thought of creating a new mail for > >>>> visibility > >>>> > and for not clogging the other thread. > >>>> > >>>> The old thread is anyways already much to long. Please make sure that > >>>> important points and decisions of that thread are also reflected in > >>>> the description of STANBOL-1279 > >>>> > >>>> best > >>>> Rupert > >>>> > >>>> > > >>>> > Thanks, > >>>> > Cristian > >>>> > >>>> > >>>> > >>>> -- > >>>> | Rupert Westenthaler rupert.westentha...@gmail.com > >>>> | Bodenlehenstraße 11 ++43-699-11108907 > >>>> | A-5500 Bischofshofen > >>>> | > REDLINK.CO.......................................................................... > >>>> | http://redlink.co/ > >>>> > >>> > >>> > >> > > > > -- > | Rupert Westenthaler rupert.westentha...@gmail.com > | Bodenlehenstraße 11 ++43-699-11108907 > | A-5500 Bischofshofen > | > REDLINK.CO.......................................................................... > | http://redlink.co/ >
