And here I was, thinking this would be simple. :) I really enjoyed myself reading about all the little details. JC, please, don't give up! :)
> I don't know why this is a problem. For Greek we have many pages with > English names too > i.e. http://el.wikipedia.org/wiki/ASCII > http://el.wikipedia.org/wiki/World_Wide_Web > > I see the following options here > A) For URIs: > 1) leave title as we get it from the Wikipedia dumps (suggested option), > since we might get some links to the other script so we can create > sameAs links with a new extractor (easy) > 2) give the option to transliterate *all* URIs to a preferred script (we > might miss some semantics when Latin was intended and we choose a > non-latin > script) The first option definitely makes more sense. > > B) for literals: > Make an option to transliterate to a preferred transliteration as > discussed > in the beginning > We don't need to handle "preserve" in the parser since the only place we > might need it is the parser and this is already handled by the mw engine > > The general outcome so far (if I understood correctly) would be to > create a general class i.e. TriplePolicy that would handle policy parsing > UriPolicy will extend TriplePolicy and > create a LiteralPolicy class that will handle literal values > > and maybe create a TransliterateSameAs extractor > > @Uros, you are the language expert here ;) can you suggest anything > different? Finally, I get to feel like an expert on something. :) I think you summed it up nicely. The suggested solution sounds reasonable, although I'm a little scared now and not sure I'd be of much help. Please do let me know if there's anything I can do, though. Best, Uros > > Cheers, > Dimitris > > > > > On Tue, Dec 3, 2013 at 11:01 PM, Jona Christopher Sahnwaldt > <[email protected] >> wrote: > >> On 3 December 2013 21:34, Jona Christopher Sahnwaldt <[email protected]> >> wrote: >> > On 3 December 2013 20:49, Andrea Di Menna <[email protected]> wrote: >> >> 2013/12/3 Jona Christopher Sahnwaldt <[email protected]> >> >>> >> >>> On 3 December 2013 18:19, Andrea Di Menna <[email protected]> wrote: >> >>> > 2013/12/3 Jona Christopher Sahnwaldt <[email protected]> >> >>> >> >> >>> >> On 3 December 2013 16:54, Andrea Di Menna <[email protected]> >> wrote: >> >>> >> > Hi, >> >>> >> > >> >>> >> > I agree with JC that probably UriPolicy is not the best place. >> >>> >> >> >>> >> I guess extending UriPolicy looks attractive because modifying >> >>> >> literals has some common needs with modifying URIs. But we should >> >>> >> rather introduce a new class StringLiteralPolicy or so and move >> some >> >>> >> code from UriPolicy to a common base class. Maybe we can share >> the >> >>> >> policy parsing code etc. But literals and URIs are quite >> different >> and >> >>> >> should probably be handled by different classes. >> >>> >> >> >>> >> Maybe we need a new Destination subclass too (or instead). >> Actually, >> >>> >> if we follow YAGNI and KISS principles we should simply use a >> >>> >> SerbianTransliterationDestination... >> >>> >> >> >>> >> > As per Uros use case I understand that what he would like to >> obtain >> >>> >> > is a >> >>> >> > duplication of quads. >> >>> >> > Probably this should be done in the Formatters or maybe as a >> >>> >> > post-processing >> >>> >> > operation? >> >>> >> > >> >>> >> > The problem is the following: >> >>> >> > - some languages are officially digraphic, that is they can use >> two >> >>> >> > different scripts (e.g. latin and cyrillic scripts) >> >>> >> > - Serbian (sr) is a digraphic language (latin and cyrillic) >> >>> >> > - Serbian wikipedia allows users to see articles in latin and >> >>> >> > cyrillic, >> >>> >> > e.g. >> >>> >> > cyrillic: >> >>> >> > >> >>> >> > >> >>> >> > >> https://sr.wikipedia.org/sr-ec/%D0%93%D0%BE%D1%81%D0%BD%D0%B5%D0%BB_(%D0%90%D1%80%D0%BA%D0%B0%D0%BD%D0%B7%D0%B0%D1%81) >> >>> >> > latin: >> >>> >> > >> >>> >> > >> >>> >> > >> https://sr.wikipedia.org/sr-el/%D0%93%D0%BE%D1%81%D0%BD%D0%B5%D0%BB_(%D0%90%D1%80%D0%BA%D0%B0%D0%BD%D0%B7%D0%B0%D1%81) >> >>> >> > - wikipedia dumps do not contain both versions but only >> cyrillic >> in >> >>> >> > 99% >> >>> >> > of >> >>> >> > the cases >> >>> >> > - if you were to extract string objects from the sr dump you >> would >> >>> >> > get >> >>> >> > cyrillic almost everywhere, for labels or for template property >> >>> >> > values >> >>> >> >> >>> >> I just looked at a few pages in the Serbian Wikipedia. >> >>> >> >> >>> >> There is a piece of MediaWiki syntax that I hadn't seen before: >> >>> >> wrapping text in -{...}- keeps it from being transliterated. In >> an >> >>> >> ideal world, we would extend the DBpedia parser to handle this... >> >>> >> >> >>> >> There are actually three ways a Serbian Wikipedia page can be >> >>> >> displayed: unchanged, transliterated to Cyrillic, transliterated >> to >> >>> >> Latin. For example, I put this wiki text on my Serbian Wikipedia >> user >> >>> >> page: >> >>> >> >> >>> >> Unprotected: Test >> >>> >> Protected: -{Test}- >> >>> >> Unprotected: ÐаÑÑÐµÑ >> >>> >> Protected: -{ÐаÑÑеÑ}- >> >>> >> >> >>> >> Depending on the URL, it is displayed in in different ways: >> >>> >> >> >>> >> http://sr.wikipedia.org/wiki/ÐоÑиÑник:Chrisahn or >> >>> >> http://sr.wikipedia.org/sr/ÐоÑиÑник:Chrisahn - unmodified >> >>> >> >> >>> >> Unprotected: Test >> >>> >> Protected: Test >> >>> >> Unprotected: ÐаÑÑÐµÑ >> >>> >> Protected: ÐаÑÑÐµÑ >> >>> >> >> >>> >> http://sr.wikipedia.org/sr-ec/ÐоÑиÑник:Chrisahn - >> transliterated to >> >>> >> Cyrillic unless protected >> >>> >> >> >>> >> УнпÑоÑеÑÑед: ТеÑÑ >> >>> >> ÐÑоÑеÑÑед: Test >> >>> >> УнпÑоÑеÑÑед: ÐаÑÑÐµÑ >> >>> >> ÐÑоÑеÑÑед: ÐаÑÑÐµÑ >> >>> >> >> >>> >> http://sr.wikipedia.org/sr-el/ÐоÑиÑник:Chrisahn - >> transliterated to >> >>> >> Latin unless protected >> >>> >> >> >>> >> Unprotected: Test >> >>> >> Protected: Test >> >>> >> Unprotected: Parser >> >>> >> Protected: ÐаÑÑÐµÑ >> >>> >> >> >>> > >> >>> > But still the content in the dumps will be the same, i.e. the >> wikitext >> >>> > you >> >>> > have saved in your page. >> >>> > No matter how you render it on the Mediawiki instance which hosts >> it. >> >>> > Correct? >> >>> >> >>> Correct. >> >>> >> >>> > >> >>> >> >> >>> >> >> >>> >> > >> >>> >> > Uros is wondering what would happen if a serbian user searches >> using >> >>> >> > for >> >>> >> > example the latin transliterated version of a cyrillic label >> (e.g. >> >>> >> > using >> >>> >> > SPARQL on Virtuoso for example). >> >>> >> > Their search would probably fail (unless Virtuoso implements >> >>> >> > transliteration >> >>> >> > on-the-fly). >> >>> >> > >> >>> >> > Romanization or Cyrillization are transliteration methods which >> are >> >>> >> > also >> >>> >> > available through ICU4J >> >>> >> > >> >>> >> > >> >>> >> > [ >> http://icu-project.org/apiref/icu4j/com/ibm/icu/text/Transliterator.html] >> >>> >> >> >>> >> Looks good, but is there an implementation for Serbian? If there >> >>> >> isn't, this probably won't help us much. Not enough to justify >> adding >> >>> >> ICU4J as a new dependency, I think. >> >>> >> >> >>> > >> >>> > Yes there is a Transliterator with ID "Serbian-Latin/BGN" (a list >> here >> >>> > >> >>> > >> http://www.avajava.com/tutorials/lessons/what-are-the-system-transliterators-available-with-icu4j.html >> , >> >>> > don't know if this is still valid) >> >>> > I have made some quick tests and it seems to work OK. >> >>> >> >>> Cool! >> >>> >> >>> > >> >>> >> >> >>> >> > >> >>> >> > I think it does not make sense to transliterate URIs but only >> string >> >>> >> > typed >> >>> >> > values. >> >>> >> >> >>> >> I don't know. Wikipedia seems to have some elaborate rules in >> place >> as >> >>> >> far as Latin/Cyrillic URLs are concerned. Maybe we should follow >> these >> >>> >> rules too? >> >>> >> >> >>> > >> >>> > Are the "preserve" rules also applied to wikilinks? If they are >> not >> then >> >>> > I >> >>> > think we should not apply transliteration to URIs. >> >>> >> >>> According to a few tests on my user page, the text (title) displayed >> >>> for a Wiki link is transliterated unless it's "protected" by >> -{...}-. >> >>> The actual link target is *always* the Cyrillic version, even if the >> >>> wiki text contains the Latin article name. Example: [[Johan Volfgang >> >>> Gete]] always results in a link to >> >>> http://sr.wikipedia.org/wiki/ÐÐ¾Ñ Ð°Ð½_ÐолÑганг_ÐеÑе . >> >> >> >> >> >> You're right (as usual ;)) >> >> I suppose the mediawiki instance transliterates the text in the >> wikilink and >> >> connects to the >> >> cyrillic page on-the-fly, if it exists. >> >> I think maybe Uros can help us understand what happens when you >> create a >> >> page, whether >> >> you have to use a cyrillic title or you can also insert a latin >> title. >> >> Also, would be interesting to understand if the mediawiki instance >> >> transliterates latin titles >> >> on page creation. >> > >> > That's controlled by the __NOTITLECONVERT__ magic word. See >> > https://www.mediawiki.org/wiki/Help:Magic_words . The Serbian variants >> > of the magic word are __ÐÐÐÐÐ__ and __BEZKN__ . See >> > >> https://git.wikimedia.org/blob/mediawiki%2Fcore.git/master/languages%2Fmessages%2FMessagesSr_ec.php >> > >> > Example: http://sr.wikipedia.org/wiki/ASCII isn't transliterated to >> > http://sr.wikipedia.org/wiki/ÐСЦÐÐ . On the contrary: >> [[ÐСЦÐÐ]] is >> > rendered as a link to http://sr.wikipedia.org/wiki/ASCII . >> > >> > As usual with MediaWiki, the devil is very much in the details. >> >> ...and the deeper you dig, the more evil you find... There are pages >> who *don't* contain __NOTITLECONVERT__ or its synonyms, and whose >> titles still aren't transliterated, e.g. >> http://sr.wikipedia.org/wiki/Little_endian or >> http://sr.wikipedia.org/wiki/Acetil ... I'm giving up. >> >> >> > >> >> One approach could be to create owl:sameAs triples linking cyrillic >> >> resources to latin resources, >> >> and then ignoring transliteration for URIs... >> >> >> >>> >> >>> If we want DBpedia to use the same policy, then we *should* >> >>> transliterate URIs. Currently, we always use the link target as it's >> >>> in the wiki source text. Example: for [[Johan Volfgang Gete]], we >> >>> generate a link to >> http://sr.dbpedia.org/resource/Johan_Volfgang_Gete >> >>> . To be consistent with Wikipedia, the link should point to >> >>> http://sr.dbpedia.org/resource/ÐÐ¾Ñ Ð°Ð½_ÐолÑганг_ÐеÑе >> instead. >> >>> >> >> >> >> See above. >> >> >> >>> >> >>> The main problem I see with transliterating URIs is configuration. >> >>> That's one of the main problems of DBpedia anyway. We're putting too >> >>> much effort into parsing configuration files. To allow >> transliteration >> >>> of URIs, we have to extend the UriPolicy syntax and parser, which is >> >>> already pretty convoluted anyway. If we used something like Spring >> >>> instead of self-made configuration stuff, we would simply add a >> class >> >>> and reference the class in the configuration. Additionally, we >> should >> >>> use different configuration objects for each language. That doesn't >> >>> have to mean that we need a separate configuration file for each >> >>> language, just that we have to initialize the extraction framework >> >>> differently for each language. This would also make UriPolicy >> >>> configuration easier. >> >>> >> >>> JC >> >> >> >> >> >> I am with you :) >> >> What about Typesafe Config? [1] >> >> >> >> [1] https://github.com/typesafehub/config >> >> >> >> Andrea >> >> >> >>> >> >>> >> >>> > >> >>> > Cheers! >> >>> > Andrea >> >>> > >> >>> >> >> >>> >> Cheers, >> >>> >> JC >> >>> >> >> >>> >> > >> >>> >> > Cheers >> >>> >> > Andrea >> >>> >> > >> >>> >> > >> >>> >> > 2013/12/3 Jona Christopher Sahnwaldt <[email protected]> >> >>> >> >> >> >>> >> >> Hi all, >> >>> >> >> >> >>> >> >> I don't think UriPolicy is a good place to do this... >> >>> >> >> >> >>> >> >> But anyway, I don't understand the problem yet. :-) >> >>> >> >> >> >>> >> >> Uros, you wrote about ISO 8859-2 and ISO 15924. >> >>> >> >> >> >>> >> >> ISO 8859-2 is a character encoding, but I'm pretty sure that >> >>> >> >> Wikipedia >> >>> >> >> is not using it, and I know that DBpedia is not using it. I >> think >> >>> >> >> Wikipedia uses UTF-8 all over the place. I know that the >> Wikipedia >> >>> >> >> XML >> >>> >> >> dumps are UTF-8 encoded, and so are the DBpedia dumps. >> >>> >> >> >> >>> >> >> ISO 15924 is not a character encoding, but a way to specify >> the >> >>> >> >> names >> >>> >> >> of scripts. See https://en.wikipedia.org/wiki/ISO_15924 >> >>> >> >> >> >>> >> >> What would romanization or cyrillization do exactly? Is there >> a >> >>> >> >> one-to-one mapping between letters? Or letter sequences? >> >>> >> >> >> >>> >> >> Cheers, >> >>> >> >> JC >> >>> >> >> >> >>> >> >> On 3 December 2013 16:02, Dimitris Kontokostas < >> [email protected]> >> >>> >> >> wrote: >> >>> >> >> > Hi Uros, >> >>> >> >> > >> >>> >> >> > Don't worry, as we said we are here to help if you get >> stuck;) >> we >> >>> >> >> > all >> >>> >> >> > started like this. >> >>> >> >> > >> >>> >> >> > If you look at the formatters package you will understand >> what's >> >>> >> >> > going >> >>> >> >> > on. >> >>> >> >> > We have formatters that write a triple based on some >> policies >> we >> >>> >> >> > define. >> >>> >> >> > We parse the policies at runtime, create formatters based on >> these >> >>> >> >> > policies >> >>> >> >> > and feed them to destinations. >> >>> >> >> > >> >>> >> >> > I think we should generalize URIPolicy to TriplePolicy and >> create >> >>> >> >> > a >> >>> >> >> > "transliterate" action. >> >>> >> >> > I made a change in the URIPolicy code to make it more >> descriptive >> >>> >> >> > [1] >> >>> >> >> > Right now we have support only for URIs but if you change >> the >> >>> >> >> > following >> >>> >> >> > it >> >>> >> >> > should be a good start to make your changes >> >>> >> >> > >> >>> >> >> > //String: Uri or Literal, Boolean: is URI or not, String: >> output >> >>> >> >> > (new >> >>> >> >> > URI >> >>> >> >> > or transliterated string) >> >>> >> >> > type Policy = (String, Boolean) => String >> >>> >> >> > >> >>> >> >> > type PolicyApplicable = (String, Boolean) => Boolean >> >>> >> >> > >> >>> >> >> > I also submitted a feature request [2], you can make a >> proper >> >>> >> >> > description >> >>> >> >> > and continue the discussion there >> >>> >> >> > >> >>> >> >> > Cheers, >> >>> >> >> > Dimitris >> >>> >> >> > >> >>> >> >> > >> >>> >> >> > [1] https://github.com/dbpedia/extraction-framework/pull/131 >> >>> >> >> > [2] >> https://github.com/dbpedia/extraction-framework/issues/130 >> >>> >> >> > >> >>> >> >> > >> >>> >> >> > On Mon, Dec 2, 2013 at 5:50 PM, Uros Milosevic >> >>> >> >> > <[email protected]> >> >>> >> >> > wrote: >> >>> >> >> >> >> >>> >> >> >> Hi Andrea/Dimitris, >> >>> >> >> >> >> >>> >> >> >> Thanks for the tips. Actually, when I said I was no core >> expert, >> >>> >> >> >> I >> >>> >> >> >> meant I >> >>> >> >> >> was an absolute beginner. :) I wanted to go with an >> extractor >> >>> >> >> >> because >> >>> >> >> >> that >> >>> >> >> >> seemed simpler (and safer) than meddling with the core. >> Most >> of >> >>> >> >> >> the >> >>> >> >> >> stuff >> >>> >> >> >> in there still seems rather confusing, but I'll look into >> it. >> >>> >> >> >> >> >>> >> >> >> So, the UriPolicy code is where the triples get written >> (pointer >> >>> >> >> >> to >> >>> >> >> >> the >> >>> >> >> >> exact line, anyone?), or is this simply where you'd like to >> place >> >>> >> >> >> the >> >>> >> >> >> new >> >>> >> >> >> code? Also, would "UriPolicy" remain an adequate name for >> the >> >>> >> >> >> class, >> >>> >> >> >> then? >> >>> >> >> >> >> >>> >> >> >> Best, >> >>> >> >> >> Uros >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> > Maybe something like: >> >>> >> >> >> > >> >>> >> >> >> > script.sr=sr-Cyrl:identity;sr-Latn:Serbian-Latin/BGN >> >>> >> >> >> > >> >>> >> >> >> > where you specify a list of (languageTag:transliterator) >> >>> >> >> >> > separated >> >>> >> >> >> > by >> >>> >> >> >> > ';' >> >>> >> >> >> > for one language? >> >>> >> >> >> > The transliterator could be either "identity" (no >> >>> >> >> >> > transformation) >> >>> >> >> >> > or >> >>> >> >> >> > a >> >>> >> >> >> > icu4j transliterator-ID. >> >>> >> >> >> > >> >>> >> >> >> > As Dimitris said, Uros please feel free to ask if you >> need >> >>> >> >> >> > help! >> >>> >> >> >> > >> >>> >> >> >> > Cheers >> >>> >> >> >> > Andrea >> >>> >> >> >> > >> >>> >> >> >> > >> >>> >> >> >> > 2013/11/30 Dimitris Kontokostas <[email protected]> >> >>> >> >> >> > >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> On Fri, Nov 29, 2013 at 5:02 PM, Andrea Di Menna >> >>> >> >> >> >> <[email protected]>wrote: >> >>> >> >> >> >> >> >>> >> >> >> >>> Hello Uros, >> >>> >> >> >> >>> >> >>> >> >> >> >>> that's a really interesting problem :) >> >>> >> >> >> >>> I am no expert either but probably the best approach >> would be >> >>> >> >> >> >>> to >> >>> >> >> >> >>> "duplicate" triples when they are going to be written >> (e.g. >> >>> >> >> >> >>> in >> >>> >> >> >> >>> the >> >>> >> >> >> >>> destinations package), instead of modifying the >> extractors. >> >>> >> >> >> >>> >> >>> >> >> >> >> >> >>> >> >> >> >> I agree, I'd suggest we extend the UriPolicy [1] >> functionality >> >>> >> >> >> >> to >> >>> >> >> >> >> do >> >>> >> >> >> >> string object transformations (now it only applies to >> URIs >> / >> >>> >> >> >> >> IRIs) >> >>> >> >> >> >> and use the configuration files to select the desired >> output >> >>> >> >> >> >> [2]. >> >>> >> >> >> >> Uros, do you want to give it a shot? You can always ask >> for >> >>> >> >> >> >> help >> >>> >> >> >> >> here >> >>> >> >> >> >> ;) >> >>> >> >> >> >> >> >>> >> >> >> >> [1] >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> https://github.com/dbpedia/extraction-framework/blob/master/core/src/main/scala/org/dbpedia/extraction/destinations/formatters/UriPolicy.scala >> >>> >> >> >> >> [2] >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> https://github.com/dbpedia/extraction-framework/blob/master/dump/extraction.default.properties#L130 >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >>> For what regards which tools to use, it looks like >> icu4j >> >>> >> >> >> >>> Translitterator >> >>> >> >> >> >>> suits your needs, e.g. >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> Transliterator.getInstance("Serbian-Latin/BGN").transliterate("ÐÐ°Ð»Ð°ÐºÐ¾Ñ >> >>> >> >> >> >>> 5 >> >>> >> >> >> >>> (енгл. Malachor V) Ñе измиÑÑена >> планеÑа Ñ >> >>> >> >> >> >>> ÑнивеÑзÑÐ¼Ñ Ð Ð°Ñова звезда.") >> >>> >> >> >> >>> >> >>> >> >> >> >>> results in >> >>> >> >> >> >>> >> >>> >> >> >> >>> Malakor 5 (engl. Malachor V) je izmiÅ¡ljena planeta u >> >>> >> >> >> >>> univerzumu >> >>> >> >> >> >>> Ratova >> >>> >> >> >> >> >>> >> >> >> >>> zvezda. >> >>> >> >> >> >>> >> >>> >> >> >> >>> What do you think? >> >>> >> >> >> >>> >> >>> >> >> >> >>> Cheers >> >>> >> >> >> >>> Andrea >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> 2013/11/29 Uros Milosevic <[email protected]> >> >>> >> >> >> >>> >> >>> >> >> >> >>>> Hi all, >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> As some of you may know, a Serbian version of DBpedia >> is >> >>> >> >> >> >>>> currently >> >>> >> >> >> >>>> in >> >>> >> >> >> >>>> the >> >>> >> >> >> >>>> works. Now, Serbian, unlike any other language in >> Europe, is >> >>> >> >> >> >>>> digraphic >> >>> >> >> >> >>>> in >> >>> >> >> >> >>>> nature, officially supporting both (Serbian) Cyrillic >> and >> >>> >> >> >> >>>> (Gaj's) >> >>> >> >> >> >>>> Latin >> >>> >> >> >> >>>> alphabet. This is absolutely fine for storing >> information in >> >>> >> >> >> >>>> any >> >>> >> >> >> >>>> modern >> >>> >> >> >> >>>> knowledge base, but can often be a major obstacle for >> >>> >> >> >> >>>> information >> >>> >> >> >> >>>> retrieval. >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> For instance, most Serbs rely on the Latin alphabet >> for >> >>> >> >> >> >>>> communication/interaction on the Web. That means a >> large >> >>> >> >> >> >>>> portion >> >>> >> >> >> >>>> of >> >>> >> >> >> >>>> the >> >>> >> >> >> >>>> information is (and, often, expected to be) encoded in >> ISO >> >>> >> >> >> >>>> 8859-2 >> >>> >> >> >> >>>> (i.e. >> >>> >> >> >> >>>> Latin-2). And, yet, 99% of the information in the >> Serbian >> >>> >> >> >> >>>> Wikipedia >> >>> >> >> >> >>>> dumps >> >>> >> >> >> >>>> is encoded in ISO 15924 (i.e. Cyrillic). So, unless >> your >> >>> >> >> >> >>>> software >> >>> >> >> >> >>>> performs >> >>> >> >> >> >>>> romanization (i.e. converts Cyrillic to Latin) or >> >>> >> >> >> >>>> cyrillization >> >>> >> >> >> >>>> (i.e. >> >>> >> >> >> >>>> vice >> >>> >> >> >> >>>> versa) on-the-fly, at retrieval time (Wikipedia >> appears >> to >> >>> >> >> >> >>>> be >> >>> >> >> >> >>>> doing >> >>> >> >> >> >>>> this), >> >>> >> >> >> >>>> many attempts at information extraction will be doomed >> to >> >>> >> >> >> >>>> fail. >> >>> >> >> >> >>>> This >> >>> >> >> >> >>>> directly affects common tasks such as keyword search, >> >>> >> >> >> >>>> label-based >> >>> >> >> >> >>>> SPARQL >> >>> >> >> >> >>>> querying, named entity recognition, etc. >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> What I would like to do is improve some of the >> existing >> >>> >> >> >> >>>> DBpedia >> >>> >> >> >> >>>> extractors, or develop new ones, that would take this >> >>> >> >> >> >>>> problem >> >>> >> >> >> >>>> into >> >>> >> >> >> >>>> consideration and perform romanization of Wikipedia >> dumps so >> >>> >> >> >> >>>> as >> >>> >> >> >> >>>> to >> >>> >> >> >> >>>> output >> >>> >> >> >> >>>> information encoded in *both* scripts. Now, I know >> storing >> >>> >> >> >> >>>> the >> >>> >> >> >> >>>> same >> >>> >> >> >> >>>> information twice might not be the most elegant >> solution, >> >>> >> >> >> >>>> but >> >>> >> >> >> >>>> unless >> >>> >> >> >> >>>> someone is to include romanization/cyrillization >> features in >> >>> >> >> >> >>>> the >> >>> >> >> >> >>>> next >> >>> >> >> >> >>>> version of SPARQL, I don't see a better solution at >> the >> >>> >> >> >> >>>> moment. >> >>> >> >> >> >>>> Of >> >>> >> >> >> >>>> course, >> >>> >> >> >> >>>> there is also the matter of perspective - one could >> argue >> >>> >> >> >> >>>> that >> >>> >> >> >> >>>> although >> >>> >> >> >> >>>> the information is the same, the very fact that >> different >> >>> >> >> >> >>>> character >> >>> >> >> >> >>>> sequences are needed to describe the same piece of >> knowledge >> >>> >> >> >> >>>> makes >> >>> >> >> >> >>>> this >> >>> >> >> >> >>>> problem fall into the domain of multilinguality. >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> So, the general idea is to use a single IRI per >> resource, >> >>> >> >> >> >>>> but >> >>> >> >> >> >>>> have >> >>> >> >> >> >>>> two >> >>> >> >> >> >>>> separate triples for any literal originally encoded in >> >>> >> >> >> >>>> cyrillic. >> >>> >> >> >> >>>> For >> >>> >> >> >> >>>> example: >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> < >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> http://sr.dbpedia.org/resource/Парсер< >> http://sr.dbpedia.org/resource/Ð&%231072;&%231088;&%231089;&%231077;&%231088<http://sr.dbpedia.org/resource/%D0%9F&%231072;&%231088;&%231089;&%231077;&%231088> >> > >> >>> >> >> >> >> >>> >> >> >> >>>> ;> >> >>> >> >> >> >>>> <http://www.w3.org/2000/01/rdf-schema#label> >> >>> >> >> >> >>>> "Парсер"@sr-Cyrl . >> >>> >> >> >> >>>> < >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> http://sr.dbpedia.org/resource/Парсер< >> http://sr.dbpedia.org/resource/Ð&%231072;&%231088;&%231089;&%231077;&%231088<http://sr.dbpedia.org/resource/%D0%9F&%231072;&%231088;&%231089;&%231077;&%231088> >> > >> >>> >> >> >> >> >>> >> >> >> >>>> ;> >> >>> >> >> >> >>>> <http://www.w3.org/2000/01/rdf-schema#label> >> >>> >> >> >> >>>> "Parser"@sr-Latn . >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> The above language tags are as per IANA Language >> Subtag >> >>> >> >> >> >>>> Registry >> >>> >> >> >> >>>> [1], >> >>> >> >> >> >>>> which lists them as redundant, though, so a "sr" tag, >> >>> >> >> >> >>>> instead, >> >>> >> >> >> >>>> could >> >>> >> >> >> >>>> be >> >>> >> >> >> >>>> enough for both. >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> I'm no DBpedia core expert, so some tips, ideas, >> directions >> >>> >> >> >> >>>> or >> >>> >> >> >> >>>> any >> >>> >> >> >> >>>> other >> >>> >> >> >> >>>> information that would help me get started would be >> much >> >>> >> >> >> >>>> appreciated! >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> Best, >> >>> >> >> >> >>>> Uros >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> [1] >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> http://www.iana.org/assignments/language-subtag-registry/language-subtag-registry >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> ------------------------------------------------------------------------------ >> >>> >> >> >> >>>> Rapidly troubleshoot problems before they affect your >> >>> >> >> >> >>>> business. >> >>> >> >> >> >>>> Most >> >>> >> >> >> >>>> IT >> >>> >> >> >> >>>> organizations don't have a clear picture of how >> application >> >>> >> >> >> >>>> performance >> >>> >> >> >> >>>> affects their revenue. With AppDynamics, you get 100% >> >>> >> >> >> >>>> visibility >> >>> >> >> >> >>>> into >> >>> >> >> >> >>>> your >> >>> >> >> >> >>>> Java,.NET, & PHP application. Start your 15-day FREE >> TRIAL >> >>> >> >> >> >>>> of >> >>> >> >> >> >>>> AppDynamics Pro! >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk >> >>> >> >> >> >>>> _______________________________________________ >> >>> >> >> >> >>>> Dbpedia-developers mailing list >> >>> >> >> >> >>>> [email protected] >> >>> >> >> >> >>>> >> >>> >> >> >> >>>> >> https://lists.sourceforge.net/lists/listinfo/dbpedia-developers >> >>> >> >> >> >>>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> ------------------------------------------------------------------------------ >> >>> >> >> >> >>> Rapidly troubleshoot problems before they affect your >> >>> >> >> >> >>> business. >> >>> >> >> >> >>> Most >> >>> >> >> >> >>> IT >> >>> >> >> >> >>> organizations don't have a clear picture of how >> application >> >>> >> >> >> >>> performance >> >>> >> >> >> >>> affects their revenue. With AppDynamics, you get 100% >> >>> >> >> >> >>> visibility >> >>> >> >> >> >>> into >> >>> >> >> >> >>> your >> >>> >> >> >> >>> Java,.NET, & PHP application. Start your 15-day FREE >> TRIAL of >> >>> >> >> >> >>> AppDynamics >> >>> >> >> >> >>> Pro! >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk >> >>> >> >> >> >>> _______________________________________________ >> >>> >> >> >> >>> Dbpedia-developers mailing list >> >>> >> >> >> >>> [email protected] >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> https://lists.sourceforge.net/lists/listinfo/dbpedia-developers >> >>> >> >> >> >>> >> >>> >> >> >> >>> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> -- >> >>> >> >> >> >> Kontokostas Dimitris >> >>> >> >> >> >> >> >>> >> >> >> > >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> > >> >>> >> >> > >> >>> >> >> > >> >>> >> >> > -- >> >>> >> >> > Kontokostas Dimitris >> >>> >> >> > >> >>> >> >> > >> >>> >> >> > >> >>> >> >> > >> >>> >> >> > >> ------------------------------------------------------------------------------ >> >>> >> >> > Rapidly troubleshoot problems before they affect your >> business. >> >>> >> >> > Most >> >>> >> >> > IT >> >>> >> >> > organizations don't have a clear picture of how application >> >>> >> >> > performance >> >>> >> >> > affects their revenue. With AppDynamics, you get 100% >> visibility >> >>> >> >> > into >> >>> >> >> > your >> >>> >> >> > Java,.NET, & PHP application. Start your 15-day FREE TRIAL >> of >> >>> >> >> > AppDynamics >> >>> >> >> > Pro! >> >>> >> >> > >> >>> >> >> > >> >>> >> >> > >> >>> >> >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk >> >>> >> >> > _______________________________________________ >> >>> >> >> > Dbpedia-developers mailing list >> >>> >> >> > [email protected] >> >>> >> >> > >> https://lists.sourceforge.net/lists/listinfo/dbpedia-developers >> >>> >> >> > >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> ------------------------------------------------------------------------------ >> >>> >> >> Rapidly troubleshoot problems before they affect your >> business. >> Most >> >>> >> >> IT >> >>> >> >> organizations don't have a clear picture of how application >> >>> >> >> performance >> >>> >> >> affects their revenue. With AppDynamics, you get 100% >> visibility >> >>> >> >> into >> >>> >> >> your >> >>> >> >> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of >> >>> >> >> AppDynamics >> >>> >> >> Pro! >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk >> >>> >> >> _______________________________________________ >> >>> >> >> Dbpedia-developers mailing list >> >>> >> >> [email protected] >> >>> >> >> https://lists.sourceforge.net/lists/listinfo/dbpedia-developers >> >>> >> > >> >>> >> > >> >>> > >> >>> > >> >> >> >> >> > > > > -- > Kontokostas Dimitris > ------------------------------------------------------------------------------ > Sponsored by Intel(R) XDK > Develop, test and display web and hybrid apps with a single code base. > Download it for free now! > http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk_______________________________________________ > Dbpedia-developers mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/dbpedia-developers > ------------------------------------------------------------------------------ Sponsored by Intel(R) XDK Develop, test and display web and hybrid apps with a single code base. Download it for free now! http://pubads.g.doubleclick.net/gampad/clk?id=111408631&iu=/4140/ostg.clktrk _______________________________________________ Dbpedia-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dbpedia-developers
