I created some synthetic data that tickles the bug reliably on my machine with
a standard virtuoso.ini (just adding the directory for the files to the
allowed list).  I'm attaching the generator program for the files and a
loading script.


On 12/18/18 9:46 AM, Peter F. Patel-Schneider wrote:
> I did a bit of digging and it sure looks as if there is a race condition in
> rdf_rl_lang_id in ttlpv.sql.   This code appears to check to see if the
> language tag is already in DB.DBA.RDF_LANGUAGE and adds it if not.  But
> another thread could do the same insert between the check and the insert, as
> far as I can tell.
> It looks to me as if the right solution is to do a soft insert and a
> subsequent query instead of a hard insert.
> However, I don't understand how locking works in SQL so there may be something
> that prevents another thread from interfering.
> peter
> On 12/18/18 8:55 AM, Peter F. Patel-Schneider wrote:
>> I'm loading the Turtle Wikidata RDF complete dump, split into pieces and
>> loaded with 10 active readers.   About half the time the load fails with one
>> or more of these errors.  The errors are always near the beginning of the
>> load---in the first group of 10 files to be loaded and near the beginning of
>> the files (generally in the first couple of hundred lines in a file of size
>> well over 1 GB).  No errors occur for any files beyond the first ten.
>> I could provide the files, but they total to about 340GB.
>> It sure looks as if there is some sort of bug when loading RDF 
>> language-tagged
>> strings, where a race condition means that two threads are trying to load the
>> same language tag into DB.DBA.RDF_LANGUAGE.  This would explain why the
>> problem occurs only at the beginning of the load, when the language tags are
>> being added to DB.DBA.RDF_LANGUAGE, and not later.  It would also explain why
>> the errors are different between different runs.  (The only other explanation
>> would be hardware errors, but this doesn't seem to be viable.)
>> It seems to me that a quick patch for this problem would be to change the
>> insert into a soft insert, but I don't know where to make this change in the 
>> code.
>> peter
>> On 12/11/18 7:11 PM, Hugh Williams wrote:
>>> Hi Peter,
>>> The triple value do indeed appear to be valid, but the problem could be
>>> somewhere else in the dataset file and not necessarily on the reported line 
>>> or
>>> line before it.
>>> Is it a public dataset you are loading and if so can you provide a copy for
>>> local testing ?
>>> Best Regards
>>> Hugh Williams
>>> Professional Services
>>> OpenLink Software
>>> Home Page: http://www.openlinksw.com
>>> Community Support: https://community.openlinksw.com
>>> Weblogs (Blogs):
>>> Company Blog: https://medium.com/openlink-software-blog
>>> Virtuoso Blog: https://medium.com/virtuoso-blog
>>> Data Access Drivers
>>> Blog: https://medium.com/openlink-odbc-jdbc-ado-net-data-access-drivers
>>> LinkedIn -- http://www.linkedin.com/company/openlink-software/
>>> Twitter  -- http://twitter.com/OpenLink
>>> Google+  -- http://plus.google.com/100570109519069333827/
>>> Facebook -- http://www.facebook.com/OpenLinkSoftware
>>> Universal Data Access, Integration, and Management Technology Providers
>>>> On 11 Dec 2018, at 17:45, Peter F. Patel-Schneider <pfpschnei...@gmail.com
>>>> <mailto:pfpschnei...@gmail.com>> wrote:
>>>> I'm loading a bunch of Turtle files and I'm getting the error
>>>> 2300 TURTLE RDF loader, line 1012: SR197: Non unique primary key on
>>>> The line in question looks fine:
>>>>   "Wikimedia template"@ki,
>>>> The line before it may indicate the issue
>>>>    "Wikimedia template"@kg,
>>>> Nonetheless this should be valid RDF so there appears to be a bug in 
>>>> Virtuoso
>>>> here.
>>>> Is there any workaround?
>>>> This is in Virtuoso 07.20.3230.
>>>> peter
>>>> _______________________________________________
>>>> Virtuoso-users mailing list
>>>> Virtuoso-users@lists.sourceforge.net
>>>> <mailto:Virtuoso-users@lists.sourceforge.net>
>>>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users

for x in range (0,20) :
    file = open('test{:0>2d}.ttl'.format(x),'w')

    for k in range(0,10) :

        file.write('<http://www.wikidata.org/entity/Q{:0>2d}{:0>2d}> <http://www.w3.org/2000/01/rdf-schema#label>\n'.format(x,k))
        for y in range (ord('a'),ord('z')+1) :
            for z in range (ord('a'),ord('z')+1) :
		file.write('    "description {:0>2d}{:0>3d}{:0>3d}"@l{:s}{:s},\n'.format(x,y,z,chr(y),chr(z)))
        file.write('  "JUNK".\n')

Attachment: test.sh
Description: application/shellscript

Virtuoso-users mailing list

Reply via email to