Hi Andy, Sorry for the late answer, but I was quite busy.
The database was as far as I can tell generated in version 4.7.0 and then upgrades to 4.8.0 and 4.9.0 were done. Datasets were created (and some deleted and created again) in all these versions. The scenario that my colleague had currently isn't reproducible after he deleted and created his dataset again. I'd have to retry the data loads for my load test scenario and see if that still triggers the issue (during the load tests many months ago that was a pretty simple scenario that always ended in the error - but that definitely was done on version 4.7.0). I'll try to execute that loading code again and see what happens and open a Github issue if it is able to reliably produce the issue in 4.9.0. We are running Jena in a k8s cluster on AWS and it uses EFS as a file store. As far as I know we don't have anything configured ourselves that would cause concurrent access, but I'll check with our OPS people to see if they can identify something on the OS level that might access the files or if they have setup a backup process. Currently we're only running 1 Jena instance per environment. regards, Jan On Wed, 30 Aug 2023 at 23:08, Andy Seaborne <a...@apache.org> wrote: > Hi Jan, > > On 30/08/2023 14:58, Jan Eerdekens wrote: > > Hi, > > > > We've been evaluating an using Jena for about 1,5 years now, but are > > recently running into a perplexing issue. In a lot of different > scenarios, > > ways of using Jena, we are getting the exceptions like the one below: > > > > > Caused by: org.apache.thrift.protocol.TProtocolException: Unrecognized > type > > 0 > > at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:140) > > ~[fuseki-server.jar:4.8.0] > > > The different scenarios where it has happened are: > > > > - LOADing data into a dataset > > - compacting a dataset > > - querying a dataset > > > > In all those case we've run into trouble and get an exception that > > mentions *org.apache.jena.tdb2.TDBException: > > NodeTableTRDF/Read* and *org.apache.thrift.protocol.TProtocolException: > > Unrecognized type 0*. > > > > What can cause this? This looks kinda similar to this mailing list > > question, > https://www.mail-archive.com/users@jena.apache.org/msg20409.html, > > where it seems data corruption is mentioned that potentially isn't > > recoverable? > > > > The first time I encountered this issue was while doing a bunch of > > sequential LOAD commands to prepare a large dataset for load testing. I > > used files of around 50mb (started off with bigger ones) and after about > 20 > > to 25 LOADs it would get this error (also the completion time of a LOAD > > would go up and up). So for this scenario I was running locally (Jena > > Fuseki running in docker/Rancher) and only running the LOADs and not much > > else except for a SELECT here and there (via the Fuseki UI) to check that > > performance while LOADing. Is there a way that that could cause data > > corruption and the exception we're seeing? > > "Unrecognized type 0" has come up in a couple of cases. > > It means the node table is corrupt but the problem was caused silently > at some point in the past. The "Unrecognized type 0" exception happens > some time later (not a few seconds - either after a restart or a long > time of usage that has churned the node cache - possibly many months). > > There have been some fixes around compaction that addressed bugs in this > area. This has been the most common problem. > > Was this database originally create before 4.8.0? > > If not, do you have a fixed scenario so that the situation can be > recreated for 4.9.0? Please raise a github issue for it. > > Another situation is if another OS process interferes with the files > (container OS or host OS). What operating system is the host machine? > > While TDB2 endeavours to protect against multiple copies of TDB running > the same files, that is imperfect if it is two containers and the > database is on a mounted docker volume used by two containers. > > One other report seemed to be a backup process was running over the > files. We didn't get to the root cause of that one. > > Andy > > > > > regards, > > > > Jan Eerdekens > > >
