Re: Fuseki https certificate problems
Hi Nikolaos, Thanks for the information. And I've put in a PR to update the Fuseki Jetty HTTPS example using the the one you tested. Andy On 07/07/2022 16:38, Nikolaos Beredimas wrote: Hi Andy, TL;DR: Password-less PKCS12 passwords just don't work. After more testing, I couldn't get a password-less PKCS12 certificate to work, no matter what I tried. And after reading around I suspect it's not just Jetty that suffers from this, so there is nothing to be done. As for the other issue I had with a specific OpenSSL version, it turns out it's a non-issue. The culprit was an unrelated certificate generation script that omitted the provided password when calling openssl. In any case the xml provided back in February is good. NB On Thu, Jul 7, 2022 at 12:42 PM Andy Seaborne wrote: Hi Nikolaos, On 06/07/2022 11:04, Nikolaos Beredimas wrote: While trying to get Fuseki running over https I found this thread from February https://jena.markmail.org/message/2kqpd2tlinpdzpna?q=ssl+order:date-backward=1 1. I can confirm the provided xml works (tested on Fuseki 4.5.0) Thanks for confirming that. 2. I am having some issues generating the needed pkcs12 certificate file. a. When trying to generate a password-less pkcs12 file (openssl ... -passout pass:) Fuseki doesn't complain when loading it, but I always get SSL handshake errors and it doesn't work. It is Jetty that is handling the certificate via the JDK. Mentions like https://stackoverflow.com/questions/58345405/how-to-use-non-password-protected-p12-ssl-certificate-in-spring-boot (which is nearly 3 years old) suggest a password was needed at some time in the past. Current jetty documentation does not mention it one way of the other. b. When trying to generate with a password I get mixed results: OpenSSL 1.1.1f 31 Mar 2020 running on WSL2 Ubuntu 20.04 works fine. Fuseki loads the certificate and works like a charm. However, if I use OpenSSL 1.1.1o 3 May 2022 (running on docker-linuxserver/docker-swag:latest) I get a strange exception stacktrace: java.io.IOException: keystore password was incorrect at sun.security.pkcs12.PKCS12KeyStore.engineLoad(Unknown Source) ~[?:?] at sun.security.util.KeyStoreDelegator.engineLoad(Unknown Source) ~[?:?] at java.security.KeyStore.load(Unknown Source) ~[?:?] at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:49) ~[fuseki-server.jar:4.5.0] ... Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. ... 28 more I'm afraid I don't know what that indicates. I would appreciate any input to pinpoint and solve any or both issues above. We'd be interested in hearing what you find out. Regards, Nikolaos Beredimas
Re: Fuseki https certificate problems
Hi Andy, TL;DR: Password-less PKCS12 passwords just don't work. After more testing, I couldn't get a password-less PKCS12 certificate to work, no matter what I tried. And after reading around I suspect it's not just Jetty that suffers from this, so there is nothing to be done. As for the other issue I had with a specific OpenSSL version, it turns out it's a non-issue. The culprit was an unrelated certificate generation script that omitted the provided password when calling openssl. In any case the xml provided back in February is good. NB On Thu, Jul 7, 2022 at 12:42 PM Andy Seaborne wrote: > Hi Nikolaos, > > > On 06/07/2022 11:04, Nikolaos Beredimas wrote: > > While trying to get Fuseki running over https I found this thread from > > February > > > https://jena.markmail.org/message/2kqpd2tlinpdzpna?q=ssl+order:date-backward=1 > > > > 1. I can confirm the provided xml works (tested on Fuseki 4.5.0) > > Thanks for confirming that. > > > > > 2. I am having some issues generating the needed pkcs12 certificate file. > > > > a. When trying to generate a password-less pkcs12 file (openssl ... > > -passout pass:) Fuseki doesn't complain when loading it, but I always get > > SSL handshake errors and it doesn't work. > > It is Jetty that is handling the certificate via the JDK. > > Mentions like > > > https://stackoverflow.com/questions/58345405/how-to-use-non-password-protected-p12-ssl-certificate-in-spring-boot > > (which is nearly 3 years old) > > suggest a password was needed at some time in the past. Current jetty > documentation does not mention it one way of the other. > > > b. When trying to generate with a password I get mixed results: > > OpenSSL 1.1.1f 31 Mar 2020 running on WSL2 Ubuntu 20.04 works fine. > Fuseki > > loads the certificate and works like a charm. > > However, if I use OpenSSL 1.1.1o 3 May 2022 (running on > > docker-linuxserver/docker-swag:latest) I get a strange exception > stacktrace: > > > > java.io.IOException: keystore password was incorrect > > at sun.security.pkcs12.PKCS12KeyStore.engineLoad(Unknown Source) ~[?:?] > > at sun.security.util.KeyStoreDelegator.engineLoad(Unknown Source) ~[?:?] > > at java.security.KeyStore.load(Unknown Source) ~[?:?] > > at > > > org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:49) > > ~[fuseki-server.jar:4.5.0] > > ... > > Caused by: java.security.UnrecoverableKeyException: failed to decrypt > safe > > contents entry: javax.crypto.BadPaddingException: Given final block not > > properly padded. Such issues can arise if a bad key is used during > > decryption. > > ... 28 more > > I'm afraid I don't know what that indicates. > > > > > > > I would appreciate any input to pinpoint and solve any or both issues > above. > > We'd be interested in hearing what you find out. > > > > > Regards, > > Nikolaos Beredimas > > >
Re: Re: [MASSMAIL]Re: Large *.dat files in Fuseki
I think we should wait for Andy here with further input as he's the
persons who basically designed and implemented all the fancy stuff and
knows better advice for sure.
@Andy Did you read the whole discussion and can you verify that it's
expected behavior that lot's of daily updates lead to such a big growth
of the node table files?
On 07.07.22 10:53, Bartalus Gáspár wrote:
Hi Lorenz,
Would you recommend using tdb1 instead of tdb2 for our use case? What would be
the differences?
We are using fuseki 4.5.0 btw.
Gaspar
On 6 Jul 2022, at 14:39, Bartalus Gáspár
wrote:
Hi,
Most of the updates are DELETE/INSERT queries, i.e
DELETE {?s ?p ?oldValue}
INSERT {?s ?p ?newValue}
WHERE {
OPTIONAL {?s ?p ?oldValue}
#derive ?newValue from somewhere
}
We also have some separate DELETE queries and INSERT queries.
I’ve tried HTTP POST /$/compact/db_name and as a result the files are getting
back to normal size. However, as far as I can tell the old files are also kept.
This is the folder structure I see:
- databases/db_name/Data-0001 - with the old large files
- databases/db_name/Data-0002 - presumably the result of the compact operation
with normal file sizes.
Is there also some operation (http or cli) that would keep only one (the
latest) data folder, i.e. delete the old files from Data-0001?
Gaspar
On 6 Jul 2022, at 12:52, Lorenz Buehmann
wrote:
Ok, interesting
so
we have
- 150k triples, rather small dataset
- loaded into 10MB node table files
- 10 updates every 5s
- which makes up to 24 * 60 * 60 / 5 * 10 ~ 200k updates per day
- and leads to 10GB node table files
Can you share the shape of those update queries?
After doing a "compact" operation, the files are getting back to "normal" size?
On 06.07.22 10:36, Bartalus Gáspár wrote:
Hi Lorenz,
Thanks for quick feedback and clarification on lucene indexes.
Here are my answers to your questions:
- We are uploading 7 ttl files to our dataset, where 1 is larger 6Mb, the
others are below 200Kb.
- The overall number of triples after data upload is ~15.
- We have around 10 SPARQL UPDATE queries that are executed on a regular and
frequent basis, i.e. every 5 seconds. We also have 5 such queries that are
executed each minute. But most of the time they do not produce any outcome,
i.e. the dataset is not altered, and when they do, there are just a couple of
triples that are added to the dataset.
- These *.dat files start from ~10Mb in size, and after a day or so some of
them grow to ~10Gb.
We have ~300 blank nodes, and ~half of the triples have a literal in the object
position, so ~75000.
Best regards,
Gaspar
On 6 Jul 2022, at 10:55, Lorenz Buehmann
wrote:
Hi and welcome Gaspar.
Those files do contain the node tables.
A Lucene index is never computed by default and would be contained in Lucene
specific index files.
Can you give some details about the
- size of the files
- the number of triples
- the number triples added/removed/changed
- the frequency of updates
- how much the files grow
- what kind of data you insert? Lots of blank nodes? Or literals?
Also, did you try a compact operation during time?
Lorenz
On 06.07.22 09:40, Bartalus Gáspár wrote:
Hi Jena support team,
We are experiencing an issue with Jena Fuseki databases. In the databases
folder we see some files called SPO.dat, OSP.dat, etc., and the size of these
files are growing quickly. From our understanding these files are containing
the Lucene indexes. We would have two questions:
1. Why are these files growing rapidly, although the underlying data (triples)
are not being changed, or only slightly changed?
2. Can we disable indexing easily, since we are not using full text searches in
our SPARQL queries?
Our usage of Jena Fuseki:
* Start the server with `fuseki-server —port 3030`
* Create databases with HTTP POST to
`/$/datasets?state=active=tdb2=db_name`
* Upload ttl files with HTTP POST to /db_name/data
Thanks in advance for your feedback, and if you’d require more input from our
side, please let me know.
Best regards,
Gaspar Bartalus
Re: Fuseki https certificate problems
Can't you just provide a keystore password? https://stackoverflow.com/questions/12862655/using-an-empty-keystore-password-used-to-be-possible On Thu, Jul 7, 2022 at 11:42 AM Andy Seaborne wrote: > > Hi Nikolaos, > > > On 06/07/2022 11:04, Nikolaos Beredimas wrote: > > While trying to get Fuseki running over https I found this thread from > > February > > https://jena.markmail.org/message/2kqpd2tlinpdzpna?q=ssl+order:date-backward=1 > > > > 1. I can confirm the provided xml works (tested on Fuseki 4.5.0) > > Thanks for confirming that. > > > > > 2. I am having some issues generating the needed pkcs12 certificate file. > > > > a. When trying to generate a password-less pkcs12 file (openssl ... > > -passout pass:) Fuseki doesn't complain when loading it, but I always get > > SSL handshake errors and it doesn't work. > > It is Jetty that is handling the certificate via the JDK. > > Mentions like > > https://stackoverflow.com/questions/58345405/how-to-use-non-password-protected-p12-ssl-certificate-in-spring-boot > > (which is nearly 3 years old) > > suggest a password was needed at some time in the past. Current jetty > documentation does not mention it one way of the other. > > > b. When trying to generate with a password I get mixed results: > > OpenSSL 1.1.1f 31 Mar 2020 running on WSL2 Ubuntu 20.04 works fine. Fuseki > > loads the certificate and works like a charm. > > However, if I use OpenSSL 1.1.1o 3 May 2022 (running on > > docker-linuxserver/docker-swag:latest) I get a strange exception stacktrace: > > > > java.io.IOException: keystore password was incorrect > > at sun.security.pkcs12.PKCS12KeyStore.engineLoad(Unknown Source) ~[?:?] > > at sun.security.util.KeyStoreDelegator.engineLoad(Unknown Source) ~[?:?] > > at java.security.KeyStore.load(Unknown Source) ~[?:?] > > at > > org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:49) > > ~[fuseki-server.jar:4.5.0] > > ... > > Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe > > contents entry: javax.crypto.BadPaddingException: Given final block not > > properly padded. Such issues can arise if a bad key is used during > > decryption. > > ... 28 more > > I'm afraid I don't know what that indicates. > > > > > > > I would appreciate any input to pinpoint and solve any or both issues above. > > We'd be interested in hearing what you find out. > > > > > Regards, > > Nikolaos Beredimas > >
Re: Fuseki https certificate problems
Hi Nikolaos, On 06/07/2022 11:04, Nikolaos Beredimas wrote: While trying to get Fuseki running over https I found this thread from February https://jena.markmail.org/message/2kqpd2tlinpdzpna?q=ssl+order:date-backward=1 1. I can confirm the provided xml works (tested on Fuseki 4.5.0) Thanks for confirming that. 2. I am having some issues generating the needed pkcs12 certificate file. a. When trying to generate a password-less pkcs12 file (openssl ... -passout pass:) Fuseki doesn't complain when loading it, but I always get SSL handshake errors and it doesn't work. It is Jetty that is handling the certificate via the JDK. Mentions like https://stackoverflow.com/questions/58345405/how-to-use-non-password-protected-p12-ssl-certificate-in-spring-boot (which is nearly 3 years old) suggest a password was needed at some time in the past. Current jetty documentation does not mention it one way of the other. b. When trying to generate with a password I get mixed results: OpenSSL 1.1.1f 31 Mar 2020 running on WSL2 Ubuntu 20.04 works fine. Fuseki loads the certificate and works like a charm. However, if I use OpenSSL 1.1.1o 3 May 2022 (running on docker-linuxserver/docker-swag:latest) I get a strange exception stacktrace: java.io.IOException: keystore password was incorrect at sun.security.pkcs12.PKCS12KeyStore.engineLoad(Unknown Source) ~[?:?] at sun.security.util.KeyStoreDelegator.engineLoad(Unknown Source) ~[?:?] at java.security.KeyStore.load(Unknown Source) ~[?:?] at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:49) ~[fuseki-server.jar:4.5.0] ... Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. ... 28 more I'm afraid I don't know what that indicates. I would appreciate any input to pinpoint and solve any or both issues above. We'd be interested in hearing what you find out. Regards, Nikolaos Beredimas
Re: [MASSMAIL]Re: Large *.dat files in Fuseki
Hi Lorenz,
Would you recommend using tdb1 instead of tdb2 for our use case? What would be
the differences?
We are using fuseki 4.5.0 btw.
Gaspar
> On 6 Jul 2022, at 14:39, Bartalus Gáspár
> wrote:
>
> Hi,
>
> Most of the updates are DELETE/INSERT queries, i.e
>
> DELETE {?s ?p ?oldValue}
> INSERT {?s ?p ?newValue}
> WHERE {
> OPTIONAL {?s ?p ?oldValue}
> #derive ?newValue from somewhere
> }
>
> We also have some separate DELETE queries and INSERT queries.
>
> I’ve tried HTTP POST /$/compact/db_name and as a result the files are getting
> back to normal size. However, as far as I can tell the old files are also
> kept. This is the folder structure I see:
> - databases/db_name/Data-0001 - with the old large files
> - databases/db_name/Data-0002 - presumably the result of the compact
> operation with normal file sizes.
>
> Is there also some operation (http or cli) that would keep only one (the
> latest) data folder, i.e. delete the old files from Data-0001?
>
> Gaspar
>
>> On 6 Jul 2022, at 12:52, Lorenz Buehmann
>> wrote:
>>
>> Ok, interesting
>>
>> so
>>
>> we have
>>
>> - 150k triples, rather small dataset
>>
>> - loaded into 10MB node table files
>>
>> - 10 updates every 5s
>>
>> - which makes up to 24 * 60 * 60 / 5 * 10 ~ 200k updates per day
>>
>> - and leads to 10GB node table files
>>
>>
>> Can you share the shape of those update queries?
>>
>>
>> After doing a "compact" operation, the files are getting back to "normal"
>> size?
>>
>>
>> On 06.07.22 10:36, Bartalus Gáspár wrote:
>>> Hi Lorenz,
>>>
>>> Thanks for quick feedback and clarification on lucene indexes.
>>>
>>> Here are my answers to your questions:
>>> - We are uploading 7 ttl files to our dataset, where 1 is larger 6Mb, the
>>> others are below 200Kb.
>>> - The overall number of triples after data upload is ~15.
>>> - We have around 10 SPARQL UPDATE queries that are executed on a regular
>>> and frequent basis, i.e. every 5 seconds. We also have 5 such queries that
>>> are executed each minute. But most of the time they do not produce any
>>> outcome, i.e. the dataset is not altered, and when they do, there are just
>>> a couple of triples that are added to the dataset.
>>> - These *.dat files start from ~10Mb in size, and after a day or so some of
>>> them grow to ~10Gb.
>>>
>>> We have ~300 blank nodes, and ~half of the triples have a literal in the
>>> object position, so ~75000.
>>>
>>> Best regards,
>>> Gaspar
>>>
>>>
>>>
On 6 Jul 2022, at 10:55, Lorenz Buehmann
wrote:
Hi and welcome Gaspar.
Those files do contain the node tables.
A Lucene index is never computed by default and would be contained in
Lucene specific index files.
Can you give some details about the
- size of the files
- the number of triples
- the number triples added/removed/changed
- the frequency of updates
- how much the files grow
- what kind of data you insert? Lots of blank nodes? Or literals?
Also, did you try a compact operation during time?
Lorenz
On 06.07.22 09:40, Bartalus Gáspár wrote:
> Hi Jena support team,
>
> We are experiencing an issue with Jena Fuseki databases. In the databases
> folder we see some files called SPO.dat, OSP.dat, etc., and the size of
> these files are growing quickly. From our understanding these files are
> containing the Lucene indexes. We would have two questions:
>
> 1. Why are these files growing rapidly, although the underlying data
> (triples) are not being changed, or only slightly changed?
> 2. Can we disable indexing easily, since we are not using full text
> searches in our SPARQL queries?
>
> Our usage of Jena Fuseki:
>
> * Start the server with `fuseki-server —port 3030`
> * Create databases with HTTP POST to
> `/$/datasets?state=active=tdb2=db_name`
> * Upload ttl files with HTTP POST to /db_name/data
>
> Thanks in advance for your feedback, and if you’d require more input from
> our side, please let me know.
>
> Best regards,
> Gaspar Bartalus
>
>
smime.p7s
Description: S/MIME cryptographic signature
