You should have another file for the dataset configuration too, inside "run/configuration/".
Sent: Tuesday, February 06, 2018 at 4:13 PM From: "Mikael Pesonen" <[email protected]> To: [email protected] Subject: Re: Limit memory usage of Fuseki server? # Licensed under the terms of http://www.apache.org/licenses/LICENSE-2.0 ## Fuseki Server configuration file. @prefix : <#> . @prefix fuseki: <http://jena.apache.org/fuseki#[http://jena.apache.org/fuseki#]> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#[http://www.w3.org/1999/02/22-rdf-syntax-ns#]> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#[http://www.w3.org/2000/01/rdf-schema#]> . @prefix ja: <http://jena.hpl.hp.com/2005/11/Assembler#[http://jena.hpl.hp.com/2005/11/Assembler#]> . [] rdf:type fuseki:Server ; # Example:: # Server-wide query timeout. # # Timeout - server-wide default: milliseconds. # Format 1: "1000" -- 1 second timeout # Format 2: "10000,60000" -- 10s timeout to first result, # then 60s timeout for the rest of query. # # See javadoc for ARQ.queryTimeout for details. # This can also be set on a per dataset basis in the dataset assembler. # # ja:context [ ja:cxtName "arq:queryTimeout" ; ja:cxtValue "30000" ] ; # Add any custom classes you want to load. # Must have a "public static void init()" method. # ja:loadClass "your.code.Class" ; # End triples. . On 6.2.2018 16:58, Lorenz Buehmann wrote: > I guess Andy was hoping to see the Fuseki config file(s) > > > On 06.02.2018 15:56, Mikael Pesonen wrote: >> Is it this one? >> >> File: /lib/systemd/system/apache-jena-fuseki.service: >> >> [Unit] >> >> Description=Apache Jena Fuseki >> >> >> [Service] >> >> Type=simple >> >> User=fuseki >> >> Environment=JAVA_HOME=/usr/lib/jvm/java-8-oracle/ >> >> Environment=FUSEKI_HOME=/opt/insight/jena/ >> >> Environment=FUSEKI_BASE=/opt/insight/jena/run >> >> ExecStart=/usr/lib/jvm/java-8-oracle/bin/java -jar >> /opt/insight/jena/fuseki-server.jar --update --port 3030 >> --loc=/opt/insight/jena_data /ds >> >> >> [Install] >> >> WantedBy=multi-user.target >> >> >> File: /lib/systemd/system/apache-jena-fuseki.path >> >> [Path] >> >> PathExistsGlob=/var/crash/*.crash >> >> Unit=apache-jena-fuseki.service >> >> >> >> On 6.2.2018 16:46, Andy Seaborne wrote: >>> What is the dataset/service setup? >>> >>> On 06/02/18 14:33, Mikael Pesonen wrote: >>>> I'm not good with Linux so don't know what setup means. We have >>>> Ubuntu 14.04, Jena Fuseki 3.6.0 with default config.ttl and running >>>> as a service. >>>> >>>> Test server which is having both issues is running on Upcloud: >>>> >>>> $ cat /proc/meminfo >>>> MemTotal: 8692992 kB >>>> MemFree: 6716504 kB >>>> Buffers: 50052 kB >>>> Cached: 508684 kB >>>> SwapCached: 0 kB >>>> Active: 1385768 kB >>>> Inactive: 469640 kB >>>> Active(anon): 1300740 kB >>>> Inactive(anon): 9076 kB >>>> Active(file): 85028 kB >>>> Inactive(file): 460564 kB >>>> Unevictable: 0 kB >>>> Mlocked: 0 kB >>>> SwapTotal: 0 kB >>>> SwapFree: 0 kB >>>> Dirty: 48 kB >>>> Writeback: 0 kB >>>> AnonPages: 1296688 kB >>>> Mapped: 125308 kB >>>> Shmem: 13128 kB >>>> Slab: 30824 kB >>>> SReclaimable: 16120 kB >>>> SUnreclaim: 14704 kB >>>> KernelStack: 1520 kB >>>> PageTables: 12920 kB >>>> NFS_Unstable: 0 kB >>>> Bounce: 0 kB >>>> WritebackTmp: 0 kB >>>> CommitLimit: 4346496 kB >>>> Committed_AS: 1669692 kB >>>> VmallocTotal: 34359738367 kB >>>> VmallocUsed: 34708 kB >>>> VmallocChunk: 34359694588 kB >>>> HardwareCorrupted: 0 kB >>>> AnonHugePages: 1193984 kB >>>> HugePages_Total: 0 >>>> HugePages_Free: 0 >>>> HugePages_Rsvd: 0 >>>> HugePages_Surp: 0 >>>> Hugepagesize: 2048 kB >>>> DirectMap4k: 46960 kB >>>> DirectMap2M: 2574336 kB >>>> DirectMap1G: 6291456 kB >>>> >>>> Anything else? >>>> >>>> The setup in which I have to concurrency error every time, is apache >>>> web server running php, which is calling Fuseki GSP with curl. With >>>> 4 concurrent tests i can't reproduce the error with some trying, >>>> with 5 concurrent tests error occurs every time. >>>> >>>> >>>> >>>> On 6.2.2018 16:14, Andy Seaborne wrote: >>>>> I don't know what's going on. I can't start to try to reproduce it >>>>> because I don't know what the trigger is. It might be a previously >>>>> corrupted database or something messing with the DB behind the >>>>> server the effect is right but it might well be something else. >>>>> >>>>> ajs6f asked for details of the setup - could you provide those >>>>> please? and a complete minimal example. The fact you can't create >>>>> one points towards an environment/config/hardware issue, not a code >>>>> bug. >>>>> >>>>> Andy >>>>> >>>>> On 06/02/18 13:14, Mikael Pesonen wrote: >>>>>> Do you have now enough info for figuring out what's going on? It's >>>>>> a bug that can be fixed? >>>>>> >>>>>> I guess this is not related to concurrency issue I sent on another >>>>>> thread? Both issues are happening on same setup. >>>>>> >>>>>> Br >>>>>> >>>>>> On 6.2.2018 14:18, Andy Seaborne wrote: >>>>>>> Expections in alloc-write seem to be a cause of you rother issues >>>>>>> so, maybe, this is the same rot cause with a different >>>>>>> manifestation. It would have been useful to know this at the >>>>>>> start of the thread. It could be the broken alloc-write is simply >>>>>>> asking for a huge amount of RAM becaue it says: >>>>>>> >>>>>>> report_vm_out_of_memory >>>>>>> >>>>>>> so it is not the system killing Fuseki, its the JVM exiting. >>>>>>> >>>>>>> With a single TDB, 12G is way too much heap (try 2G) but the >>>>>>> figure may include mapped files. >>>>>>> >>>>>>> Andy >>>>>>> >>>>>>> On 06/02/18 09:56, Mikael Pesonen wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> ok I'll try that. I had out of memory error on other server now. >>>>>>>> Is there a place I could post log files, if they help? >>>>>>>> >>>>>>>> [2018-02-06 11:48:51] BindingTDB ERROR get1(?o) >>>>>>>> org.apache.jena.tdb.base.file.FileException: In the middle of an >>>>>>>> alloc-write >>>>>>>> at >>>>>>>> org.apache.jena.tdb.base.objectfile.ObjectFileStorage.read(ObjectFileStorage.java:311) >>>>>>>> >>>>>>>> ... >>>>>>>> [2018-02-06 11:48:52] BindingTDB ERROR get1(?o) >>>>>>>> java.nio.BufferOverflowException >>>>>>>> at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:214) >>>>>>>> at sun.nio.ch.IOUtil.read(IOUtil.java:200) >>>>>>>> at >>>>>>>> sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:741) >>>>>>>> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727) >>>>>>>> at >>>>>>>> org.apache.jena.tdb.base.file.BufferChannelFile.read(BufferChannelFile.java:112) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Stack: [0x00007f1956dda000,0x00007f1956edb000], >>>>>>>> sp=0x00007f1956ed9720, free space=1021k >>>>>>>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, >>>>>>>> C=native code) >>>>>>>> V [libjvm.so+0xaba7ea] VMError::report_and_die()+0x2ba >>>>>>>> V [libjvm.so+0x4f9e3b] report_vm_out_of_memory(char const*, >>>>>>>> int, unsigned long, VMErrorType, char const*)+0x8b >>>>>>>> V [libjvm.so+0x91b613] os::Linux::commit_memory_impl(char*, >>>>>>>> unsigned long, bool)+0x103 >>>>>>>> V [libjvm.so+0x91b6dc] os::pd_commit_memory(char*, unsigned >>>>>>>> long, bool)+0xc >>>>>>>> V [libjvm.so+0x91518a] os::commit_memory(char*, unsigned long, >>>>>>>> bool)+0x2a >>>>>>>> V [libjvm.so+0x9198af] os::pd_create_stack_guard_pages(char*, >>>>>>>> unsigned long)+0x7f >>>>>>>> V [libjvm.so+0xa6049e] JavaThread::create_stack_guard_pages()+0x5e >>>>>>>> V [libjvm.so+0xa69e14] JavaThread::run()+0x34 >>>>>>>> V [libjvm.so+0x91d9d8] java_start(Thread*)+0x108 >>>>>>>> C [libpthread.so.0+0x8182] start_thread+0xc2 >>>>>>>> >>>>>>>> On 6.2.2018 10:57, Laura Morales wrote: >>>>>>>>>> kernel: Killed process 29037 (java) total-vm:12745796kB >>>>>>>>> 12.7GB? Used by Fuseki? Or does this account other processes as >>>>>>>>> well? There is no way Fuseki is using 12GB of memory for a few >>>>>>>>> thousands of triples, unless maybe memory is leaked during >>>>>>>>> update operations. You should try try: >>>>>>>>> >>>>>>>>> 1. start Fuseki with your database >>>>>>>>> 2. run a lot of live update queries (insert/delete/update data) >>>>>>>>> and monitor how your VM responds >>>>>>>>> >>>>>>>>> >>>>>>>>> Sent: Tuesday, February 06, 2018 at 9:22 AM >>>>>>>>> From: "Mikael Pesonen" <[email protected]> >>>>>>>>> To: [email protected] >>>>>>>>> Subject: Re: Limit memory usage of Fuseki server? >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> config.ttl is defaut, Fuseki is ran as service, here is the log >>>>>>>>> from >>>>>>>>> around the kill (notice fairly high VM) >>>>>>>>> >>>>>>>>> kernel: Out of memory: Kill process 29037 (java) score 807 or >>>>>>>>> sacrifice child >>>>>>>>> kernel: Killed process 29037 (java) total-vm:12745796kB, >>>>>>>>> anon-rss:3458140kB, file-rss:0kB >>>>>>>>> >>>>>>>>> systemd[1]: apache-jena-fuseki.service: Main process exited, >>>>>>>>> code=killed, status=9/KILL >>>>>>>>> systemd[1]: apache-jena-fuseki.service: Unit entered failed >>>>>>>>> state. >>>>>>>>> systemd[1]: apache-jena-fuseki.service: Failed with result >>>>>>>>> 'signal'. >>>>>>>>> >>>>>>>>> Usage is not much, around few thousands of triplets, post, get and >>>>>>>>> delete, using Fuseki's GSP over HTTP. Uptime was around one week. >>>>>>>>> >>>>>>>>> Sorry can't think of anything else to tell about the system. >>>>>>>>> >>>>>>>>> Br >>>>>>>>> >>>>>>>>> >>>>>>>>> On 5.2.2018 18:08, ajs6f wrote: >>>>>>>>>> Can you tell us more about your configuration, data, workload, >>>>>>>>>> and the timing of the problem? >>>>>>>>>> >>>>>>>>>> ajs6f >>>>>>>>>> >>>>>>>>>>> On Feb 5, 2018, at 7:08 AM, Laura Morales <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> You can tell the Kernel to limit memory usage per process or >>>>>>>>>>> per user. If you set the limit too low your process will be >>>>>>>>>>> killed because it can't allocate more memory, but you can >>>>>>>>>>> tell the Kernel to only limit resident memory while using up >>>>>>>>>>> all the virtual memory it wants. You can do this with cgroups. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Sent: Monday, February 05, 2018 at 12:56 PM >>>>>>>>>>> From: "Mikael Pesonen" <[email protected]> >>>>>>>>>>> To: [email protected] >>>>>>>>>>> Subject: Limit memory usage of Fuseki server? >>>>>>>>>>> We have a cloud server with 4gb of memory and after a while >>>>>>>>>>> system is >>>>>>>>>>> killing Fuseki server java process for using up all memory. >>>>>>>>>>> Is it >>>>>>>>>>> possible to limit memory usage, and what would be the >>>>>>>>>>> performance >>>>>>>>>>> penalty for doing that? >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Lingsoft - 30 years of Leading Language Management >>>>>>>>>>> >>>>>>>>>>> www.lingsoft.fi[http://www.lingsoft.fi][http://www.lingsoft.fi[http://www.lingsoft.fi]][http://www.lingsoft.fi[http://www.lingsoft.fi][http://www.lingsoft.fi[http://www.lingsoft.fi]]] >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Speech Applications - Language Management - Translation - >>>>>>>>>>> Reader's and Writer's Tools - Text Tools - E-books and M-books >>>>>>>>>>> >>>>>>>>>>> Mikael Pesonen >>>>>>>>>>> System Engineer >>>>>>>>>>> >>>>>>>>>>> e-mail: [email protected] >>>>>>>>>>> Tel. +358 2 279 3300 >>>>>>>>>>> >>>>>>>>>>> Time zone: GMT+2 >>>>>>>>>>> >>>>>>>>>>> Helsinki Office >>>>>>>>>>> Eteläranta 10 >>>>>>>>>>> FI-00130 Helsinki >>>>>>>>>>> FINLAND >>>>>>>>>>> >>>>>>>>>>> Turku Office >>>>>>>>>>> Kauppiaskatu 5 A >>>>>>>>>>> FI-20100 Turku >>>>>>>>>>> FINLAND >>>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Lingsoft - 30 years of Leading Language Management >>>>>>>>> >>>>>>>>> www.lingsoft.fi[http://www.lingsoft.fi][http://www.lingsoft.fi[http://www.lingsoft.fi]] >>>>>>>>> >>>>>>>>> Speech Applications - Language Management - Translation - >>>>>>>>> Reader's and Writer's Tools - Text Tools - E-books and M-books >>>>>>>>> >>>>>>>>> Mikael Pesonen >>>>>>>>> System Engineer >>>>>>>>> >>>>>>>>> e-mail: [email protected] >>>>>>>>> Tel. +358 2 279 3300 >>>>>>>>> >>>>>>>>> Time zone: GMT+2 >>>>>>>>> >>>>>>>>> Helsinki Office >>>>>>>>> Eteläranta 10 >>>>>>>>> FI-00130 Helsinki >>>>>>>>> FINLAND >>>>>>>>> >>>>>>>>> Turku Office >>>>>>>>> Kauppiaskatu 5 A >>>>>>>>> FI-20100 Turku >>>>>>>>> FINLAND -- Lingsoft - 30 years of Leading Language Management www.lingsoft.fi[http://www.lingsoft.fi] Speech Applications - Language Management - Translation - Reader's and Writer's Tools - Text Tools - E-books and M-books Mikael Pesonen System Engineer e-mail: [email protected] Tel. +358 2 279 3300 Time zone: GMT+2 Helsinki Office Eteläranta 10 FI-00130 Helsinki FINLAND Turku Office Kauppiaskatu 5 A FI-20100 Turku FINLAND
