The OOM killer is a Linux feature designed to prevent one program monopolising
the system. I've never seen it hitting Fuseki before but have seen it hit
other database processes in the past because in-memory/aggressively cached
databases like Fuseki do tend to be large memory consumers.
See the documentation for how it functions, the following is my preferred
explanation but by no means the only available article on the topic:
https://lwn.net/Articles/317814/
Is this system/VM being used for multiple processes or just Fuseki?
If the former then other processes who request lots of memory might be causing
Fuseki to get chosen by the OOM killer because it happens to be the biggest
memory consumer at the time when the system is running out of memory. If it is
the latter and the system is dedicated to Fuseki you may want to tune/disable
the OOM killer. Per the article you can turn it off entirely for specific
processes if desired while still leaving it enabled to protect the wider system.
Rob
On 10/04/2018, 14:38, "Mikael Pesonen" <mikael.peso...@lingsoft.fi> wrote:
More info:
latest Fuseki query returned 200 so no error there.
Apr 10 15:54:29 -- kernel: curl invoked oom-killer: gfp_mask=0x24201ca,
order=0, oom_score_adj=0
Apr 10 15:54:29 -- kernel: curl cpuset=/ mems_allowed=0
Apr 10 15:54:29 -- kernel: CPU: 2 PID: 26434 Comm: curl Not tainted
4.4.0-109-generic #132-Ubuntu
Apr 10 15:54:29 -- kernel: Hardware name: VMware, Inc. VMware Virtual
Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
Apr 10 15:54:29 -- kernel: 0000000000000286 7a3c1686bf608bd4
ffff88008c04ba20 ffffffff813fbb03
Apr 10 15:54:29 -- kernel: ffff88008c04bbd8 ffff88002aed0e00
ffff88008c04ba90 ffffffff8120d3de
Apr 10 15:54:29 -- kernel: ffff88008c04ba40 ffffffff81140b8a
ffff88008c04bac0 ffffffff811a84b6
Apr 10 15:54:29 -- kernel: Call Trace:
Apr 10 15:54:29 -- kernel: [<ffffffff813fbb03>] dump_stack+0x63/0x90
Apr 10 15:54:29 -- kernel: [<ffffffff8120d3de>] dump_header+0x5a/0x1c5
Apr 10 15:54:29 -- kernel: [<ffffffff81140b8a>] ?
__delayacct_freepages_end+0x2a/0x30
Apr 10 15:54:29 -- kernel: [<ffffffff811a84b6>] ?
do_try_to_free_pages+0x2a6/0x3b0
Apr 10 15:54:29 -- kernel: [<ffffffff81193f82>]
oom_kill_process+0x202/0x3c0
Apr 10 15:54:29 -- kernel: [<ffffffff811943a9>] out_of_memory+0x219/0x460
Apr 10 15:54:29 -- kernel: [<ffffffff8119a3b5>]
__alloc_pages_slowpath.constprop.88+0x965/0xb00
Apr 10 15:54:29 -- kernel: [<ffffffff8119a7d6>]
__alloc_pages_nodemask+0x286/0x2a0
Apr 10 15:54:29 -- kernel: [<ffffffff811e411c>]
alloc_pages_current+0x8c/0x110
Apr 10 15:54:29 -- kernel: [<ffffffff8119054b>]
__page_cache_alloc+0xab/0xc0
Apr 10 15:54:29 -- kernel: [<ffffffff81192a5a>] filemap_fault+0x14a/0x3f0
Apr 10 15:54:29 -- kernel: [<ffffffff811bf750>] __do_fault+0x50/0xe0
Apr 10 15:54:29 -- kernel: [<ffffffff811c32a2>]
handle_mm_fault+0xfa2/0x1820
Apr 10 15:54:29 -- kernel: [<ffffffff8121079b>] ? new_sync_write+0x9b/0xe0
Apr 10 15:54:29 -- kernel: [<ffffffff8106b687>]
__do_page_fault+0x197/0x400
Apr 10 15:54:29 -- kernel: [<ffffffff8106b912>] do_page_fault+0x22/0x30
Apr 10 15:54:29 -- kernel: [<ffffffff81846a18>] page_fault+0x28/0x30
Apr 10 15:54:29 -- kernel: Mem-Info:
Apr 10 15:54:29 -- kernel: active_anon:667278 inactive_anon:263114
isolated_anon:0
active_file:309
inactive_file:1592 isolated_file:18
unevictable:913 dirty:817
writeback:0 unstable:0
slab_reclaimable:11983
slab_unreclaimable:8639
mapped:890 shmem:1798
pagetables:8491 bounce:0
free:22224 free_pcp:0 free_cma:0
On 10.4.2018 16:26, Mikael Pesonen wrote:
>
> We still have this issue:
>
> Apr 10 15:54:29 -- kernel: Out of memory: Kill process 17136 (java)
> score 770 or sacrifice child
> Apr 10 15:54:29 -- kernel: Killed process 17136 (java)
> total-vm:9760620kB, anon-rss:3536204kB, file-rss:0kB
> Apr 10 15:54:29 -- systemd[1]: apache-jena-fuseki.service: Main
> process exited, code=killed, status=9/KILL
> Apr 10 15:54:29 -- systemd[1]: apache-jena-fuseki.service: Unit
> entered failed state.
> Apr 10 15:54:29 -- systemd[1]: apache-jena-fuseki.service: Failed with
> result 'signal'.
>
> Any idea how to solve please?
>
> On 6.2.2018 19:34, Andy Seaborne wrote:
>> In this case, the details are on the command line:
>>
>>> ExecStart=/usr/lib/jvm/java-8-oracle/bin/java -jar
>>> /opt/insight/jena/fuseki-server.jar --update --port 3030
>>> --loc=/opt/insight/jena_data /ds
>>
>> Andy
>>
>> On 06/02/18 16:11, Laura Morales wrote:
>>> You should have another file for the dataset configuration too,
>>> inside "run/configuration/".
>>>
>>> Sent: Tuesday, February 06, 2018 at 4:13 PM
>>> From: "Mikael Pesonen" <mikael.peso...@lingsoft.fi>
>>> To: users@jena.apache.org
>>> Subject: Re: Limit memory usage of Fuseki server?
>>> # Licensed under the terms of
>>> http://www.apache.org/licenses/LICENSE-2.0
>>>
>>> ## Fuseki Server configuration file.
>>>
>>> @prefix : <#> .
>>> @prefix fuseki:
>>> <http://jena.apache.org/fuseki#[http://jena.apache.org/fuseki#]> .
>>> @prefix rdf:
>>>
<http://www.w3.org/1999/02/22-rdf-syntax-ns#[http://www.w3.org/1999/02/22-rdf-syntax-ns#]>
>>> .
>>> @prefix rdfs:
>>>
<http://www.w3.org/2000/01/rdf-schema#[http://www.w3.org/2000/01/rdf-schema#]>
>>> .
>>> @prefix ja:
>>>
<http://jena.hpl.hp.com/2005/11/Assembler#[http://jena.hpl.hp.com/2005/11/Assembler#]>
>>> .
>>>
>>> [] rdf:type fuseki:Server ;
>>> # Example::
>>> # Server-wide query timeout.
>>> #
>>> # Timeout - server-wide default: milliseconds.
>>> # Format 1: "1000" -- 1 second timeout
>>> # Format 2: "10000,60000" -- 10s timeout to first result,
>>> # then 60s timeout for the rest of
>>> query.
>>> #
>>> # See javadoc for ARQ.queryTimeout for details.
>>> # This can also be set on a per dataset basis in the dataset
>>> assembler.
>>> #
>>> # ja:context [ ja:cxtName "arq:queryTimeout" ; ja:cxtValue
>>> "30000" ] ;
>>>
>>> # Add any custom classes you want to load.
>>> # Must have a "public static void init()" method.
>>> # ja:loadClass "your.code.Class" ;
>>>
>>> # End triples.
>>> .
>>>
>>>
>>>
>>> On 6.2.2018 16:58, Lorenz Buehmann wrote:
>>>> I guess Andy was hoping to see the Fuseki config file(s)
>>>>
>>>>
>>>> On 06.02.2018 15:56, Mikael Pesonen wrote:
>>>>> Is it this one?
>>>>>
>>>>> File: /lib/systemd/system/apache-jena-fuseki.service:
>>>>>
>>>>> [Unit]
>>>>>
>>>>> Description=Apache Jena Fuseki
>>>>>
>>>>>
>>>>> [Service]
>>>>>
>>>>> Type=simple
>>>>>
>>>>> User=fuseki
>>>>>
>>>>> Environment=JAVA_HOME=/usr/lib/jvm/java-8-oracle/
>>>>>
>>>>> Environment=FUSEKI_HOME=/opt/insight/jena/
>>>>>
>>>>> Environment=FUSEKI_BASE=/opt/insight/jena/run
>>>>>
>>>>> ExecStart=/usr/lib/jvm/java-8-oracle/bin/java -jar
>>>>> /opt/insight/jena/fuseki-server.jar --update --port 3030
>>>>> --loc=/opt/insight/jena_data /ds
>>>>>
>>>>>
>>>>> [Install]
>>>>>
>>>>> WantedBy=multi-user.target
>>>>>
>>>>>
>>>>> File: /lib/systemd/system/apache-jena-fuseki.path
>>>>>
>>>>> [Path]
>>>>>
>>>>> PathExistsGlob=/var/crash/*.crash
>>>>>
>>>>> Unit=apache-jena-fuseki.service
>>>>>
>>>>>
>>>>>
>>>>> On 6.2.2018 16:46, Andy Seaborne wrote:
>>>>>> What is the dataset/service setup?
>>>>>>
>>>>>> On 06/02/18 14:33, Mikael Pesonen wrote:
>>>>>>> I'm not good with Linux so don't know what setup means. We have
>>>>>>> Ubuntu 14.04, Jena Fuseki 3.6.0 with default config.ttl and running
>>>>>>> as a service.
>>>>>>>
>>>>>>> Test server which is having both issues is running on Upcloud:
>>>>>>>
>>>>>>> $ cat /proc/meminfo
>>>>>>> MemTotal: 8692992 kB
>>>>>>> MemFree: 6716504 kB
>>>>>>> Buffers: 50052 kB
>>>>>>> Cached: 508684 kB
>>>>>>> SwapCached: 0 kB
>>>>>>> Active: 1385768 kB
>>>>>>> Inactive: 469640 kB
>>>>>>> Active(anon): 1300740 kB
>>>>>>> Inactive(anon): 9076 kB
>>>>>>> Active(file): 85028 kB
>>>>>>> Inactive(file): 460564 kB
>>>>>>> Unevictable: 0 kB
>>>>>>> Mlocked: 0 kB
>>>>>>> SwapTotal: 0 kB
>>>>>>> SwapFree: 0 kB
>>>>>>> Dirty: 48 kB
>>>>>>> Writeback: 0 kB
>>>>>>> AnonPages: 1296688 kB
>>>>>>> Mapped: 125308 kB
>>>>>>> Shmem: 13128 kB
>>>>>>> Slab: 30824 kB
>>>>>>> SReclaimable: 16120 kB
>>>>>>> SUnreclaim: 14704 kB
>>>>>>> KernelStack: 1520 kB
>>>>>>> PageTables: 12920 kB
>>>>>>> NFS_Unstable: 0 kB
>>>>>>> Bounce: 0 kB
>>>>>>> WritebackTmp: 0 kB
>>>>>>> CommitLimit: 4346496 kB
>>>>>>> Committed_AS: 1669692 kB
>>>>>>> VmallocTotal: 34359738367 kB
>>>>>>> VmallocUsed: 34708 kB
>>>>>>> VmallocChunk: 34359694588 kB
>>>>>>> HardwareCorrupted: 0 kB
>>>>>>> AnonHugePages: 1193984 kB
>>>>>>> HugePages_Total: 0
>>>>>>> HugePages_Free: 0
>>>>>>> HugePages_Rsvd: 0
>>>>>>> HugePages_Surp: 0
>>>>>>> Hugepagesize: 2048 kB
>>>>>>> DirectMap4k: 46960 kB
>>>>>>> DirectMap2M: 2574336 kB
>>>>>>> DirectMap1G: 6291456 kB
>>>>>>>
>>>>>>> Anything else?
>>>>>>>
>>>>>>> The setup in which I have to concurrency error every time, is
>>>>>>> apache
>>>>>>> web server running php, which is calling Fuseki GSP with curl. With
>>>>>>> 4 concurrent tests i can't reproduce the error with some trying,
>>>>>>> with 5 concurrent tests error occurs every time.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 6.2.2018 16:14, Andy Seaborne wrote:
>>>>>>>> I don't know what's going on. I can't start to try to reproduce it
>>>>>>>> because I don't know what the trigger is. It might be a
>>>>>>>> previously
>>>>>>>> corrupted database or something messing with the DB behind the
>>>>>>>> server the effect is right but it might well be something else.
>>>>>>>>
>>>>>>>> ajs6f asked for details of the setup - could you provide those
>>>>>>>> please? and a complete minimal example. The fact you can't create
>>>>>>>> one points towards an environment/config/hardware issue, not a
>>>>>>>> code
>>>>>>>> bug.
>>>>>>>>
>>>>>>>> Andy
>>>>>>>>
>>>>>>>> On 06/02/18 13:14, Mikael Pesonen wrote:
>>>>>>>>> Do you have now enough info for figuring out what's going on?
>>>>>>>>> It's
>>>>>>>>> a bug that can be fixed?
>>>>>>>>>
>>>>>>>>> I guess this is not related to concurrency issue I sent on
>>>>>>>>> another
>>>>>>>>> thread? Both issues are happening on same setup.
>>>>>>>>>
>>>>>>>>> Br
>>>>>>>>>
>>>>>>>>> On 6.2.2018 14:18, Andy Seaborne wrote:
>>>>>>>>>> Expections in alloc-write seem to be a cause of you rother
>>>>>>>>>> issues
>>>>>>>>>> so, maybe, this is the same rot cause with a different
>>>>>>>>>> manifestation. It would have been useful to know this at the
>>>>>>>>>> start of the thread. It could be the broken alloc-write is
>>>>>>>>>> simply
>>>>>>>>>> asking for a huge amount of RAM becaue it says:
>>>>>>>>>>
>>>>>>>>>> report_vm_out_of_memory
>>>>>>>>>>
>>>>>>>>>> so it is not the system killing Fuseki, its the JVM exiting.
>>>>>>>>>>
>>>>>>>>>> With a single TDB, 12G is way too much heap (try 2G) but the
>>>>>>>>>> figure may include mapped files.
>>>>>>>>>>
>>>>>>>>>> Andy
>>>>>>>>>>
>>>>>>>>>> On 06/02/18 09:56, Mikael Pesonen wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> ok I'll try that. I had out of memory error on other server
>>>>>>>>>>> now.
>>>>>>>>>>> Is there a place I could post log files, if they help?
>>>>>>>>>>>
>>>>>>>>>>> [2018-02-06 11:48:51] BindingTDB ERROR get1(?o)
>>>>>>>>>>> org.apache.jena.tdb.base.file.FileException: In the middle
>>>>>>>>>>> of an
>>>>>>>>>>> alloc-write
>>>>>>>>>>> at
>>>>>>>>>>>
org.apache.jena.tdb.base.objectfile.ObjectFileStorage.read(ObjectFileStorage.java:311)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ...
>>>>>>>>>>> [2018-02-06 11:48:52] BindingTDB ERROR get1(?o)
>>>>>>>>>>> java.nio.BufferOverflowException
>>>>>>>>>>> at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:214)
>>>>>>>>>>> at sun.nio.ch.IOUtil.read(IOUtil.java:200)
>>>>>>>>>>> at
>>>>>>>>>>>
sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:741)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:727)
>>>>>>>>>>> at
>>>>>>>>>>>
org.apache.jena.tdb.base.file.BufferChannelFile.read(BufferChannelFile.java:112)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Stack: [0x00007f1956dda000,0x00007f1956edb000],
>>>>>>>>>>> sp=0x00007f1956ed9720, free space=1021k
>>>>>>>>>>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM
>>>>>>>>>>> code,
>>>>>>>>>>> C=native code)
>>>>>>>>>>> V [libjvm.so+0xaba7ea] VMError::report_and_die()+0x2ba
>>>>>>>>>>> V [libjvm.so+0x4f9e3b] report_vm_out_of_memory(char const*,
>>>>>>>>>>> int, unsigned long, VMErrorType, char const*)+0x8b
>>>>>>>>>>> V [libjvm.so+0x91b613] os::Linux::commit_memory_impl(char*,
>>>>>>>>>>> unsigned long, bool)+0x103
>>>>>>>>>>> V [libjvm.so+0x91b6dc] os::pd_commit_memory(char*, unsigned
>>>>>>>>>>> long, bool)+0xc
>>>>>>>>>>> V [libjvm.so+0x91518a] os::commit_memory(char*, unsigned long,
>>>>>>>>>>> bool)+0x2a
>>>>>>>>>>> V [libjvm.so+0x9198af] os::pd_create_stack_guard_pages(char*,
>>>>>>>>>>> unsigned long)+0x7f
>>>>>>>>>>> V [libjvm.so+0xa6049e]
>>>>>>>>>>> JavaThread::create_stack_guard_pages()+0x5e
>>>>>>>>>>> V [libjvm.so+0xa69e14] JavaThread::run()+0x34
>>>>>>>>>>> V [libjvm.so+0x91d9d8] java_start(Thread*)+0x108
>>>>>>>>>>> C [libpthread.so.0+0x8182] start_thread+0xc2
>>>>>>>>>>>
>>>>>>>>>>> On 6.2.2018 10:57, Laura Morales wrote:
>>>>>>>>>>>>> kernel: Killed process 29037 (java) total-vm:12745796kB
>>>>>>>>>>>> 12.7GB? Used by Fuseki? Or does this account other
>>>>>>>>>>>> processes as
>>>>>>>>>>>> well? There is no way Fuseki is using 12GB of memory for a few
>>>>>>>>>>>> thousands of triples, unless maybe memory is leaked during
>>>>>>>>>>>> update operations. You should try try:
>>>>>>>>>>>>
>>>>>>>>>>>> 1. start Fuseki with your database
>>>>>>>>>>>> 2. run a lot of live update queries (insert/delete/update
>>>>>>>>>>>> data)
>>>>>>>>>>>> and monitor how your VM responds
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Sent: Tuesday, February 06, 2018 at 9:22 AM
>>>>>>>>>>>> From: "Mikael Pesonen" <mikael.peso...@lingsoft.fi>
>>>>>>>>>>>> To: users@jena.apache.org
>>>>>>>>>>>> Subject: Re: Limit memory usage of Fuseki server?
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> config.ttl is defaut, Fuseki is ran as service, here is the
>>>>>>>>>>>> log
>>>>>>>>>>>> from
>>>>>>>>>>>> around the kill (notice fairly high VM)
>>>>>>>>>>>>
>>>>>>>>>>>> kernel: Out of memory: Kill process 29037 (java) score
>>>>>>>>>>>> 807 or
>>>>>>>>>>>> sacrifice child
>>>>>>>>>>>> kernel: Killed process 29037 (java) total-vm:12745796kB,
>>>>>>>>>>>> anon-rss:3458140kB, file-rss:0kB
>>>>>>>>>>>>
>>>>>>>>>>>> systemd[1]: apache-jena-fuseki.service: Main process exited,
>>>>>>>>>>>> code=killed, status=9/KILL
>>>>>>>>>>>> systemd[1]: apache-jena-fuseki.service: Unit entered failed
>>>>>>>>>>>> state.
>>>>>>>>>>>> systemd[1]: apache-jena-fuseki.service: Failed with result
>>>>>>>>>>>> 'signal'.
>>>>>>>>>>>>
>>>>>>>>>>>> Usage is not much, around few thousands of triplets, post,
>>>>>>>>>>>> get and
>>>>>>>>>>>> delete, using Fuseki's GSP over HTTP. Uptime was around one
>>>>>>>>>>>> week.
>>>>>>>>>>>>
>>>>>>>>>>>> Sorry can't think of anything else to tell about the system.
>>>>>>>>>>>>
>>>>>>>>>>>> Br
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 5.2.2018 18:08, ajs6f wrote:
>>>>>>>>>>>>> Can you tell us more about your configuration, data,
>>>>>>>>>>>>> workload,
>>>>>>>>>>>>> and the timing of the problem?
>>>>>>>>>>>>>
>>>>>>>>>>>>> ajs6f
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Feb 5, 2018, at 7:08 AM, Laura Morales
>>>>>>>>>>>>>> <laure...@mail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You can tell the Kernel to limit memory usage per process or
>>>>>>>>>>>>>> per user. If you set the limit too low your process will be
>>>>>>>>>>>>>> killed because it can't allocate more memory, but you can
>>>>>>>>>>>>>> tell the Kernel to only limit resident memory while using up
>>>>>>>>>>>>>> all the virtual memory it wants. You can do this with
>>>>>>>>>>>>>> cgroups.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sent: Monday, February 05, 2018 at 12:56 PM
>>>>>>>>>>>>>> From: "Mikael Pesonen" <mikael.peso...@lingsoft.fi>
>>>>>>>>>>>>>> To: users@jena.apache.org
>>>>>>>>>>>>>> Subject: Limit memory usage of Fuseki server?
>>>>>>>>>>>>>> We have a cloud server with 4gb of memory and after a while
>>>>>>>>>>>>>> system is
>>>>>>>>>>>>>> killing Fuseki server java process for using up all memory.
>>>>>>>>>>>>>> Is it
>>>>>>>>>>>>>> possible to limit memory usage, and what would be the
>>>>>>>>>>>>>> performance
>>>>>>>>>>>>>> penalty for doing that?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Lingsoft - 30 years of Leading Language Management
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
www.lingsoft.fi[http://www.lingsoft.fi][http://www.lingsoft.fi[http://www.lingsoft.fi]][http://www.lingsoft.fi[http://www.lingsoft.fi][http://www.lingsoft.fi[http://www.lingsoft.fi]]]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Speech Applications - Language Management - Translation -
>>>>>>>>>>>>>> Reader's and Writer's Tools - Text Tools - E-books and
>>>>>>>>>>>>>> M-books
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Mikael Pesonen
>>>>>>>>>>>>>> System Engineer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> e-mail: mikael.peso...@lingsoft.fi
>>>>>>>>>>>>>> Tel. +358 2 279 3300
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Time zone: GMT+2
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Helsinki Office
>>>>>>>>>>>>>> Eteläranta 10
>>>>>>>>>>>>>> FI-00130 Helsinki
>>>>>>>>>>>>>> FINLAND
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Turku Office
>>>>>>>>>>>>>> Kauppiaskatu 5 A
>>>>>>>>>>>>>> FI-20100 Turku
>>>>>>>>>>>>>> FINLAND
>>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Lingsoft - 30 years of Leading Language Management
>>>>>>>>>>>>
>>>>>>>>>>>>
www.lingsoft.fi[http://www.lingsoft.fi][http://www.lingsoft.fi[http://www.lingsoft.fi]]
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Speech Applications - Language Management - Translation -
>>>>>>>>>>>> Reader's and Writer's Tools - Text Tools - E-books and M-books
>>>>>>>>>>>>
>>>>>>>>>>>> Mikael Pesonen
>>>>>>>>>>>> System Engineer
>>>>>>>>>>>>
>>>>>>>>>>>> e-mail: mikael.peso...@lingsoft.fi
>>>>>>>>>>>> Tel. +358 2 279 3300
>>>>>>>>>>>>
>>>>>>>>>>>> Time zone: GMT+2
>>>>>>>>>>>>
>>>>>>>>>>>> Helsinki Office
>>>>>>>>>>>> Eteläranta 10
>>>>>>>>>>>> FI-00130 Helsinki
>>>>>>>>>>>> FINLAND
>>>>>>>>>>>>
>>>>>>>>>>>> Turku Office
>>>>>>>>>>>> Kauppiaskatu 5 A
>>>>>>>>>>>> FI-20100 Turku
>>>>>>>>>>>> FINLAND
>>>
>>> --
>>> Lingsoft - 30 years of Leading Language Management
>>>
>>> www.lingsoft.fi[http://www.lingsoft.fi]
>>>
>>> Speech Applications - Language Management - Translation - Reader's
>>> and Writer's Tools - Text Tools - E-books and M-books
>>>
>>> Mikael Pesonen
>>> System Engineer
>>>
>>> e-mail: mikael.peso...@lingsoft.fi
>>> Tel. +358 2 279 3300
>>>
>>> Time zone: GMT+2
>>>
>>> Helsinki Office
>>> Eteläranta 10
>>> FI-00130 Helsinki
>>> FINLAND
>>>
>>> Turku Office
>>> Kauppiaskatu 5 A
>>> FI-20100 Turku
>>> FINLAND
>>>
>
--
Lingsoft - 30 years of Leading Language Management
www.lingsoft.fi
Speech Applications - Language Management - Translation - Reader's and Writer's Tools - Text Tools - E-books and M-books
Mikael Pesonen
System Engineer
e-mail: mikael.peso...@lingsoft.fi
Tel. +358 2 279 3300
Time zone: GMT+2
Helsinki Office
Eteläranta 10
FI-00130 Helsinki
FINLAND
Turku Office
Kauppiaskatu 5 A
FI-20100 Turku
FINLAND
