#323: Deadlock when decoding ogg files.
-------------------------+--------------------------------------------------
Reporter: toots | Owner: admin
Type: Bugs | Status: new
Priority: 1 | Milestone:
Component: Liquidsoap | Version: 0.9.1+svn
Resolution: | Keywords:
Mac: 1 | Linux: 1
Netbsd: 1 | Other: 1
Freebsd: 1 |
-------------------------+--------------------------------------------------
Description changed by toots:
Old description:
> I have a reproducible deadlock when deconding files with {{{root.sync}}}
> set to false.
>
> Here is the script:
> {{{
> # Log to stdout
> set("log.file",false)
> set("log.stdout",true)
> set("log.level",5)
> set("root.sync",false)
>
> s = single(conservative=true,"annotate:bli=\"blo\":/tmp/test.ogg")
>
> output.dummy(mksafe(s))
> }}}
>
> This seems to happen with any ogg file.
>
> Here's the latest log on my last test:
> {{{
> 2009/10/26 21:23:54 [ogg.demuxer:4] Found a ogg logical stream, serial:
> 2c0722a5
> 2009/10/26 21:23:54 [ogg.demuxer:4] Trying ogg/vorbis format
> 2009/10/26 21:23:54 [decoder:3] Decoder OGG chosen for "/tmp/test.ogg".
> 2009/10/26 21:23:54 [ogg.demuxer:4] Found a ogg logical stream, serial:
> 2c0722a5
> 2009/10/26 21:23:54 [ogg.demuxer:4] Trying ogg/vorbis format
> 2009/10/26 21:23:54 [src_3710:4] Remaining: 457603, queued: 0, adding:
> 1323000 (RID 22)
> 2009/10/26 21:23:57 [ogg.demuxer:4] Reached last page of logical stream
> 2c0722a5
> 2009/10/26 21:23:57 [src_3710:3] Finished with "/tmp/test.ogg".
> 2009/10/26 21:23:57 [src_3710:4] Remaining: 0, queued: 1323000, taking:
> 1323000
> 2009/10/26 21:23:57 [ogg.demuxer:4] Found a ogg logical stream, serial:
> 2c0722a5
> 2009/10/26 21:23:57 [ogg.demuxer:4] Trying ogg/vorbis format
> 2009/10/26 21:23:57 [src_3710:3] Prepared "/tmp/test.ogg" (RID 22).
> 2009/10/26 21:23:57 [decoder:4] Trying OGG decoder for "/tmp/test.ogg"...
> 2009/10/26 21:23:57 [ogg.demuxer:4] Found a ogg logical stream, serial:
> 2c0722a5
> 2009/10/26 21:23:57 [ogg.demuxer:4] Trying ogg/vorbis format
> 2009/10/26 21:23:57 [decoder:3] Decoder OGG chosen for "/tmp/test.ogg".
> 2009/10/26 21:23:57 [ogg.demuxer:4] Found a ogg logical stream, serial:
> 2c0722a5
> 2009/10/26 21:23:57 [ogg.demuxer:4] Trying ogg/vorbis format
> 2009/10/26 21:23:57 [src_3710:4] Remaining: 456781, queued: 0, adding:
> 1323000 (RID 23)
> 2009/10/26 21:24:00 [ogg.demuxer:4] Reached last page of logical stream
> 2c0722a5
> 2009/10/26 21:24:00 [src_3710:3] Finished with "/tmp/test.ogg".
> 2009/10/26 21:24:00 [src_3710:4] Remaining: 0, queued: 1323000, taking:
> 1323000
> 2009/10/26 21:24:00 [ogg.demuxer:4] Found a ogg logical stream, serial:
> 2c0722a5
> 2009/10/26 21:24:00 [ogg.demuxer:4] Trying ogg/vorbis format
> 2009/10/26 21:24:00 [src_3710:3] Prepared "/tmp/test.ogg" (RID 23).
> 2009/10/26 21:24:00 [decoder:4] Trying OGG decoder for "/tmp/test.ogg"...
> 2009/10/26 21:24:00 [ogg.demuxer:4] Found a ogg logical stream, serial:
> 2c0722a5
> 2009/10/26 21:24:00 [ogg.demuxer:4] Trying ogg/vorbis format
> 2009/10/26 21:24:00 [decoder:3] Decoder OGG chosen for "/tmp/test.ogg".
> 2009/10/26 21:24:00 [ogg.demuxer:4] Found a ogg logical stream, serial:
> 2c0722a5
> 2009/10/26 21:24:00 [ogg.demuxer:4] Trying ogg/vorbis format
> 2009/10/26 21:24:00 [src_3710:4] Remaining: 455960, queued: 0, adding:
> 1323000 (RID 24)
> 2009/10/26 21:24:03 [ogg.demuxer:4] Reached last page of logical stream
> 2c0722a5
> }}}
>
> It always deadlock with the same pattern: a Remaining and then little
> time after the "Reached last page" and then it is gone.
>
> Attaching a gdb to the running process, I get:
> {{{
> (gdb) thread apply all bt
>
> Thread 6 (Thread 0x7f85d1803910 (LWP 4635)):
> #0 0x00007f85d1fcbd82 in select () from /lib/libc.so.6
> #1 0x00000000005aab30 in unix_select ()
> #2 0x00000000005c3670 in caml_c_call ()
> #3 0x0000000000000007 in ?? ()
> #4 0x0000000000000001 in ?? ()
> #5 0x0000000000000001 in ?? ()
> #6 0x00007f85d3932128 in ?? ()
> #7 0x000000000089c360 in camlDuppy__25 ()
> #8 0x00000000008931a0 in camlTutils__41 ()
> #9 0x00007f85d3932490 in ?? ()
> #10 0x00000000005a809c in caml_mutex_unlock ()
> #11 0x00000000005135d6 in camlDuppy__queue_199 ()
> #12 0x0000000002db5e90 in ?? ()
> #13 0x00000000022b1838 in ?? ()
> #14 0x0000000000893170 in camlTutils__38 ()
> #15 0x00000000008931a0 in camlTutils__41 ()
> #16 0x0000000002494a38 in ?? ()
> #17 0x0000000000506cf2 in camlTutils__fun_358 ()
> #18 0x00007f85d1802f90 in ?? ()
> #19 0x0000000000506b0e in camlTutils__fun_358 ()
> #20 0x0000000000000001 in ?? ()
> #21 0x00007f85d24601f0 in ?? () from /lib/libpthread.so.0
> #22 0x00000000022b1668 in ?? ()
> #23 0x00000000005662ee in camlThread__fun_127 ()
> #24 0x00007f85d1802fb0 in ?? ()
> #25 0x000000000056628c in camlThread__fun_127 ()
> #26 0x0000000000000000 in ?? ()
>
> Thread 5 (Thread 0x7f85d1002910 (LWP 4636)):
> #0 0x00007f85d1fcbd82 in select () from /lib/libc.so.6
> #1 0x00000000005a8846 in caml_thread_tick ()
> #2 0x00007f85d246473a in start_thread () from /lib/libpthread.so.0
> #3 0x00007f85d1fd249d in clone () from /lib/libc.so.6
> #4 0x0000000000000000 in ?? ()
>
> Thread 4 (Thread 0x7f85d0801910 (LWP 4637)):
> #0 0x00007f85d2468f99 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1 0x00000000005a7f64 in caml_condition_wait ()
> #2 0x00000000005c3670 in caml_c_call ()
> #3 0x00007f85d3930fb0 in ?? ()
> #4 0x00000000022b1808 in ?? ()
> #5 0x000000000089c0f8 in camlDuppy__12 ()
> #6 0x00000000008931a0 in camlTutils__41 ()
> #7 0x0000000002494a38 in ?? ()
> #8 0x0000000000506cf2 in camlTutils__fun_358 ()
> #9 0x00007f85d0800f90 in ?? ()
> #10 0x0000000000506b0e in camlTutils__fun_358 ()
> #11 0x0000000000000000 in ?? ()
>
> Thread 3 (Thread 0x7f85d0000910 (LWP 4638)):
> #0 0x00007f85d2468f99 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1 0x00000000005a7f64 in caml_condition_wait ()
> #2 0x00000000005c3670 in caml_c_call ()
> #3 0x0000000000000003 in ?? ()
> #4 0x0000000000893120 in camlTutils__15 ()
> #5 0x0000000002498aa8 in ?? ()
> #6 0x00000000022b17c0 in ?? ()
> #7 0x00007f85d18b1328 in ?? ()
> #8 0x00000000005158cd in camlDtools__thread_348 ()
> #9 0x00007f85cfffff90 in ?? ()
> #10 0x0000000000515833 in camlDtools__thread_348 ()
> #11 0x0000000000000000 in ?? ()
>
> Thread 2 (Thread 0x7f85cf7ff910 (LWP 4639)):
> #0 0x00000000005b68c0 in caml_page_table_lookup ()
> #1 0x00000000005b5894 in mark_slice ()
> #2 0x00000000005b60fc in caml_major_collection_slice ()
> #3 0x00000000005b65b5 in caml_minor_collection ()
> #4 0x00000000005b6618 in caml_check_urgent_gc ()
> #5 0x00000000005b74c7 in caml_alloc_string ()
> #6 0x00000000005b9a68 in caml_create_string ()
> #7 0x00000000005c3670 in caml_c_call ()
> #8 0x00000000024df508 in ?? ()
> #9 0x0000000000002001 in ?? ()
> #10 0x0000000000000001 in ?? ()
> #11 0x00000000005c36c6 in caml_start_program ()
> #12 0x00007f85cf7fece0 in ?? ()
> #13 0x00000000005c36fd in caml_start_program ()
> #14 0x00007f85cf7fecd0 in ?? ()
> #15 0x0000000000510b4a in camlOgg__read_129 ()
> #16 0x00007f85cf7fecb0 in ?? ()
> #17 0x0000000002251600 in ?? ()
> #18 0x00007f85cf7fec60 in ?? ()
> #19 0x00007f85d3940c00 in ?? ()
> #20 0x00007f85cf7fece0 in ?? ()
> #21 0x0000000000000000 in ?? ()
>
> Thread 1 (Thread 0x7f85d39456f0 (LWP 4628)):
> #0 0x00007f85d246c6d7 in do_sigwait () from /lib/libpthread.so.0
> #1 0x00007f85d246c767 in sigwait () from /lib/libpthread.so.0
> #2 0x00000000005a7dc9 in caml_wait_signal ()
> #3 0x00000000005c3670 in caml_c_call ()
> #4 0x00007fff2f95b340 in ?? ()
> #5 0x00000000005180c9 in camlDtools__wait_signal_340 ()
> #6 0x00007fff2f95b340 in ?? ()
> #7 0x0000000000518287 in camlDtools__main_343 ()
> #8 0x00007f85d18b1328 in ?? ()
> #9 0x00007f85d3933a08 in ?? ()
> #10 0x0000000002486e30 in ?? ()
> #11 0x00000000005183ec in camlDtools__catch_353 ()
> #12 0x00007fff2f95b3a0 in ?? ()
> #13 0x0000000000518315 in camlDtools__catch_353 ()
> #14 0x0000000002486e00 in ?? ()
> #15 0x00007f85d390f488 in ?? ()
> #16 0x000000000089db48 in camlDtools__59 ()
> #17 0x000000000045ab34 in camlMain__entry ()
> #18 0x00007f85d390f488 in ?? ()
> #19 0x00007f85d390f468 in ?? ()
> #20 0x00007f85d3909060 in ?? ()
> #21 0x0000000000453549 in caml_program ()
> #22 0x00000000003fadd4 in ?? ()
> #23 0x00000000005c36c6 in caml_start_program ()
> #24 0x0000000000000000 in ?? ()
> }}}
>
> The relevant part, related to the root thread, which is the one
> deadlocking, seems to be:
> {{{
> Thread 2 (Thread 0x7f85cf7ff910 (LWP 4639)):
> #0 0x00000000005b68c0 in caml_page_table_lookup ()
> #1 0x00000000005b5894 in mark_slice ()
> #2 0x00000000005b60fc in caml_major_collection_slice ()
> #3 0x00000000005b65b5 in caml_minor_collection ()
> #4 0x00000000005b6618 in caml_check_urgent_gc ()
> #5 0x00000000005b74c7 in caml_alloc_string ()
> #6 0x00000000005b9a68 in caml_create_string ()
> #7 0x00000000005c3670 in caml_c_call ()
> }}}
>
> The string creation is surely the {{{String.create}}} call in Ogg.Sync. I
> have also seen a backtrace where it showed the {{{Unix.read}}} which is
> coming right after it.
>
> As it seems for now, this deadlock seems to be caused by some bad
> interactions in a binding with the {{{Gc}}}. It surely needs more time to
> investigate...
New description:
I have a reproducible deadlock when decoding ogg files with
{{{root.sync}}} set to false.
Here is the script:
{{{
# Log to stdout
set("log.file",false)
set("log.stdout",true)
set("log.level",5)
set("root.sync",false)
s = single(conservative=true,"annotate:bli=\"blo\":/tmp/test.ogg")
output.dummy(mksafe(s))
}}}
This seems to happen with any ogg file.
Here's the latest log on my last test:
{{{
2009/10/26 21:23:54 [ogg.demuxer:4] Found a ogg logical stream, serial:
2c0722a5
2009/10/26 21:23:54 [ogg.demuxer:4] Trying ogg/vorbis format
2009/10/26 21:23:54 [decoder:3] Decoder OGG chosen for "/tmp/test.ogg".
2009/10/26 21:23:54 [ogg.demuxer:4] Found a ogg logical stream, serial:
2c0722a5
2009/10/26 21:23:54 [ogg.demuxer:4] Trying ogg/vorbis format
2009/10/26 21:23:54 [src_3710:4] Remaining: 457603, queued: 0, adding:
1323000 (RID 22)
2009/10/26 21:23:57 [ogg.demuxer:4] Reached last page of logical stream
2c0722a5
2009/10/26 21:23:57 [src_3710:3] Finished with "/tmp/test.ogg".
2009/10/26 21:23:57 [src_3710:4] Remaining: 0, queued: 1323000, taking:
1323000
2009/10/26 21:23:57 [ogg.demuxer:4] Found a ogg logical stream, serial:
2c0722a5
2009/10/26 21:23:57 [ogg.demuxer:4] Trying ogg/vorbis format
2009/10/26 21:23:57 [src_3710:3] Prepared "/tmp/test.ogg" (RID 22).
2009/10/26 21:23:57 [decoder:4] Trying OGG decoder for "/tmp/test.ogg"...
2009/10/26 21:23:57 [ogg.demuxer:4] Found a ogg logical stream, serial:
2c0722a5
2009/10/26 21:23:57 [ogg.demuxer:4] Trying ogg/vorbis format
2009/10/26 21:23:57 [decoder:3] Decoder OGG chosen for "/tmp/test.ogg".
2009/10/26 21:23:57 [ogg.demuxer:4] Found a ogg logical stream, serial:
2c0722a5
2009/10/26 21:23:57 [ogg.demuxer:4] Trying ogg/vorbis format
2009/10/26 21:23:57 [src_3710:4] Remaining: 456781, queued: 0, adding:
1323000 (RID 23)
2009/10/26 21:24:00 [ogg.demuxer:4] Reached last page of logical stream
2c0722a5
2009/10/26 21:24:00 [src_3710:3] Finished with "/tmp/test.ogg".
2009/10/26 21:24:00 [src_3710:4] Remaining: 0, queued: 1323000, taking:
1323000
2009/10/26 21:24:00 [ogg.demuxer:4] Found a ogg logical stream, serial:
2c0722a5
2009/10/26 21:24:00 [ogg.demuxer:4] Trying ogg/vorbis format
2009/10/26 21:24:00 [src_3710:3] Prepared "/tmp/test.ogg" (RID 23).
2009/10/26 21:24:00 [decoder:4] Trying OGG decoder for "/tmp/test.ogg"...
2009/10/26 21:24:00 [ogg.demuxer:4] Found a ogg logical stream, serial:
2c0722a5
2009/10/26 21:24:00 [ogg.demuxer:4] Trying ogg/vorbis format
2009/10/26 21:24:00 [decoder:3] Decoder OGG chosen for "/tmp/test.ogg".
2009/10/26 21:24:00 [ogg.demuxer:4] Found a ogg logical stream, serial:
2c0722a5
2009/10/26 21:24:00 [ogg.demuxer:4] Trying ogg/vorbis format
2009/10/26 21:24:00 [src_3710:4] Remaining: 455960, queued: 0, adding:
1323000 (RID 24)
2009/10/26 21:24:03 [ogg.demuxer:4] Reached last page of logical stream
2c0722a5
}}}
It always deadlock with the same pattern: a Remaining and then little time
after the "Reached last page" and then it is gone.
Attaching a gdb to the running process, I get:
{{{
(gdb) thread apply all bt
Thread 6 (Thread 0x7f85d1803910 (LWP 4635)):
#0 0x00007f85d1fcbd82 in select () from /lib/libc.so.6
#1 0x00000000005aab30 in unix_select ()
#2 0x00000000005c3670 in caml_c_call ()
#3 0x0000000000000007 in ?? ()
#4 0x0000000000000001 in ?? ()
#5 0x0000000000000001 in ?? ()
#6 0x00007f85d3932128 in ?? ()
#7 0x000000000089c360 in camlDuppy__25 ()
#8 0x00000000008931a0 in camlTutils__41 ()
#9 0x00007f85d3932490 in ?? ()
#10 0x00000000005a809c in caml_mutex_unlock ()
#11 0x00000000005135d6 in camlDuppy__queue_199 ()
#12 0x0000000002db5e90 in ?? ()
#13 0x00000000022b1838 in ?? ()
#14 0x0000000000893170 in camlTutils__38 ()
#15 0x00000000008931a0 in camlTutils__41 ()
#16 0x0000000002494a38 in ?? ()
#17 0x0000000000506cf2 in camlTutils__fun_358 ()
#18 0x00007f85d1802f90 in ?? ()
#19 0x0000000000506b0e in camlTutils__fun_358 ()
#20 0x0000000000000001 in ?? ()
#21 0x00007f85d24601f0 in ?? () from /lib/libpthread.so.0
#22 0x00000000022b1668 in ?? ()
#23 0x00000000005662ee in camlThread__fun_127 ()
#24 0x00007f85d1802fb0 in ?? ()
#25 0x000000000056628c in camlThread__fun_127 ()
#26 0x0000000000000000 in ?? ()
Thread 5 (Thread 0x7f85d1002910 (LWP 4636)):
#0 0x00007f85d1fcbd82 in select () from /lib/libc.so.6
#1 0x00000000005a8846 in caml_thread_tick ()
#2 0x00007f85d246473a in start_thread () from /lib/libpthread.so.0
#3 0x00007f85d1fd249d in clone () from /lib/libc.so.6
#4 0x0000000000000000 in ?? ()
Thread 4 (Thread 0x7f85d0801910 (LWP 4637)):
#0 0x00007f85d2468f99 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005a7f64 in caml_condition_wait ()
#2 0x00000000005c3670 in caml_c_call ()
#3 0x00007f85d3930fb0 in ?? ()
#4 0x00000000022b1808 in ?? ()
#5 0x000000000089c0f8 in camlDuppy__12 ()
#6 0x00000000008931a0 in camlTutils__41 ()
#7 0x0000000002494a38 in ?? ()
#8 0x0000000000506cf2 in camlTutils__fun_358 ()
#9 0x00007f85d0800f90 in ?? ()
#10 0x0000000000506b0e in camlTutils__fun_358 ()
#11 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7f85d0000910 (LWP 4638)):
#0 0x00007f85d2468f99 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005a7f64 in caml_condition_wait ()
#2 0x00000000005c3670 in caml_c_call ()
#3 0x0000000000000003 in ?? ()
#4 0x0000000000893120 in camlTutils__15 ()
#5 0x0000000002498aa8 in ?? ()
#6 0x00000000022b17c0 in ?? ()
#7 0x00007f85d18b1328 in ?? ()
#8 0x00000000005158cd in camlDtools__thread_348 ()
#9 0x00007f85cfffff90 in ?? ()
#10 0x0000000000515833 in camlDtools__thread_348 ()
#11 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7f85cf7ff910 (LWP 4639)):
#0 0x00000000005b68c0 in caml_page_table_lookup ()
#1 0x00000000005b5894 in mark_slice ()
#2 0x00000000005b60fc in caml_major_collection_slice ()
#3 0x00000000005b65b5 in caml_minor_collection ()
#4 0x00000000005b6618 in caml_check_urgent_gc ()
#5 0x00000000005b74c7 in caml_alloc_string ()
#6 0x00000000005b9a68 in caml_create_string ()
#7 0x00000000005c3670 in caml_c_call ()
#8 0x00000000024df508 in ?? ()
#9 0x0000000000002001 in ?? ()
#10 0x0000000000000001 in ?? ()
#11 0x00000000005c36c6 in caml_start_program ()
#12 0x00007f85cf7fece0 in ?? ()
#13 0x00000000005c36fd in caml_start_program ()
#14 0x00007f85cf7fecd0 in ?? ()
#15 0x0000000000510b4a in camlOgg__read_129 ()
#16 0x00007f85cf7fecb0 in ?? ()
#17 0x0000000002251600 in ?? ()
#18 0x00007f85cf7fec60 in ?? ()
#19 0x00007f85d3940c00 in ?? ()
#20 0x00007f85cf7fece0 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f85d39456f0 (LWP 4628)):
#0 0x00007f85d246c6d7 in do_sigwait () from /lib/libpthread.so.0
#1 0x00007f85d246c767 in sigwait () from /lib/libpthread.so.0
#2 0x00000000005a7dc9 in caml_wait_signal ()
#3 0x00000000005c3670 in caml_c_call ()
#4 0x00007fff2f95b340 in ?? ()
#5 0x00000000005180c9 in camlDtools__wait_signal_340 ()
#6 0x00007fff2f95b340 in ?? ()
#7 0x0000000000518287 in camlDtools__main_343 ()
#8 0x00007f85d18b1328 in ?? ()
#9 0x00007f85d3933a08 in ?? ()
#10 0x0000000002486e30 in ?? ()
#11 0x00000000005183ec in camlDtools__catch_353 ()
#12 0x00007fff2f95b3a0 in ?? ()
#13 0x0000000000518315 in camlDtools__catch_353 ()
#14 0x0000000002486e00 in ?? ()
#15 0x00007f85d390f488 in ?? ()
#16 0x000000000089db48 in camlDtools__59 ()
#17 0x000000000045ab34 in camlMain__entry ()
#18 0x00007f85d390f488 in ?? ()
#19 0x00007f85d390f468 in ?? ()
#20 0x00007f85d3909060 in ?? ()
#21 0x0000000000453549 in caml_program ()
#22 0x00000000003fadd4 in ?? ()
#23 0x00000000005c36c6 in caml_start_program ()
#24 0x0000000000000000 in ?? ()
}}}
The relevant part, related to the root thread, which is the one
deadlocking, seems to be:
{{{
Thread 2 (Thread 0x7f85cf7ff910 (LWP 4639)):
#0 0x00000000005b68c0 in caml_page_table_lookup ()
#1 0x00000000005b5894 in mark_slice ()
#2 0x00000000005b60fc in caml_major_collection_slice ()
#3 0x00000000005b65b5 in caml_minor_collection ()
#4 0x00000000005b6618 in caml_check_urgent_gc ()
#5 0x00000000005b74c7 in caml_alloc_string ()
#6 0x00000000005b9a68 in caml_create_string ()
#7 0x00000000005c3670 in caml_c_call ()
}}}
The string creation is surely the {{{String.create}}} call in Ogg.Sync. I
have also seen a backtrace where it showed the {{{Unix.read}}} which is
coming right after it.
As it seems for now, this deadlock seems to be caused by some bad
interactions in a binding with the {{{Gc}}}. It surely needs more time to
investigate...
--
--
Ticket URL: <http://savonet.rastageeks.org/ticket/323#comment:1>
Savonet <http://savonet.rastageeks.org/>
Let's program our stream !
------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Savonet-trac mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/savonet-trac
