[Chicken-users] Best way to share memory between C and Chicken

2013-05-03 Thread Pedro Melendez
Hi all, Sorry if this question is obvious, but I couldn't find what I were looking for in the documentation so maybe you guys can help me. I am developing a prototype of a server that would serve 3D seismic images across the network. This task requires to process big files (~4 GB) with existing

Re: [Chicken-users] Best way to share memory between C and Chicken

2013-05-03 Thread Dan Leslie
If it's heterogeneous data I tend to use blobs with helper functions to cast segments to records without undertaking a copy. https://wiki.call-cc.org/man/4/Unit%20library#blobs If it's homogenous data then the srfi-4 unit provides all sorts of vectors that you can use.

Re: [Chicken-users] Best way to share memory between C and Chicken

2013-05-03 Thread Thomas Chust
On 2013-05-03 20:04, Pedro Melendez wrote: [...] I am developing a prototype of a server that would serve 3D seismic images across the network. [...] Hello Pedro, it's nice to see I'm not the only geophysicist who likes to use Scheme :-) [...] Giving the size of the file, I want to share

Re: [Chicken-users] Best way to share memory between C and Chicken

2013-05-03 Thread Ivan Raikov
Hello, I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. If your C code is to be linked with your Chicken code, you can pass the pointer to your data from C to Scheme and use

Re: [Chicken-users] Best way to share memory between C and Chicken

2013-05-03 Thread Thomas Chust
On 2013-05-04 00:26, Ivan Raikov wrote: [...] I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. [...] Hello, would that be related to the fact that CHICKEN has a copying garbage

Re: [Chicken-users] Best way to share memory between C and Chicken

2013-05-03 Thread Dan Leslie
I was just poking through posix, posix-shm, posix-utils, and posix-extras and it seems that none of them implement semaphores! Am I missing something, or is this actually the case? -Dan On 5/3/2013 3:26 PM, Ivan Raikov wrote: Hello, I really strongly advise _against_ using SRFI-4 vectors

Re: [Chicken-users] Best way to share memory between C and Chicken

2013-05-03 Thread Ivan Raikov
Hello, Yes, it seems that copying/moving around large vectors puts a lot of pressure on the memory subsystem. Other than that, SRFI-4 vectors are fine. Ivan On May 4, 2013 7:57 AM, Thomas Chust ch...@web.de wrote: On 2013-05-04 00:26, Ivan Raikov wrote: [...] I really strongly advise

Re: [Chicken-users] Best way to share memory between C and Chicken

2013-05-03 Thread Ivan Raikov
Are you talking about POSIX semaphores, sem_wait(3) and friends, or just the general semaphor data structure? If the former, then the Chicken developers are eagerly awaiting your patches ;-) If the latter, take a look at the synch and mailbox eggs. They have mutex-like functionality that can be

Re: [Chicken-users] Best way to share memory between C and Chicken

2013-05-03 Thread Dan Leslie
Yah, I meant sem_wait et al. I was under the impression that synch and mailbox rely on srfi-18 structures, which would make them 'green' threads-only and not particularly suitable for multi process synchronization. Relatedly, is anyone poking at implementing native threads? I've been digging

Re: [Chicken-users] Best way to share memory between C and Chicken

2013-05-03 Thread Ivan Raikov
I think you can try to have native threads by running a separate instance of the Chicken runtime for each thread, but are you sure that you will really achieve a significant speedup over Unix processes and/or MPI? It's not the early nineties anymore... I suggest prototyping code and running