[Chicken-users] Best way to share memory between C and Chicken
Hi all, Sorry if this question is obvious, but I couldn't find what I were looking for in the documentation so maybe you guys can help me. I am developing a prototype of a server that would serve 3D seismic images across the network. This task requires to process big files (~4 GB) with existing C code that is desirable to maintain. I plan to write the server itself in Chicken scheme but I would need to maintain the existing code in C that opens and process those files. Giving the size of the file, I want to share the memory space between C and Chicken and avoid copying values between areas. Is that even possible? Anyone has an idea on how can I address this? Thanks in advance! Pedro -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
If it's heterogeneous data I tend to use blobs with helper functions to cast segments to records without undertaking a copy. https://wiki.call-cc.org/man/4/Unit%20library#blobs If it's homogenous data then the srfi-4 unit provides all sorts of vectors that you can use. https://wiki.call-cc.org/man/4/Unit%20srfi-4 Keep in mind that foreign-lambdas will take blob and srfi-4 vectors as parameter types and convert them to their relevant C types without undertaking a copy. https://wiki.call-cc.org/man/4/Accessing%20external%20objects -Dan On 5/3/2013 11:04 AM, Pedro Melendez wrote: Hi all, Sorry if this question is obvious, but I couldn't find what I were looking for in the documentation so maybe you guys can help me. I am developing a prototype of a server that would serve 3D seismic images across the network. This task requires to process big files (~4 GB) with existing C code that is desirable to maintain. I plan to write the server itself in Chicken scheme but I would need to maintain the existing code in C that opens and process those files. Giving the size of the file, I want to share the memory space between C and Chicken and avoid copying values between areas. Is that even possible? Anyone has an idea on how can I address this? Thanks in advance! Pedro -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
On 2013-05-03 20:04, Pedro Melendez wrote: [...] I am developing a prototype of a server that would serve 3D seismic images across the network. [...] Hello Pedro, it's nice to see I'm not the only geophysicist who likes to use Scheme :-) [...] Giving the size of the file, I want to share the memory space between C and Chicken and avoid copying values between areas. Is that even possible? [...] Both CHICKEN's blobs and SRFI-4 homogeneous vectors can be converted to native pointers to their data contents through the FFI. As a silly example, this will allocate a big array of floating point numbers, set one of the array elements using native code and read it bacl from Scheme: (use srfi-4) (define big-array (make-f64vector (expt 2 20))) (define set-some-element! (foreign-lambda* void ((nonnull-f64vector v)) v[1024] = 42.23;)) (set-some-element! big-array) (display (f64vector-ref big-array 1024)) (newline) If your data is not a homogeneous numeric array, you could use blobs instead. If you absolutely have to allocate unmanaged memory on the C side an pass pointers there back to Scheme rather than creating managed Scheme objects and passing pointers to their contents to the C code, take a look at CHICKEN's lolevel unit. CHICKEN Scheme can also deal with raw native pointers to a certain degree! Ciao, Thomas -- When C++ is your hammer, every problem looks like your thumb. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
Hello, I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. If your C code is to be linked with your Chicken code, you can pass the pointer to your data from C to Scheme and use SRFI-4 foreign pointers to access it (see unit lolevel for details). If the C code is running as a separate process, you could try using posix-shm to create shared memory between processes and then use foreign pointers in the Chicken process. Ivan On May 4, 2013 3:04 AM, Pedro Melendez pmelen...@pevicom.com wrote: Hi all, Sorry if this question is obvious, but I couldn't find what I were looking for in the documentation so maybe you guys can help me. I am developing a prototype of a server that would serve 3D seismic images across the network. This task requires to process big files (~4 GB) with existing C code that is desirable to maintain. I plan to write the server itself in Chicken scheme but I would need to maintain the existing code in C that opens and process those files. Giving the size of the file, I want to share the memory space between C and Chicken and avoid copying values between areas. Is that even possible? Anyone has an idea on how can I address this? Thanks in advance! Pedro -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
On 2013-05-04 00:26, Ivan Raikov wrote: [...] I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. [...] Hello, would that be related to the fact that CHICKEN has a copying garbage collector or are there other hidden performance implications of SRFI-4 vectors? Ciao, Thomas -- When C++ is your hammer, every problem looks like your thumb. -- When C++ is your hammer, every problem looks like your thumb. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
I was just poking through posix, posix-shm, posix-utils, and posix-extras and it seems that none of them implement semaphores! Am I missing something, or is this actually the case? -Dan On 5/3/2013 3:26 PM, Ivan Raikov wrote: Hello, I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. If your C code is to be linked with your Chicken code, you can pass the pointer to your data from C to Scheme and use SRFI-4 foreign pointers to access it (see unit lolevel for details). If the C code is running as a separate process, you could try using posix-shm to create shared memory between processes and then use foreign pointers in the Chicken process. Ivan On May 4, 2013 3:04 AM, Pedro Melendez pmelen...@pevicom.com mailto:pmelen...@pevicom.com wrote: Hi all, Sorry if this question is obvious, but I couldn't find what I were looking for in the documentation so maybe you guys can help me. I am developing a prototype of a server that would serve 3D seismic images across the network. This task requires to process big files (~4 GB) with existing C code that is desirable to maintain. I plan to write the server itself in Chicken scheme but I would need to maintain the existing code in C that opens and process those files. Giving the size of the file, I want to share the memory space between C and Chicken and avoid copying values between areas. Is that even possible? Anyone has an idea on how can I address this? Thanks in advance! Pedro -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org mailto:Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
Hello, Yes, it seems that copying/moving around large vectors puts a lot of pressure on the memory subsystem. Other than that, SRFI-4 vectors are fine. Ivan On May 4, 2013 7:57 AM, Thomas Chust ch...@web.de wrote: On 2013-05-04 00:26, Ivan Raikov wrote: [...] I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. [...] Hello, would that be related to the fact that CHICKEN has a copying garbage collector or are there other hidden performance implications of SRFI-4 vectors? Ciao, Thomas -- When C++ is your hammer, every problem looks like your thumb. -- When C++ is your hammer, every problem looks like your thumb. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
Are you talking about POSIX semaphores, sem_wait(3) and friends, or just the general semaphor data structure? If the former, then the Chicken developers are eagerly awaiting your patches ;-) If the latter, take a look at the synch and mailbox eggs. They have mutex-like functionality that can be used in place of proper semaphores. Ivan On May 4, 2013 7:59 AM, Dan Leslie d...@ironoxide.ca wrote: I was just poking through posix, posix-shm, posix-utils, and posix-extras and it seems that none of them implement semaphores! Am I missing something, or is this actually the case? -Dan On 5/3/2013 3:26 PM, Ivan Raikov wrote: Hello, I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. If your C code is to be linked with your Chicken code, you can pass the pointer to your data from C to Scheme and use SRFI-4 foreign pointers to access it (see unit lolevel for details). If the C code is running as a separate process, you could try using posix-shm to create shared memory between processes and then use foreign pointers in the Chicken process. Ivan On May 4, 2013 3:04 AM, Pedro Melendez pmelen...@pevicom.com wrote: Hi all, Sorry if this question is obvious, but I couldn't find what I were looking for in the documentation so maybe you guys can help me. I am developing a prototype of a server that would serve 3D seismic images across the network. This task requires to process big files (~4 GB) with existing C code that is desirable to maintain. I plan to write the server itself in Chicken scheme but I would need to maintain the existing code in C that opens and process those files. Giving the size of the file, I want to share the memory space between C and Chicken and avoid copying values between areas. Is that even possible? Anyone has an idea on how can I address this? Thanks in advance! Pedro -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing listChicken-users@nongnu.orghttps://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
Yah, I meant sem_wait et al. I was under the impression that synch and mailbox rely on srfi-18 structures, which would make them 'green' threads-only and not particularly suitable for multi process synchronization. Relatedly, is anyone poking at implementing native threads? I've been digging around a bit but haven't had much time to progress very far. I'm hesitant to take responsibility for writing a semaphore egg, but what the hell. I'll start something on GitHub this weekend. -Dan On 5/3/2013 4:22 PM, Ivan Raikov wrote: Are you talking about POSIX semaphores, sem_wait(3) and friends, or just the general semaphor data structure? If the former, then the Chicken developers are eagerly awaiting your patches ;-) If the latter, take a look at the synch and mailbox eggs. They have mutex-like functionality that can be used in place of proper semaphores. Ivan On May 4, 2013 7:59 AM, Dan Leslie d...@ironoxide.ca mailto:d...@ironoxide.ca wrote: I was just poking through posix, posix-shm, posix-utils, and posix-extras and it seems that none of them implement semaphores! Am I missing something, or is this actually the case? -Dan On 5/3/2013 3:26 PM, Ivan Raikov wrote: Hello, I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. If your C code is to be linked with your Chicken code, you can pass the pointer to your data from C to Scheme and use SRFI-4 foreign pointers to access it (see unit lolevel for details). If the C code is running as a separate process, you could try using posix-shm to create shared memory between processes and then use foreign pointers in the Chicken process. Ivan On May 4, 2013 3:04 AM, Pedro Melendez pmelen...@pevicom.com mailto:pmelen...@pevicom.com wrote: Hi all, Sorry if this question is obvious, but I couldn't find what I were looking for in the documentation so maybe you guys can help me. I am developing a prototype of a server that would serve 3D seismic images across the network. This task requires to process big files (~4 GB) with existing C code that is desirable to maintain. I plan to write the server itself in Chicken scheme but I would need to maintain the existing code in C that opens and process those files. Giving the size of the file, I want to share the memory space between C and Chicken and avoid copying values between areas. Is that even possible? Anyone has an idea on how can I address this? Thanks in advance! Pedro -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org mailto:Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org mailto:Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
I think you can try to have native threads by running a separate instance of the Chicken runtime for each thread, but are you sure that you will really achieve a significant speedup over Unix processes and/or MPI? It's not the early nineties anymore... I suggest prototyping code and running real-world benchmarks first. But thanks for your efforts on semaphores. I think wrapping the C API is the easy part, you can probably just use c-pointer for the sem_t type. I am guessing the hard part would be ensuring consistent semantics between Unix and the pinnacle of free-market capitalism, Microsoft Windows. On Sat, May 4, 2013 at 8:39 AM, Dan Leslie d...@ironoxide.ca wrote: Yah, I meant sem_wait et al. I was under the impression that synch and mailbox rely on srfi-18 structures, which would make them 'green' threads-only and not particularly suitable for multi process synchronization. Relatedly, is anyone poking at implementing native threads? I've been digging around a bit but haven't had much time to progress very far. I'm hesitant to take responsibility for writing a semaphore egg, but what the hell. I'll start something on GitHub this weekend. -Dan On 5/3/2013 4:22 PM, Ivan Raikov wrote: Are you talking about POSIX semaphores, sem_wait(3) and friends, or just the general semaphor data structure? If the former, then the Chicken developers are eagerly awaiting your patches ;-) If the latter, take a look at the synch and mailbox eggs. They have mutex-like functionality that can be used in place of proper semaphores. Ivan On May 4, 2013 7:59 AM, Dan Leslie d...@ironoxide.ca wrote: I was just poking through posix, posix-shm, posix-utils, and posix-extras and it seems that none of them implement semaphores! Am I missing something, or is this actually the case? -Dan On 5/3/2013 3:26 PM, Ivan Raikov wrote: Hello, I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. If your C code is to be linked with your Chicken code, you can pass the pointer to your data from C to Scheme and use SRFI-4 foreign pointers to access it (see unit lolevel for details). If the C code is running as a separate process, you could try using posix-shm to create shared memory between processes and then use foreign pointers in the Chicken process. Ivan On May 4, 2013 3:04 AM, Pedro Melendez pmelen...@pevicom.com wrote: Hi all, Sorry if this question is obvious, but I couldn't find what I were looking for in the documentation so maybe you guys can help me. I am developing a prototype of a server that would serve 3D seismic images across the network. This task requires to process big files (~4 GB) with existing C code that is desirable to maintain. I plan to write the server itself in Chicken scheme but I would need to maintain the existing code in C that opens and process those files. Giving the size of the file, I want to share the memory space between C and Chicken and avoid copying values between areas. Is that even possible? Anyone has an idea on how can I address this? Thanks in advance! Pedro -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing listChicken-users@nongnu.orghttps://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
