Re: [Chicken-users] Best way to share memory between C and Chicken
Nevertheless: maybe it would be feasible to do all second gen gc in a dedicated thread and force a minor gc before passing data between threads? Would it? Unfortunately, execution and garbage collection are tightly interleaved in the cheney-on-the-mta model. But the idea is intriguing. Implementing it would be the usual hell, of course. Yet it sounds like something one should think about. Very interesting. cheers, felix ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
Felix scripsit: Unfortunately, execution and garbage collection are tightly interleaved in the cheney-on-the-mta model. Minor collection is, but major collection could be run in a separate thread so that it can run concurrently (even on a second core) with the rest of the program. As it happens, I was reading about a concurrent GC just the other day which sounded interesting -- and just happens to be named Chicken, how cool is that? http://www.cs.technion.ac.il/~erez/Papers/real-time-pldi.pdf -- John Cowan http://ccil.org/~cowanco...@ccil.org We want more school houses and less jails; more books and less arsenals; more learning and less vice; more constant work and less crime; more leisure and less greed; more justice and less revenge; in fact, more of the opportunities to cultivate our better natures. --Samuel Gompers ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
*cringe* This will certainly expose what is and is not thread-safe within Chicken. I've worked with a /team/ in implementing such behaviour in the Unreal engine, albeit a frankenengine of sorts, and with three dedicated programmers the endeavour required about a month. But we had the benefit of knowing the totality of the code's usage, whereas implementing such behaviour could break all sorts of applications written for Chicken Scheme. OTOH, this would be an enormous boon to the flexibility of Chicken Scheme. If it were possible to safely run GC on separate threads, or at least have a sort of objective-C level of control over GC, it would make implementing native threaded applications far less... /hazy/ Just my two cents, but if the Chicken Team were to pursue this I'd like to see it as a major revision feature. IE, Chicken 5: now with very different GC behaviour! -Dan ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
From: John Cowan co...@mercury.ccil.org Subject: Re: [Chicken-users] Best way to share memory between C and Chicken Date: Mon, 6 May 2013 08:21:34 -0400 Felix scripsit: Unfortunately, execution and garbage collection are tightly interleaved in the cheney-on-the-mta model. Minor collection is, but major collection could be run in a separate thread so that it can run concurrently (even on a second core) with the rest of the program. As it happens, I was reading about a concurrent GC just the other day which sounded interesting -- and just happens to be named Chicken, how cool is that? http://www.cs.technion.ac.il/~erez/Papers/real-time-pldi.pdf Nice. What a coincidence. I'll have to read it. But implementing a collector like this is such a major piece of work that I don't see this ever happening. It took years to get the current GC into a reliable state. Running the collector, even only for major collections concurrently will need numerous changes. Currently existing headers in oldspace are modified with a forwarding pointer and thus be invalidated during GC. Those forwarding pointers could perhaps be moved into a separate table, but things don't stop there. The mutator may destructively mutate already copied oldspace data - these mutations would need to be tracked and properly applied to the copies in newspace, and so on and so on. As Dan already pointed out, there are countless places where thread-safety suddenly becomes an issue. In short: give me six months, fully compensated and you get something that works like crap. :-) cheers, felix ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
I cannot answer your question about performance, but you have prompted me
to create a simple example with POSIX shared memory. Attached are two
files, producer.c and consumer.scm. The producer (a C program) writes
strings to a POSIX shared memory file, and the consumer (a Scheme program)
reads them, prints them and then unlinks the buffer. The producer uses
mmap, as Dan mentioned in his reply. This may be clunky, but there is no
need to worry about synchronization.
Using e.g. Linux shmget for shared memory between threads _might_ give you
higher performance, but then you have to mess around with semaphores and so
on. I will not have time to adapt the code you link to, but I can help if
you are having trouble with a particular construct in posix-shm.
-Ivan
On Sun, May 5, 2013 at 8:07 AM, Jörg F. Wittenberger
joerg.wittenber...@softeyes.net wrote:
On May 4 2013, Ivan Raikov wrote:
I think you can try to have native threads by running a separate instance
of the Chicken runtime for each thread, but are you sure that you will
really achieve a significant speedup over Unix processes and/or MPI?
It's not the early nineties anymore... I suggest prototyping code and
running real-world benchmarks first.
Hm. I dunno, I'm old school.
So far I have at least one case, where I've been resorting to native
threads because I didn't see another solution. Maybe you/anybody
could enlighten me?
The case at hand: I'm running sqlite queries, which can take a long time.
I don't want to block the chicken system while those are busy. Worse: I'm
using the sqlite VFS to get the actual data blocks ... from chicken, hence
the chicken thread MUST not wait for sqlite... (for reference an old
version of the code is here http://lists.nongnu.org/**
archive/html/chicken-users/**2010-01/msg00046.htmlhttp://lists.nongnu.org/archive/html/chicken-users/2010-01/msg00046.html)
As I said: I'm old school - could I really expect a reasonable performance
from running a separate sqlite process and cop the data back and forth
between the processes? Suggestion how to do this are welcome!
On Sat, May 4, 2013 at 8:39 AM, Dan Leslie d...@ironoxide.ca wrote:
...
Relatedly, is anyone poking at implementing native threads?
I've been digging around a bit but haven't had much time to progress very
far.
In fact I'm contemplating about this for a while and never got the
courage to try. (Especially because I don't see how I could create
a half-way meaningful benchmark beforehand to judge the gain/loss
early in the development process.)
The bottleneck for me is usually blocking on i/o. Wrt. this I do not
expect (as in wild guess - no benchmarks done) much from multiple
threads.
However some things I'm doing are just computations. Am I thinking
early nineties style when I assume that chicken will utilize only
one cpu core? If so, I'd prefer not spare the others all the time.
My problem: I can't see a good way to share chicken-managed memory
between threads. I recall having seen a paper on using a dedicated
thread for memory allocation...but can't find it anymore.
Nevertheless: maybe it would be feasible to do all second gen
gc in a dedicated thread and force a minor gc before passing data
between threads? Would it?
Best
/Jörg
#include sys/types.h
#include sys/mman.h
#include sys/stat.h/* For mode constants */
#include unistd.h
#include fcntl.h /* For O_* constants */
#include stdio.h
#include stdlib.h
#include string.h
#include errno.h
#define BUFFER_SIZE 32768
#define BUFFER_NAME /shmtest1
void buffer_create(int size, int *fileno, char **addr)
{
int perms = 0600;
char *laddr;
*addr = 0;
*fileno = shm_open(BUFFER_NAME, O_CREAT | O_RDWR, perms);
if (errno 0)
{
perror(failed to create shared memory file handle);
exit (1);
}
if ((ftruncate(*fileno, size)) == -1)
{/* Sets the size */
perror(ftruncate failed on shared memory file handle);
exit(1);
}
laddr = (char*)mmap(NULL, size, PROT_WRITE, MAP_SHARED, *fileno, 0);
if (errno 0)
{
perror (failed to mmap shared memory file handle);
exit (1);
}
*addr = laddr;
}
void buffer_clear(char **sbuff, char *start)
{
*sbuff = start;
}
int buffer_size(char *sbuff, char *start)
{
return (sbuff-start);
}
void buffer_close(int fileno, char *start)
{
if ((munmap(start, BUFFER_SIZE)) == -1)
{
perror(munmap failed on shared buffer);
exit(1);
}
if ((close(fileno)) == -1)
{
perror(close failed on shared buffer file handle);
exit(1);
}
}
void insert_item(char *item, char **shared_buffer, char *shared_start)
{
int n = strlen(item);
memcpy (*shared_buffer, item, n);
*shared_buffer = *shared_buffer+n;
}
int main(int argc, const char **argv)
{
char *shared_start, *shared_buffer;
int shared_fileno;
buffer_create(BUFFER_SIZE, shared_fileno,
Re: [Chicken-users] Best way to share memory between C and Chicken
From: Thomas Chust ch...@web.de Subject: Re: [Chicken-users] Best way to share memory between C and Chicken Date: Sat, 04 May 2013 00:57:25 +0200 On 2013-05-04 00:26, Ivan Raikov wrote: [...] I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. [...] Hello, would that be related to the fact that CHICKEN has a copying garbage collector or are there other hidden performance implications of SRFI-4 vectors? SRFI-4 vectors live in the normal, heap that is subject to garbage collection, so, yes - they are always copied completely on every major collection (and the first minor GC during the lifetime of the SRFI-4 vector object). But note that the SRFI-4 constructors accept an optional argument which specifies that the vector should be allocated in static, non-GC'd memory. In that case the storage has to be reclaimed explicitly (using release-number-vector, which is, funnily enough, not documented - well, it is now) or by giving yet another optional argument to use a finalizer for de-allocation. cheers, felix ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
On 2013-05-05 13:54, Felix wrote: [...] SRFI-4 vectors live in the normal, heap that is subject to garbage collection, [...] But note that the SRFI-4 constructors accept an optional argument which specifies that the vector should be allocated in static, non-GC'd memory. [...] Ah, cool. That's probably the perfect solution for large numeric arrays! Ciao, Thomas -- When C++ is your hammer, every problem looks like your thumb. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
I'll probably write the wiki page while taking the bus to work next week, but for now I'm unaware of any issues and have a working example. There's a test.scm file in the repo that gives a quick rundown on how to use a (non-shared) semaphore. That is to say, if you're using fork() and only wish to synchronize processes in this sub-tree then you should be good with the example in test.scm. If you wish to synchronize unrelated processes then you'll need to use posix-shm to store the semaphores. This O'Reilly page gives a good walk through: http://www.linuxdevcenter.com/pub/a/linux/2007/05/24/semaphores-in-linux.html?page=5 -Dan ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
On May 4 2013, Ivan Raikov wrote: I think you can try to have native threads by running a separate instance of the Chicken runtime for each thread, but are you sure that you will really achieve a significant speedup over Unix processes and/or MPI? It's not the early nineties anymore... I suggest prototyping code and running real-world benchmarks first. Hm. I dunno, I'm old school. So far I have at least one case, where I've been resorting to native threads because I didn't see another solution. Maybe you/anybody could enlighten me? The case at hand: I'm running sqlite queries, which can take a long time. I don't want to block the chicken system while those are busy. Worse: I'm using the sqlite VFS to get the actual data blocks ... from chicken, hence the chicken thread MUST not wait for sqlite... (for reference an old version of the code is here http://lists.nongnu.org/archive/html/chicken-users/2010-01/msg00046.html ) As I said: I'm old school - could I really expect a reasonable performance from running a separate sqlite process and cop the data back and forth between the processes? Suggestion how to do this are welcome! On Sat, May 4, 2013 at 8:39 AM, Dan Leslie d...@ironoxide.ca wrote: ... Relatedly, is anyone poking at implementing native threads? I've been digging around a bit but haven't had much time to progress very far. In fact I'm contemplating about this for a while and never got the courage to try. (Especially because I don't see how I could create a half-way meaningful benchmark beforehand to judge the gain/loss early in the development process.) The bottleneck for me is usually blocking on i/o. Wrt. this I do not expect (as in wild guess - no benchmarks done) much from multiple threads. However some things I'm doing are just computations. Am I thinking early nineties style when I assume that chicken will utilize only one cpu core? If so, I'd prefer not spare the others all the time. My problem: I can't see a good way to share chicken-managed memory between threads. I recall having seen a paper on using a dedicated thread for memory allocation...but can't find it anymore. Nevertheless: maybe it would be feasible to do all second gen gc in a dedicated thread and force a minor gc before passing data between threads? Would it? Best /Jörg ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
With semaphores and shared memory you can basically emulate the behaviour you would expect from native threads. Though I'm not certain about how comparable it is in performance, I understand it to be quite zippy. Ie, shm_open will give you a file descriptor, which you can then treat as a memory buffer by using mmap, and using semaphores you can safely signal across processes whether someone is in a critical section or not. To me, the drawback is in that creating a shared memory object and wrapping it in mmap is clunkier than just sharing all in-process memory with a bunch of native threads. But the upshot is that it can be a lot safer in practice, especially when you have many hands in the pot and not much rigor is being taken towards thread safety. Regarding Chicken threads, as far as I'm aware SRFI-18 threads in Chicken are 'green', in that they run on a single native thread and cannot avail themselves of additional native threads. You mention that you've been poking at native threads for a while. Do you have a native thread implementation that could be packaged up as an egg? I think some people would be eager to try it out, even if its incomplete. Maybe this would be a good opportunity to get used to Git via a GitHub project? ;) Thanks, -Dan ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
If it's heterogeneous data I tend to use blobs with helper functions to cast segments to records without undertaking a copy. https://wiki.call-cc.org/man/4/Unit%20library#blobs If it's homogenous data then the srfi-4 unit provides all sorts of vectors that you can use. https://wiki.call-cc.org/man/4/Unit%20srfi-4 Keep in mind that foreign-lambdas will take blob and srfi-4 vectors as parameter types and convert them to their relevant C types without undertaking a copy. https://wiki.call-cc.org/man/4/Accessing%20external%20objects -Dan On 5/3/2013 11:04 AM, Pedro Melendez wrote: Hi all, Sorry if this question is obvious, but I couldn't find what I were looking for in the documentation so maybe you guys can help me. I am developing a prototype of a server that would serve 3D seismic images across the network. This task requires to process big files (~4 GB) with existing C code that is desirable to maintain. I plan to write the server itself in Chicken scheme but I would need to maintain the existing code in C that opens and process those files. Giving the size of the file, I want to share the memory space between C and Chicken and avoid copying values between areas. Is that even possible? Anyone has an idea on how can I address this? Thanks in advance! Pedro -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
On 2013-05-03 20:04, Pedro Melendez wrote: [...] I am developing a prototype of a server that would serve 3D seismic images across the network. [...] Hello Pedro, it's nice to see I'm not the only geophysicist who likes to use Scheme :-) [...] Giving the size of the file, I want to share the memory space between C and Chicken and avoid copying values between areas. Is that even possible? [...] Both CHICKEN's blobs and SRFI-4 homogeneous vectors can be converted to native pointers to their data contents through the FFI. As a silly example, this will allocate a big array of floating point numbers, set one of the array elements using native code and read it bacl from Scheme: (use srfi-4) (define big-array (make-f64vector (expt 2 20))) (define set-some-element! (foreign-lambda* void ((nonnull-f64vector v)) v[1024] = 42.23;)) (set-some-element! big-array) (display (f64vector-ref big-array 1024)) (newline) If your data is not a homogeneous numeric array, you could use blobs instead. If you absolutely have to allocate unmanaged memory on the C side an pass pointers there back to Scheme rather than creating managed Scheme objects and passing pointers to their contents to the C code, take a look at CHICKEN's lolevel unit. CHICKEN Scheme can also deal with raw native pointers to a certain degree! Ciao, Thomas -- When C++ is your hammer, every problem looks like your thumb. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
Hello, I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. If your C code is to be linked with your Chicken code, you can pass the pointer to your data from C to Scheme and use SRFI-4 foreign pointers to access it (see unit lolevel for details). If the C code is running as a separate process, you could try using posix-shm to create shared memory between processes and then use foreign pointers in the Chicken process. Ivan On May 4, 2013 3:04 AM, Pedro Melendez pmelen...@pevicom.com wrote: Hi all, Sorry if this question is obvious, but I couldn't find what I were looking for in the documentation so maybe you guys can help me. I am developing a prototype of a server that would serve 3D seismic images across the network. This task requires to process big files (~4 GB) with existing C code that is desirable to maintain. I plan to write the server itself in Chicken scheme but I would need to maintain the existing code in C that opens and process those files. Giving the size of the file, I want to share the memory space between C and Chicken and avoid copying values between areas. Is that even possible? Anyone has an idea on how can I address this? Thanks in advance! Pedro -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
On 2013-05-04 00:26, Ivan Raikov wrote: [...] I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. [...] Hello, would that be related to the fact that CHICKEN has a copying garbage collector or are there other hidden performance implications of SRFI-4 vectors? Ciao, Thomas -- When C++ is your hammer, every problem looks like your thumb. -- When C++ is your hammer, every problem looks like your thumb. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
I was just poking through posix, posix-shm, posix-utils, and posix-extras and it seems that none of them implement semaphores! Am I missing something, or is this actually the case? -Dan On 5/3/2013 3:26 PM, Ivan Raikov wrote: Hello, I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. If your C code is to be linked with your Chicken code, you can pass the pointer to your data from C to Scheme and use SRFI-4 foreign pointers to access it (see unit lolevel for details). If the C code is running as a separate process, you could try using posix-shm to create shared memory between processes and then use foreign pointers in the Chicken process. Ivan On May 4, 2013 3:04 AM, Pedro Melendez pmelen...@pevicom.com mailto:pmelen...@pevicom.com wrote: Hi all, Sorry if this question is obvious, but I couldn't find what I were looking for in the documentation so maybe you guys can help me. I am developing a prototype of a server that would serve 3D seismic images across the network. This task requires to process big files (~4 GB) with existing C code that is desirable to maintain. I plan to write the server itself in Chicken scheme but I would need to maintain the existing code in C that opens and process those files. Giving the size of the file, I want to share the memory space between C and Chicken and avoid copying values between areas. Is that even possible? Anyone has an idea on how can I address this? Thanks in advance! Pedro -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org mailto:Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
Hello, Yes, it seems that copying/moving around large vectors puts a lot of pressure on the memory subsystem. Other than that, SRFI-4 vectors are fine. Ivan On May 4, 2013 7:57 AM, Thomas Chust ch...@web.de wrote: On 2013-05-04 00:26, Ivan Raikov wrote: [...] I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. [...] Hello, would that be related to the fact that CHICKEN has a copying garbage collector or are there other hidden performance implications of SRFI-4 vectors? Ciao, Thomas -- When C++ is your hammer, every problem looks like your thumb. -- When C++ is your hammer, every problem looks like your thumb. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
Are you talking about POSIX semaphores, sem_wait(3) and friends, or just the general semaphor data structure? If the former, then the Chicken developers are eagerly awaiting your patches ;-) If the latter, take a look at the synch and mailbox eggs. They have mutex-like functionality that can be used in place of proper semaphores. Ivan On May 4, 2013 7:59 AM, Dan Leslie d...@ironoxide.ca wrote: I was just poking through posix, posix-shm, posix-utils, and posix-extras and it seems that none of them implement semaphores! Am I missing something, or is this actually the case? -Dan On 5/3/2013 3:26 PM, Ivan Raikov wrote: Hello, I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. If your C code is to be linked with your Chicken code, you can pass the pointer to your data from C to Scheme and use SRFI-4 foreign pointers to access it (see unit lolevel for details). If the C code is running as a separate process, you could try using posix-shm to create shared memory between processes and then use foreign pointers in the Chicken process. Ivan On May 4, 2013 3:04 AM, Pedro Melendez pmelen...@pevicom.com wrote: Hi all, Sorry if this question is obvious, but I couldn't find what I were looking for in the documentation so maybe you guys can help me. I am developing a prototype of a server that would serve 3D seismic images across the network. This task requires to process big files (~4 GB) with existing C code that is desirable to maintain. I plan to write the server itself in Chicken scheme but I would need to maintain the existing code in C that opens and process those files. Giving the size of the file, I want to share the memory space between C and Chicken and avoid copying values between areas. Is that even possible? Anyone has an idea on how can I address this? Thanks in advance! Pedro -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing listChicken-users@nongnu.orghttps://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
Yah, I meant sem_wait et al. I was under the impression that synch and mailbox rely on srfi-18 structures, which would make them 'green' threads-only and not particularly suitable for multi process synchronization. Relatedly, is anyone poking at implementing native threads? I've been digging around a bit but haven't had much time to progress very far. I'm hesitant to take responsibility for writing a semaphore egg, but what the hell. I'll start something on GitHub this weekend. -Dan On 5/3/2013 4:22 PM, Ivan Raikov wrote: Are you talking about POSIX semaphores, sem_wait(3) and friends, or just the general semaphor data structure? If the former, then the Chicken developers are eagerly awaiting your patches ;-) If the latter, take a look at the synch and mailbox eggs. They have mutex-like functionality that can be used in place of proper semaphores. Ivan On May 4, 2013 7:59 AM, Dan Leslie d...@ironoxide.ca mailto:d...@ironoxide.ca wrote: I was just poking through posix, posix-shm, posix-utils, and posix-extras and it seems that none of them implement semaphores! Am I missing something, or is this actually the case? -Dan On 5/3/2013 3:26 PM, Ivan Raikov wrote: Hello, I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. If your C code is to be linked with your Chicken code, you can pass the pointer to your data from C to Scheme and use SRFI-4 foreign pointers to access it (see unit lolevel for details). If the C code is running as a separate process, you could try using posix-shm to create shared memory between processes and then use foreign pointers in the Chicken process. Ivan On May 4, 2013 3:04 AM, Pedro Melendez pmelen...@pevicom.com mailto:pmelen...@pevicom.com wrote: Hi all, Sorry if this question is obvious, but I couldn't find what I were looking for in the documentation so maybe you guys can help me. I am developing a prototype of a server that would serve 3D seismic images across the network. This task requires to process big files (~4 GB) with existing C code that is desirable to maintain. I plan to write the server itself in Chicken scheme but I would need to maintain the existing code in C that opens and process those files. Giving the size of the file, I want to share the memory space between C and Chicken and avoid copying values between areas. Is that even possible? Anyone has an idea on how can I address this? Thanks in advance! Pedro -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org mailto:Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org mailto:Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Best way to share memory between C and Chicken
I think you can try to have native threads by running a separate instance of the Chicken runtime for each thread, but are you sure that you will really achieve a significant speedup over Unix processes and/or MPI? It's not the early nineties anymore... I suggest prototyping code and running real-world benchmarks first. But thanks for your efforts on semaphores. I think wrapping the C API is the easy part, you can probably just use c-pointer for the sem_t type. I am guessing the hard part would be ensuring consistent semantics between Unix and the pinnacle of free-market capitalism, Microsoft Windows. On Sat, May 4, 2013 at 8:39 AM, Dan Leslie d...@ironoxide.ca wrote: Yah, I meant sem_wait et al. I was under the impression that synch and mailbox rely on srfi-18 structures, which would make them 'green' threads-only and not particularly suitable for multi process synchronization. Relatedly, is anyone poking at implementing native threads? I've been digging around a bit but haven't had much time to progress very far. I'm hesitant to take responsibility for writing a semaphore egg, but what the hell. I'll start something on GitHub this weekend. -Dan On 5/3/2013 4:22 PM, Ivan Raikov wrote: Are you talking about POSIX semaphores, sem_wait(3) and friends, or just the general semaphor data structure? If the former, then the Chicken developers are eagerly awaiting your patches ;-) If the latter, take a look at the synch and mailbox eggs. They have mutex-like functionality that can be used in place of proper semaphores. Ivan On May 4, 2013 7:59 AM, Dan Leslie d...@ironoxide.ca wrote: I was just poking through posix, posix-shm, posix-utils, and posix-extras and it seems that none of them implement semaphores! Am I missing something, or is this actually the case? -Dan On 5/3/2013 3:26 PM, Ivan Raikov wrote: Hello, I really strongly advise _against_ using SRFI-4 vectors for 4G files, as I have experienced serious performance issues even with vectors of a few million elements. If your C code is to be linked with your Chicken code, you can pass the pointer to your data from C to Scheme and use SRFI-4 foreign pointers to access it (see unit lolevel for details). If the C code is running as a separate process, you could try using posix-shm to create shared memory between processes and then use foreign pointers in the Chicken process. Ivan On May 4, 2013 3:04 AM, Pedro Melendez pmelen...@pevicom.com wrote: Hi all, Sorry if this question is obvious, but I couldn't find what I were looking for in the documentation so maybe you guys can help me. I am developing a prototype of a server that would serve 3D seismic images across the network. This task requires to process big files (~4 GB) with existing C code that is desirable to maintain. I plan to write the server itself in Chicken scheme but I would need to maintain the existing code in C that opens and process those files. Giving the size of the file, I want to share the memory space between C and Chicken and avoid copying values between areas. Is that even possible? Anyone has an idea on how can I address this? Thanks in advance! Pedro -- T: +1 (416) - 357.5356 Skype ID: pmelendezu ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing listChicken-users@nongnu.orghttps://lists.nongnu.org/mailman/listinfo/chicken-users ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
