[Chicken-hackers] Need help to understand C_mutate better.
Hi all, this goes to both chicken-users and chicken-hackers. I feel it's more appropriate to chicken-hackers, but recently I bothered the users with help requests on the issue - maybe it's useful to learn where this ends up. The problem I ran into: Much depending on optimisation being done or not there is a probability of a rather complex chicken application to run into a mode (usually right within the initialisation process, but sometimes later too) where it would eat all memory. Until today no way to derive a test case. (Which makes me look stupid, having spent 20 days on it.) Using glibc's mtrace function I've been able to see that a good run, where the programm would work for 16718 lines in the trace file and 4 heap resize operation did need two reallocations in C_mutate. A bad run of approximately the same size (16636 lines in the mtrace report) shows exactly the same pattern of heap resize operations. However instead of two entries from C_mutate I get 852. Towards the end of the log there is almost only C_mutate growing it's mutation stack more and more. I wonder if something like this has ever been observed before. Is there any know pitfall how one could create such a condition? So far I've been able to reproduce this problem on Ubuntu gcc 4.5.2 . Strangely the debian system with gcc 4.4.5 is not effected. Neither is the Sheeva Plug. Thanks for every hint /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Need help to understand C_mutate better.
Hi, Trying to get a better understanding where C_mutate might be called in a way it can end up growing the mutation_stack beyond bounds, I found in runtime.c: C_h_pair C_h_vector C_h_structure C_h_list Those appear to be obsolete. Or at least I can't find any reference to them from the rest of the code. The only one from the C_h_ family would be C_h_intern where the c-backend.scm compiles calls into the generated .c files. Should those functions go? For the C_mutate, whoever has any wild guess how this behaviour could come up, please help me with that one. (BTW: I'm rather positive that this has nothing to do with changes I made to my chicken. Those I came to make because the I've been trying to locate that problem. As a secondary effect the system ran somewhat faster, increasing the probability of the issue to coming up. Therefore my bet is that it's in native chicken too.) On Oct 20 2011, Jörg F. Wittenberger wrote: Towards the end of the log there is almost only C_mutate growing it's mutation stack more and more. I wonder if something like this has ever been observed before. Is there any know pitfall how one could create such a condition? So far I've been able to reproduce this problem on Ubuntu gcc 4.5.2 . Strangely the debian system with gcc 4.4.5 is not effected. Neither is the Sheeva Plug. Thanks for every hint ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Need help to understand C_mutate better.
On Oct 21 2011, Jörg F. Wittenberger wrote: For the C_mutate, whoever has any wild guess how this behaviour could come up, please help me with that one. I wonder: C_mutate is (in runtime.c) sometimes used like this if(C_in_stackp(x)) C_mutate(y,x); else y=x; and sometimes the C_in_stackp() is not there. I fail to see that this missing in stack test would harm; but maybe that's just may fault. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Need help to understand C_mutate better.
On Oct 21 2011, Felix wrote:
For the C_mutate, whoever has any wild guess how this
behaviour could come up, please help me with that one.
Do you use C_mutate in C code?
Nowhere.
The mutation stack will
grow until a garbage collection takes place, so if you
invoke C_mutate in a C loop without giving GC a chance,
the mutations will just add up.
If I compile (maybe too much) with disable-interrupts
could this prevent a garbage collection to happen?
But I doubt, because I see the programm working as expected
except for slowness and memory consumption. This means
scheduling and i/o works. The i/o is definately run
from code, which has interrupts enabled.
So far I found a horrible workaround: set C_enable_interrupts
to zero! I'm afraid this works only, because I have some
modules compiled with disable-interrupts where I actually
want thread switches. Within those there is a manual
check for interrupts. I'm afraid in the workaround mode
this is why it's working at all (and not at the ARM).
The stack-test you pointed out is just a small optimization, the
mutation stack is needed to keep track of pointers in the heap that
might point to data in the nursery (stack). Adding these checks can
avoid adding the mutation-stack entry sometimes. If GC takes place
often enough (which it does in normal, compiled Scheme code), it is
probably not too much of an overhead to just add the mutation-stack
entry.
I'm attaching a diff against current git master.
This does NOT help with the C_mutate issue I see.
It pull the simple tests first and inlines C_in_stackp .
Running on that code it looks as if the optimization is not
that small.
Note that I did not complete all of the C_h_* family, since
I'm afraid those are really obsolete.
/Jörg
--- /home/jfw/build/Scheme/chicken-core/runtime.c 2011-10-14 18:24:28.0 +0200
+++ runtime.c 2011-10-21 14:38:42.269533964 +0200
@@ -1790,10 +1790,20 @@
return C_restore;
}
+C_inline C_regparm int C_i_in_stackp(C_word x)
+{
+ C_word *ptr = (C_word *)(C_uword)x;
+
+#if C_STACK_GROWS_DOWNWARD
+ return ptr = C_stack_pointer_test ptr = stack_bottom;
+#else
+ return ptr C_stack_pointer_test ptr = stack_bottom;
+#endif
+}
void C_fcall C_callback_adjust_stack(C_word *a, int size)
{
- if(!chicken_is_running !C_in_stackp((C_word)a)) {
+ if(!chicken_is_running !C_i_in_stackp((C_word)a)) {
if(debug_mode)
C_dbg(C_text(debug),
C_text(callback invoked in lower stack region - adjusting limits:\n
@@ -1968,7 +1978,7 @@
key = hash_string(len, str, stable-size);
if(C_truep(s = lookup(key, len, str, stable))) {
-if(C_in_stackp(s)) C_mutate(slot, s);
+if(C_i_in_stackp(s)) C_mutate(slot, s);
return s;
}
@@ -2109,19 +2119,11 @@
return sym;
}
-
C_regparm int C_in_stackp(C_word x)
{
- C_word *ptr = (C_word *)(C_uword)x;
-
-#if C_STACK_GROWS_DOWNWARD
- return ptr = C_stack_pointer_test ptr = stack_bottom;
-#else
- return ptr C_stack_pointer_test ptr = stack_bottom;
-#endif
+ return C_i_in_stackp(x);
}
-
C_regparm int C_fcall C_in_heapp(C_word x)
{
C_byte *ptr = (C_byte *)(C_uword)x;
@@ -2569,7 +2571,7 @@
while(n--) {
x = va_arg(v, C_word);
-if(C_in_stackp(x)) C_mutate(p++, x);
+if(C_i_in_stackp(x)) C_mutate(p++, x);
else *(p++) = x;
}
@@ -2593,7 +2595,7 @@
while(n--) {
x = va_arg(v, C_word);
-if(C_in_stackp(x)) C_mutate(p++, x);
+if(C_i_in_stackp(x)) C_mutate(p++, x);
else *(p++) = x;
}
@@ -2670,7 +2672,7 @@
C_uword count, bytes;
C_word *p, **msp, bucket, last, item, container;
C_header h;
- C_byte *tmp, *start;
+ C_byte *tmp, *start = C_fromspace_top;
LF_LIST *lfn;
C_SCHEME_BLOCK *bp;
C_GC_ROOT *gcrp;
@@ -2696,7 +2698,7 @@
gc_mode = GC_MINOR;
/* Entry point for second-level GC (on explicit request or because of full fromspace): */
- if(C_setjmp(gc_restart) || (start = C_fromspace_top) = C_fromspace_limit) {
+ if(C_setjmp(gc_restart) || start = C_fromspace_limit) {
if(gc_bell) {
C_putchar(7);
C_fflush(stdout);
@@ -3424,7 +3426,7 @@
if(is_fptr(h)) /* forwarded? update l-table entry */
loc = locative_table[ i ] = fptr_to_ptr(h);
/* otherwise it must have been GC'd (since this is a minor one) */
-else if(C_in_stackp(loc)) {
+else if(C_i_in_stackp(loc)) {
locative_table[ i ] = C_SCHEME_UNDEFINED;
C_set_block_item(loc, 0, 0);
++invalidated;
@@ -3441,7 +3443,7 @@
C_set_block_item(loc, 0, (C_uword)fptr_to_ptr(h) + offset);
hi = i + 1;
}
-else if(C_in_stackp(obj)) { /* pointed-at object GC'd, locative is invalid */
+else if(C_i_in_stackp(obj)) { /* pointed-at object GC'd, locative is invalid */
locative_table[ i ] = C_SCHEME_UNDEFINED;
C_set_block_item(loc, 0, 0);
}
@@ -5025,7 +5027,8 @@
if(C_immediatep(x) || C_block_header(x) != C_PAIR_TAG)
Re: [Chicken-hackers] Need help to understand C_mutate better.
On Oct 21 2011, Jörg F. Wittenberger wrote: The mutation stack will grow until a garbage collection takes place, so if you invoke C_mutate in a C loop without giving GC a chance, the mutations will just add up. Reading C_reclaim still without fully understanding how exactly is will interact with C_context_switch it appears to me that if there where always an interrupt pending (or at least too often) then chicken would dispatch to the interrupt handler, which will switch context. I don't see how the C code would return to the (minor) gc. In handle_interrupt is even mentioned that it must not return. If the next signal arrives before the next gc run this would take the same path. This way the mutation stack will grow. Slightly at first, but the larger it becomes, the longer it takes to relocate, thereby increasing the probability of yet another turn. If that was correct, it looks as if it would be better to move the interrupt handling towards the end of gc. (Not sure that I could do that.) Given that what I have to do to kickstart the excessive memory consumption it involves massive handling signals and i/o. /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Need help to understand C_mutate better.
From that theory I've been working in trialerror mode again. I added a (gc) call to the end of my SIGCHLD handler. Guess what: It did not run into the first, almost-for-sure case. Often enough to make me walk the dog waiting for the second case: the laptop coming back from a longer suspension period. Now I wonder: as far as I understand C_mutate it will remember all assignments where the assigned value in within the nursery. True? If true, than I don't understand why we allow the mutation_stack to ever grow beyond the size of the nursery. So far untested/unimplemented (I'd rather get that understanding of the mutation stack before I continue here): If we would check a mutation stack usage limit before the interrupt limit and at least enforce a minor gc on excess, we might be able to leave the interrupt handling at the begin of the C_reclaim and still avoid running havoc. Reasonable? /Jörg On Oct 21 2011, Jörg F. Wittenberger wrote: The mutation stack will grow until a garbage collection takes place, so if you invoke C_mutate in a C loop without giving GC a chance, the mutations will just add up. Reading C_reclaim still without fully understanding how exactly is will interact with C_context_switch it appears to me that if there where always an interrupt pending (or at least too often) then chicken would dispatch to the interrupt handler, which will switch context. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Need help to understand C_mutate better.
On Oct 22 2011, Felix wrote: From: Jörg F. Wittenberger joerg.wittenber...@softeyes.net Now I wonder: as far as I understand C_mutate it will remember all assignments where the assigned value in within the nursery. It will remember all assignments, later GC will make the (important) distinction whether there is heap-to-nursery pointer. I've got that. I meant that it would remember assignments to catch all those, which would be important (heap-to-nursery). Sorry haven't tried to describe what it does obviously, but what I understood it's purpose would be. Though it would sometimes and sometimes not be called, since a check might avoid the C_mutate when not needed. It might be an option to do the in-stack test directly inside C_mutate. I don't think it buys much, though. The patch I attached to the other posting would just do that. Plus it did so inline instead of calling a procedure for the stack check. And it pulled the in-stack and the immediate check before the C_mutate call. Which would save at least one procedure call per assignment another one if the in-stack check failed plus the processing at gc time. Since you confirmed the background so far, I still wonder: If true, than I don't understand why we allow the mutation_stack to ever grow beyond the size of the nursery. It's not the *nursery* size, but the heap size. But otherwise the point of my posting was: So far untested/unimplemented (I'd rather get that understanding of the mutation stack before I continue here): If we would check a mutation stack usage limit before the interrupt limit and at least enforce a minor gc on excess, we might be able to leave the interrupt handling at the begin of the C_reclaim and still avoid running havoc. Reasonable? Best Regards /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Need help to understand C_mutate better.
On Oct 24 2011, Felix wrote: Running on that code it looks as if the optimization is not that small. So is performance the point of this patch? Because it is not completely clear what this patch addresses. The mutation stack only overflows if you avoid GC (which can be easily traced), are you doing that? Yes it is here. It does not address any bug at all. /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Need help to understand C_mutate better.
On Oct 24 2011, Felix wrote: From: Jörg F. Wittenberger joerg.wittenber...@softeyes.net Subject: Re: [Chicken-hackers] Need help to understand C_mutate better. Date: 21 Oct 2011 15:58:17 +0200 On Oct 21 2011, Jörg F. Wittenberger wrote: The mutation stack will grow until a garbage collection takes place, so if you invoke C_mutate in a C loop without giving GC a chance, the mutations will just add up. Reading C_reclaim still without fully understanding how exactly is will interact with C_context_switch it appears to me that if there where always an interrupt pending (or at least too often) then chicken would dispatch to the interrupt handler, which will switch context. I don't see how the C code would return to the (minor) gc. In handle_interrupt is even mentioned that it must not return. It doesn't - the GC is aborted. Interrupt frequency may simply not be too excessive or GC will not run. This means *really* excessive, though. Excessively excessive, which might indicate a design problem. What would be the design rational for which interrupt handling must be able to suppress gc? Couldn't either the signal handler return to gc, or - my preferred alternative - the minor gc run prior to the signal handler? Would make chicken robust to the case at hand. Also keeping the size of the mutation_stack down in reasonable bonds would IMHO make chicken better. (More conservative wrt. resources.) When circumstances put too much stress on chicken, then it *can* run havoc. If there where no escape, that would be a design issue. Fortunately escape is: run (gc) in signal handlers. That way it's transformed into a usability issue. If the next signal arrives before the next gc run this would take the same path. This way the mutation stack will grow. Slightly at first, but the larger it becomes, the longer it takes to relocate, thereby increasing the probability of yet another turn. What rate of signals/interrupts would you expect for a heavily loaded instance of your application? My prog works quite well under normal circumstances, and shows no problems at all under rscheme. It then runs in about 70M RAM and CPU like 0-5%. There are usually 50-100 subprocesses doing ssl connections. (Mostly standby) And there is this sqlite3 database. Once upon a time the database is refreshed from disk content. Thats a background job. Upon startup or when the laptop comes back from sleep, it might find out the database needs refresh. Chicken will read the disk file by file and talk to the sqlite3 database (other pthread). At the same time those 50-100 connection are reattempt. Several of those will fail for sure. Thus we'll see up to retry-times many subprocesses die off within short time. This is sometimes enough. (roughly 50%+/-20%) Given that what I have to do to kickstart the excessive memory consumption it involves massive handling signals and i/o. You can set C_post_gc_hook or set-gc-report! to enable debug output which will show you if GC's take place. I will also add some debug output for mutation-stack reallocation. Thanks. -:g -:D plus glib mtrace did the job for me so far. Also I hope we'll not see it soon again. (I need remember to remove that (gc) from my signal handler, once I can.) Best Regards /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Need help to understand C_mutate better.
On Oct 25 2011, Felix wrote: Couldn't either the signal handler return to gc, or - my preferred alternative - the minor gc run prior to the signal handler? Would make chicken robust to the case at hand. This sounds possible, but it will trigger a collection even in cases where it is not needed, that is, when C_reclaim was entered due to an interrupt. Sure it would. But this should be little to no cost over all. It would be a minor gc. Since this clear the mutation stack (O(n) in entries) doing so later (with a some more entries in the m-stack) should IMHO cost the same time later, shouldn't it? /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] an interesting idea?
I just updated to current master.
First notice: with the modified allocation scheme for the m-stack
I observe a larger memory footprint.
The attached diff does roughly half the time the chicken executable
needs compiling it's own source.
IMHO it would be an interesting idea to convert C_mutate to
a macro, which would avoid any procedure call by lifting the check
if(!C_immediatep(val) C_i_in_stackp(val)) {
from the current C_mutate procedure into the TDB C_mutate macro
and call the TDB C_do_mutate procedure only in case.
/Jörg
--- /home/jfw/build/Scheme/chicken-core/runtime.c 2011-10-26 18:49:46.0 +0200
+++ runtime.c 2011-10-26 21:05:11.363641790 +0200
@@ -1792,7 +1792,7 @@
void C_fcall C_callback_adjust_stack(C_word *a, int size)
{
- if(!chicken_is_running !C_in_stackp((C_word)a)) {
+ if(!chicken_is_running !C_i_in_stackp((C_word)a)) {
if(debug_mode)
C_dbg(C_text(debug),
C_text(callback invoked in lower stack region - adjusting limits:\n
@@ -1966,11 +1966,8 @@
key = hash_string(len, str, stable-size);
- if(C_truep(s = lookup(key, len, str, stable))) {
-if(C_in_stackp(s)) C_mutate(slot, s);
-
-return s;
- }
+ if(C_truep(s = lookup(key, len, str, stable)))
+return C_mutate(slot, s);
s = C_static_string(C_heaptop, len, str);
return add_symbol(C_heaptop, key, s, stable);
@@ -2109,7 +2106,7 @@
}
-C_regparm int C_in_stackp(C_word x)
+C_inline C_regparm int C_i_in_stackp(C_word x)
{
C_word *ptr = (C_word *)(C_uword)x;
@@ -2120,6 +2117,10 @@
#endif
}
+C_regparm int C_in_stackp(C_word x)
+{
+ return C_i_in_stackp(x);
+}
C_regparm int C_fcall C_in_heapp(C_word x)
{
@@ -2539,7 +2540,7 @@
{
unsigned int mssize, newmssize, bytes;
- if(!C_immediatep(val)) {
+ if(!C_immediatep(val) C_i_in_stackp(val)) {
#ifdef C_GC_HOOKS
if(C_gc_mutation_hook != NULL C_gc_mutation_hook(slot, val)) return val;
#endif
@@ -3362,7 +3363,7 @@
if(is_fptr(h)) /* forwarded? update l-table entry */
loc = locative_table[ i ] = fptr_to_ptr(h);
/* otherwise it must have been GC'd (since this is a minor one) */
-else if(C_in_stackp(loc)) {
+else if(C_i_in_stackp(loc)) {
locative_table[ i ] = C_SCHEME_UNDEFINED;
C_set_block_item(loc, 0, 0);
++invalidated;
@@ -3379,7 +3380,7 @@
C_set_block_item(loc, 0, (C_uword)fptr_to_ptr(h) + offset);
hi = i + 1;
}
-else if(C_in_stackp(obj)) { /* pointed-at object GC'd, locative is invalid */
+else if(C_i_in_stackp(obj)) { /* pointed-at object GC'd, locative is invalid */
locative_table[ i ] = C_SCHEME_UNDEFINED;
C_set_block_item(loc, 0, 0);
}
@@ -7985,10 +7986,10 @@
flist-next = finalizer_list;
finalizer_list = flist;
- if(C_in_stackp(x)) C_mutate(flist-item, x);
+ if(C_i_in_stackp(x)) C_mutate(flist-item, x);
else flist-item = x;
- if(C_in_stackp(proc)) C_mutate(flist-finalizer, proc);
+ if(C_i_in_stackp(proc)) C_mutate(flist-finalizer, proc);
else flist-finalizer = proc;
++live_finalizer_count;
___
Chicken-hackers mailing list
Chicken-hackers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] Overhaul interrupt handling
On Oct 27 2011, Felix wrote: This patch adds some cleanups and enhancements to the interrupt and signal-handling facilities, as posted recently. I found the patch would rename never_mind_edsgar into i_like_spaghetti. Thus the reference in this message would need a change now. http://lists.nongnu.org/archive/html/chicken-users/2011-10/msg00037.html But maybe it's easier to just give the count a value. /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] Overhaul interrupt handling
On Oct 27 2011, Felix wrote: I found the patch would rename never_mind_edsgar into i_like_spaghetti. Yeah, it's a great idea, isn't it? I'm particularly proud of this one. Sure it is! My Congratulations. BTW: recently I started to wonder: it's a nice feature that chicken comes with only two files runtime.c and chicken.h as C-level dependencies. This should be kept. SQLite happens to do something similar (normal build dependencies in just two files). As it grew it slightly changed the way the files are produced. They are now assembled from several source files. Pros: - still only two files as build dependency - independent code is kept in different files - smaller files are easier to navigate - diffs are easier to maintain and review It might be a good idea for chicken to assemble runtime.c and chicken.h from a subdirectory runtime of .c and .h files, each concerned with a particular topic like gc, signal, equal, numeric, time, string, ffi, ... This would make it much simpler to avoid too many overlapping differences. Cheers /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] Language shoot-out: chicken vs. chicken-boot ends 1:2
I'm surprised: yesterday I posted this trivial diff http://lists.nongnu.org/archive/html/chicken-hackers/2011-10/msg00119.html after I found that this would double the speed of the chicken compiler. (the chicken compiler run itself, not the whole build process with gcc et al (since not effected); also other software will not benefit as much as the compiler run: for my prog for instance I found at most 25% of a difference) Now I spent about an hour to try to create a chicken-boot executable, which would perform as the installed chicken. No way!: $ time ./chicken-boot irregex.scm -optimize-level 2 -include-path . -include-path ./ -inline -ignore-repository -feature chicken-bootstrap -no-warnings -explicit-use -no-trace -output-file irregex.c real0m19.479s user0m19.220s sys 0m0.170s $ time chicken irregex.scm -optimize-level 2 -include-path . -include-path ./ -inline -ignore-repository -feature chicken-bootstrap -no-warnings -explicit-use -no-trace -output-file irregex.c real0m10.189s user0m9.980s sys 0m0.170s What's the difference? (I recompiled the installed chicken even twice uninstalling it in between.) The 19 second figure btw is within the measuring error to the 21 seconds as git master would need on my machine. Which means there is (probably just)one important difference in those two compilers. I'm really curious to learn about it! Cheers /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] On Hash Collisions (28C3)
Hi all, I never thought about that one, but it could be a real issue. Julian Wälde and Alexander Klink reporting: http://www.nruns.com/_downloads/advisory28122011.pdf I have not figured whether or not chicken would be vulnerable. Anybody able to do so? Best regards /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] On Hash Collisions (28C3)
On Dec 31 2011, John Cowan wrote: Thomas Bushnell, BSG scripsit: Huh? The point is that well-chosen hash collisions can force the algorithm into its worst case behavior, and if that's linear, it's a problem. Choosing a linear algorithm to begin with is hardly a win! Let me second that one. The point is _not_ about replacing hash tables with something else (especially not something of higher complexity cost; let alone the need to change whatever program that would be) elsewhere when compiled by chicken. The point is hash table implementation in chicken. If the default hash function is vulnerable I'd like at least to know about it. Once I know, I'll estimate the cost to fix it to save my brain from remebering that I'm not supposed to use hash tables where appropriate because they could degenerate into an issue. But it's worst-case as well as best-case linear. From the abstract: # If the language does not provide a randomized hash function or the # application server does not recognize attacks using multi-collisions, # an attacker can degenerate the hash table by sending lots of colliding # keys. The algorithmic complexity of inserting n elements into the table # then goes to O(n**2), making it possible to exhaust hours of CPU time # using a single HTTP request. Inserting n keys into an a-list can never take more than O(n) time. At the cost of lookup time - not exactly a big win. Limiting the number of parameters seems like a good idea too. That might be a point - but that should be application specific. I'd really dislike, if there where an arbitrary limit to the amount of keys a hash table can hold. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] On Hash Collisions (28C3)
On Jan 2 2012, Ivan Raikov wrote: I also do not understand why using different data structures is not under consideration. This might be a matter of taste wrt. maintaining software. I prefer to choose data structures for their merits. When it turns out that the _implementation_ is (kind of) buggy, I aim at fixing it once at for all uses and users. Rather than me and all going out and reverse a well made decision (here: to use a hash table); all must change a lot of code (i.e. create more bugs). /Jörg (And don't forget that there is still the symbol table, which should eventually considered too. Going to be hard to avoid that one in your program ;-) ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] SRFI-69 compatibility problem and hash table segfaults
On Jan 7 2012, Peter Bex wrote: The thought just popped into my head that my hash table patch breaks SRFI-69 compatibility. When the user passes a custom hash procedure, That would be bad bad. In the end. For fixing the extra argument problem, here are some options I considered: 1) Pass the randomization value by a SRFI-49 parameter. ... Of course, all approaches have disadvantages. 1 seems the least invasive and error-prone but it's just ugly. IMHO too guly. 2) Get rid of randomization-per-hashtable and have one global randomization value instead, like low-level hash tables. Symbol tables could still have per-table randomization. Users must take care of randomization themselves. Haven't had time to read things up. A question: is randomization the only option to fix the issue? To my understanding (read guesswork) additional randomisation would fix any vulnerable hash function. But aren't there any good hash functions to begin with? If we could have non-vulnerable hash functions for the defaults and those hash-* functions as exported, then I'd feel much easier with the next topic 3) Assume the user knows what they're doing when passing a custom hash function and let them handle the randomization themselves. This will mean when re-using existing core hash procedures the user must remember to pass their own value to that procedure. If the user wants per-hash-table randomization they must take care of this themselves by passing a different closure for each table. given fixed defaults I'd vote for (3): 3 means that in some cases the user might want to use the chicken-supplied ones but needs to remember to use the randomization factor. 3 is of course the easiest since it basically just punts on the issue and lets the user sort it out (but only when passing a custom procedure!). I'd assume that whoever passes a custom procedure either re-uses the default hash-* functions on some value selector or really, really knows what to do. For the latter a footnote in the manual to remind them on the issue would do even better then a well meant solution which doesn't fit their needs perfectly. make use of the hash table's randomization value, this becomes difficult because then when you use hash-table-copy the first hash table stays around in the closure and never gets GCed. Such a memory leak will soon show up and spoil the fun. Sorry for the late reply. /Jerry ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] Flow-analysis enhancement for assigned variables
On Jan 30 2012, Peter Bex wrote: On Mon, Jan 30, 2012 at 11:49:15AM +0100, Jörg F. Wittenberger wrote: I'm asking this because I'm trying for several weeks to track down a certain segfault. As it happens I can observe that one only on ARM, never on AMD64. It occurs randomly and rare enough to make a full call trace a no-go. (Still often enough to be grave.) Are you running a recent enough Chicken? Jim recently fixed a one-off error in the allocation of symbol memory which rarely get triggered. Possibly you're seeing the effects of this bug, so it might help to update if you aren't running this fix yet. See changeset 4685d89153353be2175387cd74204fc246ec Yes I do. This fix was my first hope, but did not change a bit. /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] Flow-analysis enhancement for assigned variables
On Jan 30 2012, Felix wrote: If they're not tracked, could this cause the scrutinizer to make invalid type inferences and code replacements? May I expand: Felix, assuming you have at least seen the results of some mistaken type tag, what would be the resulting effect to the running chicken program? I don't understand completely: do you mean an incorrect specialization done by the compiler? In that case necessary type checks may be omitted, resulting in incorrect data references (and thus crashes, most likely). Exactly. My bet was that at some point some incorrect specialization would have crashed some of your code. I guess something damages a type tag, subsequently having the gc scan and fail on arbitrary addresses. However I lack a theory how that would be possible. The type-tag is in the header, the first word of a data object. Off-by-one errors or modifying data beyond its true size may certainly damage the header of the object following the former object in memory. I know; just after the fact, that is sitting on the core dump, *I* have no idea how to find out what kind of object it was collecting and what whould be the object just before. This knowledge should help me to track the crash down. Just I'm missing either the tool or the knowledge how to apply and which tool. ((So far I did one thing: there are only so many spots, where my code would use C to write some memory. Those are now (a) double checked, (b) I added temporary intentional corruption there like writing off-by-one; while this is no proof, the core dump always came up elswhere.)) /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] irregex - a pathological case
I'm afraid I ran into a pathological case for irregex. For me this hangs in a tight loop: $ csi CHICKEN (c)2008-2011 The Chicken Team (c)2000-2007 Felix L. Winkelmann Version 4.7.4 (custom) linux-unix-gnu-x86-64 [ 64bit manyargs dload ptables ] compiled 2012-01-12 on ajax (Linux) #;1 (use irregex) ; loading /usr/lib/chicken/6/irregex.import.so ... ; loading library irregex ... #;2 (define m (irregex https?://(?:([^:@]+)(?::([^@]+))@)?([^:/]+)(?::([^/]+))?/(.*))) #;3 (irregex-match m https://localhost:2143/Abb8999dd38524dcc113f977d378a9ee0?_v=asql=select+count%28*%29+--+r.s+as+s%2C+t.o+as+o%0D%0A++from+rdf_range+as+r%0D%0A++left+join+rdf_tc+as+tc+on+r.o+%3D+tc.o%0D%0A++left+join+literal+as+lsubc+on+lsubc.s+%3D+tc.p+and%0D%0Alsubc.v+%3D+%27http%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23subClassOf%27%0D%0A++join+rdfnode+as+t+on+r.s+%3D+t.s+and+%28r.o+%3D+t.o+or+tc.s+%3D+t.o%29;) /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATH] Use hash table instead of flat list for lambda literals
On Feb 13 2012, Peter Bex wrote: On Mon, Feb 13, 2012 at 03:11:05PM -0700, Alan Post wrote: I do think that choice is fine, yes. Does the hash table resize itself when it gets too many entries in it? No, unfortunately these hash tables are pretty basic and too low-level to do such things. Just in case hash table turn out not to be too helpful (and not implying that balanced trees would really help) it might be helpful to have some alternative. Best regards /Jörg (Maybe someone fluent with the egg system could turn this into an egg anyway?) This file should is original name aggregate-chicken.scm http://askemos.org/A5ffdc9d14b6e2dbb899e2240ea8ea99b It implements a hashtable API to LLRB-trees. Two examples: a) LLRB tree indexed by symbols implemented as a pure data structure and b) a tree indexed by strings and updated in place with no additional allocation. The actual implementation is done by this code http://askemos.org/A532a4158b6424c2c2d2b8b5b08d32994 llrbtree.scm (In comment this contains yet one more example: and integer-index tree). The code conditionally expands either to the use of records (for the sake of safety and testing) or direct ##sys#slot references (for speed). This code is actually rather well tested. (As you might have noted I'm running quite some network related code constantly. This used to spell problems with chicken's time queue handling. That's the code, which replaced those linear lists in my chicken. (Amounts to x*10^6 threads a day installing timeouts around their i/o. The symbol and string tables are actually used to implement variable ennvironments.) ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Thrush operator
On Feb 14 2012, Moritz Heidkamp wrote: Hello Chickeneers, find attached a suggested addition to miscmacros: the thrush operator family. Apparently it stems from Raymond Smullyan's book on combinatorial logic To Mock a Mockingbird, which I haven't read (yet). I've first learned about it through Clojure and have found it quite nice for increasing the readability of chained calls. See Debasish Ghosh's post [1] about it for some background. Hi Moritz, May I suggest a possibly obvious extension (Long ago I've been playing with similar notations): I rather loved to think instead of inserting the (*single*) value of the expression into the next expression of something along the lines of physical wires: connect (pass) all the result(s) of the first expression into the next (which would then need to accept as many values as the stage before returns. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Thrush operator
On Feb 15 2012, Moritz Heidkamp wrote: Hi Jörg, Jörg F. Wittenberger joerg.wittenber...@softeyes.net writes: I rather loved to think instead of inserting the (*single*) value of the expression into the next expression of something along the lines of physical wires: connect (pass) all the result(s) of the first expression into the next (which would then need to accept as many values as the stage before returns. if I understand you correctly, this is exactly what the starred versions, `-*' and `-*', do. Or did I get you wrong? Not at all! It's been me, who missed this second form. It is the form I meant to suggest. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] See you at CeBit !
Hi Chickeners, to those of you who will be at CeBit the next days: join us at our both at the security plaza hall 12! We'll show you a pure chicken network! ( askemos.org ) Look out for the logo from softeyes.net . Looking forward to see you there! /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Clean versus pure in types.db
On Mar 12 2012, John Cowan wrote: Felix scripsit: I used pure only half-heartedly. Strictly speaking a pure function should not even throw an error, the pure meaning: this procedure will not have any effect whatsoever, regardless of arguments (so it can be removed if the result is unused). (length 42) will signal an error, so it is not pure. Frankly, if it were treated as pure I doubt if any Real World programs would be affected. While I'd bet what is located at the lower end of my spine on that. ;-) I've never had much sympathy for the CL and R6RS viewpoint that programmers should be able to count on a run-time exception being raised when they've done something silly. Just because silly programs exist. I've contributed to that pile myself. While I agree that programs SHOULD not depend on such r/t exceptions eventually they do. But your are right, John: it would be an interesting option to have a compiler switch, which would treat all clean declarations as if they where declared pure (e.g., as if things where one level stricter than known to work so far). /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] #723 ; was Re: What's left for the next release?
#723: looks weird, but looking into that will need time Hm. I'm sitting on 4.7.5 and can not confirm the crash an 64bit linux compiled with -O3. Note however, that I'm running from some runtime modifications as I posted about here: http://lists.nongnu.org/archive/html/chicken-users/2011-09/msg00209.html Maybe this is helpful. /Jerry . ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] #723 ; was Re: What's left for the next release?
On Apr 16 2012, Christian Kellermann wrote: * Jörg F. Wittenberger joerg.wittenber...@softeyes.net [120416 16:51]: #723: looks weird, but looking into that will need time Hm. I'm sitting on 4.7.5 and can not confirm the crash an 64bit linux compiled with -O3. Does it work with -O2 also? Yes. At least it ran for 1074490 iterations compiled at -O2. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] poll(2) with chicken: works - sort of
On May 14 2012, Moritz Heidkamp wrote:
Jörg F. Wittenberger joerg.wittenber...@softeyes.net writes:
Anybody interested to pull this into the official chicken core?
That's definitely interesting. I wonder if it would be possible to
provide hooks in the core so that the scheduler can be replaced from
user code?
I'd really like it to be that way. Personally I'm not a big fan
of srfi-18 style threads. In practice I use an IMHO more Schemish
set of threading primitives[1] (build for the sake of being standards
based on top of s18, but I'm sure I could do them much better
at the level of switching continuation.
Funny enough: here the title of my master thesis ('93):
process scheduling at the application layer.
That way it would be possible to use even more efficient OS
specific APIs like epoll. Or perhaps the whole issue could be
side-stepped by using something like libev instead?
These would be the interesting consequences of having an
stable API and no more in the core. (I would not love to
have an general dependency on libev for chicken.)
We could have a bunch of srfi-18 (as is, modified,
build on libev or it's alternatives) plus some alternate
thread interfaces.
best regards
/Jerry
[1]: I'm using e.g. !apply, !map and !mapfold where
!apply would create and immediately join (possibly with
timeout) a thread to apply the function to it's arguments.
!map does what map does, just one thread per element in paralles
and !mapfold accumulates the results in unspecified order
(as available) possibly only waiting for an specified number of
map results to become available.
.
___
Chicken-hackers mailing list
Chicken-hackers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] poll(2) with chicken: works - sort of
To summarize: at least the select interface should be kept around. Either as cond-expand compile time choice or via modules to load. Also I wonder how broadly applicable the claims of performance gains actually are and how much is just folklore. Assuming that often enough the implementation of one interface will just relay on the other one, I'd guess mileage may vary from system to system. /Jerry On May 14 2012, John Cowan wrote: Christian Kellermann scripsit: Also a question to others: Do we have a supported system that is know for not providing poll()? Haiku's network documentation is nonexistent, but BeOS definitely did not document poll() if it supported it at all. Someone close to the ground needs to check. Win32 has no poll(), so it has to be emulated using select(). http://www.readusenet.com/windows-all-win32/1131068-win32-poll-equivalent.html claims to be such an emulator; of course it inherits the problems of select(), but at least presents the correct interface. Note that Cygwin does have poll(). ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] poll(2) with chicken: works - sort of
Hi John, well the implementation is messy and the API as I mentioned is the intended idea. I'll attach the messy code. But when reading it, please swap !map for map-parallel - each of them does what the other should. Just it's one more change to my code base to be done I'm still experimenting with how exactly parameters like timeouts should be passed to !map for instance. Here an usage example: (lambda (source quorum local target) (!mapfold ;; map this for each host (lambda (voter) (let ((addr ((host-lookup) voter))) (if voter (let ((reply (http-get-privileges source addr local target))) (parse-capability-statement (message-body reply))) (raise-service-unavailable-condition voter ;; the KONS: combine the result and i; ;; by modifying n, the accumulation process could be terminated ;; earlier (it's the number of pending elements) (lambda (n result i) (values n (fold (lambda (e i1) (if (member e i) i1 (cons e i1))) i result))) ;; the initial value for the FOLD operation '() ;; the input (list) to map over quorum ;; count #t means: count all or up to timeout count: #t timeout: (respond-timeout-interval) )) On May 14 2012, John Cowan wrote: Jörg F. Wittenberger scripsit: [1]: I'm using e.g. !apply, !map and !mapfold where !apply would create and immediately join (possibly with timeout) a thread to apply the function to it's arguments. !map does what map does, just one thread per element in paralles and !mapfold accumulates the results in unspecified order (as available) possibly only waiting for an specified number of map results to become available. Can you provide more details? This looks like a very interesting API indeed. Thanks. ;; (C) 2008, 2009, 2010 (cond-expand ((or rscheme) (define (delay*thunk thunk name) (let ((mutex (make-mutex name))) (lazy (let ((tmp mutex) (result #f)) (dynamic-wind (lambda () (mutex-lock! tmp)) (lambda () (if mutex (begin (set! result (call-with-values thunk eager)) (set! mutex #f))) result) (lambda () (mutex-unlock! tmp))) (define-macro (delay* expr name) `(delay*thunk (lambda () ,expr) name))) (else (define (delay*thunk thunk name) (let ((mutex (make-mutex (cons name '*single-use* (lazy (make-eager-promise (let ((m mutex) (result #f)) (dynamic-wind (lambda () (mutex-lock! m)) (lambda () (if mutex (begin (set! result (call-with-values thunk list)) (set! mutex #f))) result) (lambda () (mutex-unlock! m))) ;; (let ((t (make-thread thunk name))) ;; (delay (begin ;; (if (eq? (thread-state t) 'created) ;; (handle-exceptions ex #f (thread-start! t))) ;; (thread-join! t ) (define-syntax delay* (syntax-rules () ((_ expr name) (delay*thunk (lambda () expr) name)) (define (!%call-with-values timeout abort-handler thunk receiver) (let* ((thread (thread-start! (make-thread thunk !call-with-values))) (alive (and abort-handler (let ((parent (current-thread))) (thread-start! (make-thread (lambda () (guard (ex (else #f)) (thread-join! parent)) (abort-handler thread)) !call-with-values-alive)) (call-with-values (cond (timeout (lambda () (let ((pending #f)) (dynamic-wind (lambda () (set! pending (register-timeout-message! timeout (current-thread (if abort-handler (lambda () (guard (ex ((timeout-object? ex) (abort-handler thread) (raise ex))) (thread-join! thread))) (lambda () (thread-join! thread))) (lambda () (cancel-timeout-message! pending) (if alive (thread-terminate! alive))) (abort-handler (lambda () (dynamic-wind (lambda () #t) (lambda () (thread-join! thread)) (lambda () (if alive (thread-terminate! alive)) (else (lambda () (thread-join! thread receiver))) (define (!call-with-values thunk receiver . rest) (let ((timeout (if (pair? rest) (car rest) #f)) (abort-handler (if (pair? rest) (let ((rest (cdr rest))) (if (pair? rest) (car rest) thread-signal-timeout!)) thread-signal-timeout!))) (if (or timeout abort-handler) (!%call-with-values timeout abort-handler thunk receiver) (call-with-values thunk receiver (cond-expand ((or rscheme) (define (!order/thunk thunk name) (let ((thread (thread-start! (make-thread thunk (if (string? name) name (format ~a name)) (lazy (call-with-values (lambda () (guard (ex ((uncaught-exception? ex) (raise (uncaught-exception-reason ex (thread-join! thread))) eager) (else (define (!order/thunk thunk name) (let ((thread (thread
[Chicken-hackers] tcp-connect looses socket-fd upon timeout
this should help: === --- tcp.scm +++ tcp.scm @@ -612,10 +612,11 @@ ##sys#current-thread (+ (current-milliseconds) tmc) ) ) (##sys#thread-block-for-i/o! ##sys#current-thread s #:all) (yield) (when (##sys#slot ##sys#current-thread 13) +(##net#close s) (##sys#signal-hook #:network-timeout-error 'tcp-connect connect operation timed out tmc s) ) (loop2) ) ) )) ... ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] Invalid forwarded object howto?
Please could somebody enlighten me, what those forwarded objects are? A short note on purpose and encoding should be enough. The real question would be: how can it happen, that I can create an invalid forwarded object - or for that matter - how would I avoid it? A different though related question: I found that when member tests against a list, which turns out to be an invalid forwarded object when printed, it may segfault. Thanks for any hint! /Jerry .. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] howto interpret gdb's stacktrace
Thanks Felix, On May 28 2012, Felix wrote: Should I doubt the trace? Was this code compiled with C optimization options? If yes, any assumption about the correctness of the trace is moot. It was compiled with -O3 . Case closed. (Actually I'm not a big user of debuggers: does this no optimizations or forget the trace hold for all C programms?) /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] catch exceptions in finalizers, remove dynamic resizing of finalizer vector
On Jun 19 2012, Felix wrote: The attached patch adds exception handling around the invocation of finalizers, which are shown as warnings (unless warnings are disabled) but do not otherwise trigger errors (similar to the way errors in separate threads are handled). I still experienced random crashes when running the stress test in #866. What helped was removing the dynamic This maybe related, but maybe not at all. To me it looks as a repeated occurrence of this problem: http://lists.nongnu.org/archive/html/chicken-users/2008-09/msg00025.html However my wild guess would be that the static array for finalizers is a red herring. It ran quite comfortable once finalizers where wraped by an exception handler. However I too see random crashes all day. To me it looks more like a gc problem (wild guess work again). I've been able to produce *some* binaries, usually just minor code changes - that is minor wrt. Scheme - which would crash sooner or later or exhibit another problem: The same binary would start up different. As early as during the initialization of all modules it select one of two modi: (A): run normal for minutes or hour until switch to (B) and (B): eat up at least 512kB mutation stack, slow down to a grind and repeat resizing the mutation stack. (at this point my session freezes; I never managed to be fast enough to stop and attach to the process) My most recent update to chicken trunk made this come up so frequetly that I started these days to experiment with a workaround. I introduced a mutation_stack_limit and forced a gc whenever the mutation_stack has more than a constant (currently I'm using 1024) entries. This made sure it never enters mode (B). At the cost of: # define C_stack_probe(p) (C_stress ((C_word *)(p) C_stack_limit) !C_mutation_limit) you see: binary incompatible, runtime.c exports yet another C_word* and each C_stack_check involves yet another memory access. (((I've been wondering: this C_mutation_limit is actually a boolean. Maybe I could fold it into the LSB the the C_stack_limit to avoid the extra memory access.))) This cost is so far bearable, given that I can continue to run my older code as it was. Coming back to maybe related: It also went to the benefit to reduce the felt frequency of those nasty segfaults… …if only I could say, that is killed them. It did not. That's what makes me believe the static array might be read herring. Hope this helps. /Jerry ... ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] catch exceptions in finalizers, remove dynamic resizing of finalizer vector
On Jun 19 2012, Peter Bex wrote: What I don't quite understand is all this talk about threads. The test program doesn't even create any extra threads, so how could there be any race conditions? There shouldn't *be* any concurrency in this case, should there? Isn't this more of a text-book-example of an issue? By definition a finalizer is a unit of execution to be scheduled to run when a certain condition is met. Letting alone discussions which data structure is called a thread: this requires some, possibly minimal scheduling. Hence worry about concurrency. BTW: there are many more such issues. E.g. delay. The promise is supposed to be computed just once. Try: (use srfi-18) (define foo (let ((called 0)) (delay (begin (set! called (+ called 1)) (thread-sleep! 10) called (do ((i 0 (+ i 1))) ((= i 5)) (thread-start! (lambda () (force foo (force foo) To summarize: I subscribe to Felix view that it would be great to have the core system thread free. However I'm honestly unsure that this is at all possible, given that we have threads masquerading under the name of promises and finalizers, which inherently require some scheduling. However I'd welcome to expel the scheduler from the base system and instead have some simpler API, where I can register a continuation to wait for events from files, time, signals other threads and other resources. The current scheduler as well as fictionally once based on zmq as suggested by whom on the list the other day should have an easy time to hook into this API. --- Quite a project IMHO. Best Regards /Jerry . ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] catch exceptions in finalizers, remove dynamic resizing of finalizer vector
On Jun 19 2012, Felix wrote: each object. The latter takes up more memory when you don't have a lot of finalizers (and finalizers are slow, so it's best not to generate too many of them). Exactly. Finalizers are junk. Finalizers are the last resort. Creating excessively many of them is a design error. (That was just for the record) Anecdotal evidence: I already began to convert my code, killing finalizers the other day. If I only knew a recipe to get around them, I'd vote to not have them at all. /Jerry ... ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] catch exceptions in finalizers, remove dynamic resizing of finalizer vector
On Jun 19 2012, Peter Bex wrote: Right now it looks to me like there's an optimization that's messing things up. Try compiling this file with -O0 and then with -O1: (let lp ((x (list 1 2 3))) (set-finalizer! x (lambda _ #f)) (lp (list 1 2 3))) With -O1 or higher this will raise a suspicious looking error: Error: call of non-procedure: #unspecified I also see this with 4.7.0, even when compiled with -O0. With -O1 on 4.7.0, it shows this error and _then_ craps out with an OOM error. This would kinda explain, why my eat all memory issue is so apparent on the recent mint install I did and rare on older systems. This system has gcc (Ubuntu/Linaro 4.6.1-9ubuntu3) 4.6.1 it is the first one to complain about chicken code for me: atomic.c: In function 'f_5446': atomic.c:5062:4: warning: array subscript is above array bounds [-Warray-bounds] atomic.c: In function 'f_3797': atomic.c:1534:4: warning: array subscript is above array bounds [-Warray-bounds] atomic.c: In function 'f_3760': atomic.c:7751:4: warning: array subscript is above array bounds [-Warray-bounds] atomic.c: In function 'f_3733': atomic.c:8312:4: warning: array subscript is above array bounds [-Warray-bounds] atomic.c: In function 'f_3791': Note that I'm seen the problem on code, which is compiled with -scrutinize and gcc -O2. This *might* hint somewhere…? Anybody running Clang? /Jerry . ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Please vote for CR #931, #932
On Oct 15 2012, Felix wrote: Hi! Since these two CRs don't seem to be overly controversial, we can probably vote on it right now: http://bugs.call-cc.org/ticket/931 Dislike (requires me to double check that I stick to standard; while I really like it when I see additional dependencies spelled out in code I never wrote or at least forgot about.) http://bugs.call-cc.org/ticket/932 Favorable. cheers, felix ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] process-fork: option to kill all but the current thread in the child process
On Oct 28 2012, Felix wrote: From: Felix fe...@call-with-current-continuation.org Subject: [PATCH] process-fork: option to kill all but the current thread in the child process Date: Sun, 28 Oct 2012 09:10:41 -0400 (EDT) This patch allows process-fork to kill all existing threads but the current one when running the child process. Just reviewed that one. It's going to do MOST of the job. However users still ought to be vary: Threads blocked on mutex's, mailboxes etc. will not be killed. This may be a problem if unblocked later on. Unfortunately I don't have a good suggestion how this could be handled. Except if we had a weak referencing table of all threads which haven't terminated. /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] PS: which one
Sorry, fat fingers at the phone. Wanted to mail the next line in my addressbook. On Dec 28 2012, Jörg F. Wittenberger wrote: I'd love to get this done by sunday. There*s been a job offer at the mailing list of the chicken compiler. I intent to apply. By Monday. Sending only the reference to the chicken mailing list archive and askemos.org along with my name. ... ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] A couple of questions
# Askemos – What is it? I keep receiving positive comments to Askemos/BALL-related code posted to Chicken lists. (Some examples below.) However people seem almost scared to use it for their own good. This code base became what I'd call a swiss-army knife to **simplify** tasks like backup, version control, content management and (web)development. -- I'm afraid there's a prevalent documentation issue. Given that we're talking about quite some man-years of development and testing by our team, the scaring appears less a surprise. To those even remotely interested: *please* bomb me with your questions, requirements and interests (private replies - don't spam the lists). I want to improve the docs, most needed info first. I want to serve your needs, not what I imagine your needs would be. Find some features/keywords I guess you might love for inspiration after the example comments. But: ask for anything you'd like to see answered. Thanks for your time. /Jörg ## Comment Examples On 25 Jan 2009, Alaric Snell-Pym wrote relating to proposed llrbtree I wrote for chickens scheduler: Yes! I've been impressed with Jorg's work there, even though it won't directly affect me since I'm not doing anything highly multithreaded. (see http://lists.nongnu.org/archive/html/chicken-users/2009-01/msg00104.html ) On Oct 17th 2011 David-Sarah Hopwood of tahoe-lafs fame wrote: Tahoe-LAFS is solving a much simpler problem than … Askemos On 13 Jan 2013 Ian Grigg (iang.org) wrote: I'm actually unclear on what Askemos is… (later - found this http://ball.askemos.org/A26b5619be8d5e3348cca356acfc8efea/ ) http://ball.askemos.org/A26b5619be8d5e3348cca356acfc8efea/DevelopmentNetwork Wait -- I see an identifier. It leads to the same document on multiple sites. How does it do that? Are the sites sharing multiple copies, and keeping track in the background? Or accessing a common database? How do the sites keep track of changes? (No common database. Active replication tolerating byzantine faults. Updates while 2/3rd of nodes are avail.) On Jan 25 2013, Daniel Leslie wrote: I keep running across amazingly useful code snippets from you in the chicken-users list archive … examples: … http://lists.nongnu.org/archive/html/chicken-users/2010-12/msg00218.html (A pthread-interface to run blocking C code without blocking the Chicken thread. Plus a sqlite-interface using the former. The latter uses sqlite's virtual file system to retrieve data blocks via Chicken-Scheme again. This is the code used in turn for fault-tolerant replication of the database(s) itself. BTW: Does anybody know another (No~ or)SQL database with simple, peer-to-peer replication protecting against byzantine faults?) On Jan 21 2013, Matt Gushee wrote: I wanted to say that I share Daniel's thoughts about Askemos. It seems like a really intriguing project, and I would like to learn more about it at some point. For the moment, my concerns are a bit more mundane: I want to create a CMS that is broadly similar to existing products such as Drupal, but with better performance and usability ... and that is usable in conventional web hosting environments. Question: What means something to be usable in a conventional web hosting environment? No chance to run a binary to serve requests? ## Features - What's important to you? If you have time enough to dig deeper: askemos.org is about designrequirements ball.askemos.org about the actual software. ### Permission Control * No omni-privileged root/administrator * Unlimited many, structured capabilities (algebraic) * Similar to http://www.iang.org/papers/ricardian_contract.html but applicable to running processes instead of static contracts * legal opinion finds it superior to PKI. PDF (German): http://www.askemos.org/A0e80fdd97a7b6e7af87c5d294f39a96c ### License * With RScheme: GPL (contracted over several years) * With Chicken: BSD (tentative, some GPLed code snippets still not replaced by BSD equivalents) -- (and unfortunately still not running stable for days without restart as the RScheme version does) ### Networking * SSL implementation as plugin (free the Scheme thread) * encryption/auth plugin also to ease use of alternatives (GNUnet?) * SOCKS4a for use with Tor * Reverse use of HTTP to ease tunneling firewalls * Updates DNS dynamically * persistent connections forsake of speed ### MIME handling * See: http://ball.askemos.org/?template=htmldoc PDF on-the-fly (datetime of retrieval at the bottom of the page) * Defaults to XML for stored content (to avoid forcing a messy wiki-syntax onto users) even though applications may default to other syntax for edits. Compare: http://askemos.org/A74c3f2644819d5eb04fa18af4cb3972d?_v=spo_id=1623 - markdown, the default http://askemos.org/A74c3f2644819d5eb04fa18af4cb3972d?_contenttype=text%2fxml_v=spo_id=1623 - XML, the internal
Re: [Chicken-hackers] [Chicken-users] using types
On Feb 21 2013, Jörg F. Wittenberger wrote:
Somehow I can't verify that my type declarations are actually
effective.
I've been able to verify that my .types files are not ever consulted.
Using strace I found that the foo.types file is searched for in
stat(/usr/lib/chicken/6/foo.types, 0x7fffc1895400) = -1 ENOENT (No such
file or directory)
But since that's not an extension but just a simple module, which is
part of a larger program, there is no good reason to ever install
the .types file in the repository.
To deal with the issue I tried to use -setup-mode. No effect at all.
Further it seem to be the case that CHICKEN_REPOSITORY must be s imple
directory and may not be some colon-separated path.
At this moment I don't see any solution how to make chicken read
the .types files it produced *within* a single directory.
How would I do that?
Thanks a lot!
/Jörg
So far I noticed that when I make usage-error wrt. types I
- within a single file that is -
get an error in tje compilation step.. Which is what I like.
Now that I've got a .types file for each module, I tried to
enforce this situation by violating the declared procedures
signature. But it did *not* give me the expected error.
Maybe I'm doing something wrong?
So far I'm relying on these lines from manual/Types:
If library code is used with {{require-extension}} or {{(declare (unit
...))}} and a {{.types}} file of the same name exists in the
extension repository path, then it is automatically consulted. This
allows code using these libraries to take advantage of type-information
for library definitions.
I have a file foo.types, which declares
(: bar (fixnum) - . *)
in foobar.scm I have
(declare
(unit foobar)
(uses foo)
...)
but the attempt to provoke an error by calling
(bar some-fixnum-which-is-from-working-code-anyway 27)
-- which clearly violates the given signature above giving
the additional parameter 27
(while the signature in turn was verified to be correct in foo.types)
I did *not* result in any warning, let alone the stop-compile-on-error
I hoped to receive.
I'd like to add that prior to this experiment I went without
the (declare ... (uses foo) ...) having only an import statement
(import foo) in my foobar.scm - while this would be an analogous
use to the declare-variant it would be undocumented AFAI can see.
Now I dunno how I can make (sure) use of my type declarations
short of manually adding a -types foo.types entry for each an
every import in the scheme source to my Makefile.
Any help much apprecuated.
/Jörg
___
Chicken-users mailing list
chicken-us...@nongnu.org
https://lists.nongnu.org/mailman/listinfo/chicken-u
___
Chicken-hackers mailing list
Chicken-hackers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] mutex-lock! in types.db
it appears that the mutex-lock! declaration in types.db is not currect. The second optional argument according to my reading of may be either a thread (struct) or #f ; however types.db only expects a thread. best /Jörg ... ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] Whishlist entry: macro expansion for type declarations.
Hi all, maybe that's not a wishlist entry, but just me missing somthing? When adding type declarations to some source, which has grown for over a decade, I ran into this situation: There's are widespread use cases of some types, which are actually implemented as vectors, pairs or lists with a certain substructure. The original idea to add types to this otherwise working (and relied upon) program was to simplify the structure. Thus I don't just want to add the type as is happens to be using a long wielded construct like (or boolean (pair (struct whatever) (pair symbol maybesomthing))) which would be what I'd have to write next dozends of times. Once I'd be through this, I'd like to replace the (pair ...) stuff and selectively remove the boolean alternative. This is so error-prone that I don't dare! My great idea was, to write a handful of templates using define-syntax and then use these where appropriate. This would allow me to change these as desired and then fix all the resulting warnings ((second wishlist entry: have a compiler switch to turn these warnings into errors)) accordingly. So I wrote (define-syntax :foo-type-or-not: (syntax-rules () ((_) (or boolean (pair (struct whatever) (pair symbol maybesomthing)) and used it like this (: bar (my params - (:foo-type-or-not:))) But: chicken complains that the (:foo-type-or-not:) macro I defined would be illegal in the type annotation. I take this as sign that the macro was not expanded at all. Correct? Any way to get this done? If not, would it be feasible to add this expansion step into chicken? Thanks a lot! /Jerry ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] 4.7.5 modified - 4.8.2: heap full while resizing is back
At 21 Oct 2011 I proposed some changes, which apparently have made it half-way into the chicken core so far. See http://lists.nongnu.org/archive/html/chicken-hackers/2011-10/msg00084.html At 19 Jun 2012 in http://lists.nongnu.org/archive/html/chicken-hackers/2012-06/msg00051.html I mentioned some more changes I made, which eventually made those heap full while resizing go away. (I kept the mutation stack under a limit.) Now with my compile upgrade I somehow killed this change (since parts of it apparently went into the core). However the problem is back. I can't run my code for any reasonable amount of time. While it runs almost forever with the modified 4.7.5 code within about 128MB of memory, it grows beyond any reasonable size in seconds again. I'll try to bring my changes back in locally. But I'll need help to prepare diffs for integration. Who's interested? Best Regards /Jörg .. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] 4.7.5 modified - 4.8.2: heap full while resizing is back
Thanks Mario for your remark. However, the bug will not effect me. Let's say I don't remember whom I eventually sent my initial implementation as offered here http://lists.nongnu.org/archive/html/chicken-hackers/2012-05/msg8.html The code I'm using since doesn't have that problem. Dunno how it came in. Thanks anyway. /Jörg On Mar 4 2013, Mario Domenech Goulart wrote: Hi Jörg, On 04 Mar 2013 13:55:01 +0100 Jörg F. Wittenberger joerg.wittenber...@softeyes.net wrote: At 21 Oct 2011 I proposed some changes, which apparently have made it half-way into the chicken core so far. See http://lists.nongnu.org/archive/html/chicken-hackers/2011-10/msg00084.html At 19 Jun 2012 in http://lists.nongnu.org/archive/html/chicken-hackers/2012-06/msg00051.html I mentioned some more changes I made, which eventually made those heap full while resizing go away. (I kept the mutation stack under a limit.) Now with my compile upgrade I somehow killed this change (since parts of it apparently went into the core). However the problem is back. I can't run my code for any reasonable amount of time. While it runs almost forever with the modified 4.7.5 code within about 128MB of memory, it grows beyond any reasonable size in seconds again. I'll try to bring my changes back in locally. But I'll need help to prepare diffs for integration. Who's interested? Beware that chickens 4.8.0 have a bug in the scheduler that affects Linux. See http://code.call-cc.org/cgi-bin/gitweb.cgi?p=chicken-core.git;a=commit;h=2a9696691c2792f981f0adfd46208d419eaad867 for more information. The fix for that issue will probably be available in 4.8.0.3 soon. Best wishes. Mario ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] ##sys#double-number gone; intetional?
On Mar 4 2013, Peter Bex wrote: On Mon, Mar 04, 2013 at 03:32:52PM +0100, Moritz Heidkamp wrote: Hi Jörg, Jörg F. Wittenberger joerg.wittenber...@softeyes.net writes: I found that ##sys#double-number is gone from library.scm . Still it's there in chicken.h . Mistake or intended? At least I'd consider it worth a news item. the ##sys# API is not even documented so I don't think it needs to be mentioned in the NEWS. In fact, AFIUI the point of identifiers having that prefix (and ##core#) is that those can be changed without considering backwards compatibility. Otherwise they could just be exported without the prefix. +1; you mess with internals, you get burned :) Thanks for that hint, helpful enough. So how would I convert from a known-to-be integer to a fixnum? BTW: Another question, how would I use chicken without messing with the internals? Cheers, Peter ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] ##sys#double-number gone; intetional?
On Mar 4 2013, Christian Kellermann wrote: * Jörg F. Wittenberger joerg.wittenber...@softeyes.net [130304 14:57]: I found that ##sys#double-number is gone from library.scm . Still it's there in chicken.h . Mistake or intended? Are you mixing up files? Or using a chicken.h from a previous build? In my version there is no trace of double-number in chicken.h (nor the rest of the repo). In my version of chicken.h (untouched from git pull) there is at line 1305 #define C_a_double_to_num(ptr, n) C_double_to_number(C_flonum(ptr, n)) ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] ##sys#double-number gone; intetional?
Thanks a lot, this one seems to do the job. On Mar 4 2013, Christian Kellermann wrote: * Jörg F. Wittenberger joerg.wittenber...@softeyes.net [130304 14:57]: I found that ##sys#double-number is gone from library.scm . Still it's there in chicken.h . Mistake or intended? At least I'd consider it worth a news item. What should I use instead? It has been removed on Oct 4th with commit 6e10dfb17175faa307a2a7230cf705c987af85c5 I guess an alternative would be to use inexact-exact? Kind regards, Christian ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] On scheduler not handling bad fd's WAS: Re: 4.7.5 modified - 4.8.2: heap full while resizing is back
Just for the record: the bad-fd's bug is pretty old. I posted a first attempt to fix it here: http://lists.nongnu.org/archive/html/chicken-users/2008-08/msg00094.html (Since these fixes never made it into the mainline, I'm sitting on a modified chicken runtime ever since. There are also a couple of other bugs and improvements I made to scheduler.scm and sefi-18.scm since. But they more or less all depend - or at least are only tested against - the modified version, since the mainline runtime was never able to run my code base [or at least it was so crawling slow that I could not stand it.) On Mar 4 2013, Jörg F. Wittenberger wrote: Thanks Mario for your remark. However, the bug will not effect me. Let's say I don't remember whom I eventually sent my initial implementation as offered here http://lists.nongnu.org/archive/html/chicken-hackers/2012-05/msg8.html The code I'm using since doesn't have that problem. Dunno how it came in. Thanks anyway. /Jörg On Mar 4 2013, Mario Domenech Goulart wrote: Hi Jörg, On 04 Mar 2013 13:55:01 +0100 Jörg F. Wittenberger joerg.wittenber...@softeyes.net wrote: At 21 Oct 2011 I proposed some changes, which apparently have made it half-way into the chicken core so far. See http://lists.nongnu.org/archive/html/chicken-hackers/2011-10/msg00084.html At 19 Jun 2012 in http://lists.nongnu.org/archive/html/chicken-hackers/2012-06/msg00051.html I mentioned some more changes I made, which eventually made those heap full while resizing go away. (I kept the mutation stack under a limit.) Now with my compile upgrade I somehow killed this change (since parts of it apparently went into the core). However the problem is back. I can't run my code for any reasonable amount of time. While it runs almost forever with the modified 4.7.5 code within about 128MB of memory, it grows beyond any reasonable size in seconds again. I'll try to bring my changes back in locally. But I'll need help to prepare diffs for integration. Who's interested? Beware that chickens 4.8.0 have a bug in the scheduler that affects Linux. See http://code.call-cc.org/cgi-bin/gitweb.cgi?p=chicken-core.git;a=commit;h=2a9696691c2792f981f0adfd46208d419eaad867 for more information. The fix for that issue will probably be available in 4.8.0.3 soon. Best wishes. Mario ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken- ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] 4.7.5 modified - 4.8.2: heap full while resizing is back
Solved again. Please tell me what to do with the fix. On Mar 4 2013, Jörg F. Wittenberger wrote: At 21 Oct 2011 I proposed some changes, which apparently have made it half-way into the chicken core so far. See http://lists.nongnu.org/archive/html/chicken-hackers/2011-10/msg00084.html At 19 Jun 2012 in http://lists.nongnu.org/archive/html/chicken-hackers/2012-06/msg00051.html I mentioned some more changes I made, which eventually made those heap full while resizing go away. (I kept the mutation stack under a limit.) Now with my compiler updade I somehow killed this change (since parts of it apparently went into the core). However the problem is back. I can't run my code for any reasonable amount of time. While it runs almost forever with the modified 4.7.5 code within about 128MB of memory, it grows beyond any reasonable size in seconds again. I'll try to bring my changes back in locally. But I'll need help to prepare diffs for integration. Who's interested? Best Regards /Jörg .. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hack ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] just to let you know
On Mar 5 2013, Dan Leslie wrote: Do you have any telemetry that might indicate why this is? Yes, in a way. And fixed, sorry for the noise: I've been using the not documented API to make-input-port. In particular the 5th (2nd optional) argument read-string. (See http://lists.nongnu.org/archive/html/chicken-users/2008-08/msg00110.html for details.) When burned with the ##sys#double-number issue *and* the changed API of ##sys#scan-buffer-line I disabled the use of the undocumented API. After re-enabling at least the read-string, I got the old performance back. /Jörg BTW: thanks for the hint to git-GUIs. (My problem however is, that git uses slightly different vocabulary than those VC systems I'm famillar with. I'd still have to make head or tail of screen space and probably read quite an amount of docs. But: these days when walking the dog, the children left a snowball of about 50cm diameter in my pathway - I fell down full length and just yesterday somebody came horse riding in front of mine - I didn't see until close as about 7 meters. Also I hade to see a doctor to cure my spine-pain because I bent forward to read the screen too long. If you see what I mean.) -Dan On 3/5/2013 11:46 AM, Jörg F. Wittenberger wrote: so far I replaced two out of five hosts running the chicken version of askemos/ball with the one compiled using chicken 4.8.2. (No changes to the prog itself.) According to the statistics so far, performance is about half of 4.7.5 . (The test-transaction is trivial, thus measuring basically i/o-troughput.) /Jörg .. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] strange error message, please help with interpretation
In my prog I have a structure (define-record-type uri ... (path uri-path) ...) --: Which happened to work for years so far. Unchanged :-- Now, after upgrading from 4.7.5 to 4.8.2 I found this in the log: bad argument type - not a structure of the required type (bad argument type - not a structure of the required type (#uri uri) uri-path) That is, not always; it's still working much more than 99% of the time. So how would I interpret this message. As far as I can see, this tells me that somehow a typecheck failed on a (struct uri) testing it to be a (struct uri) -- which would have been supposed to succeed. Am I missing something Just too strange. Thanks a lot /Jörg ... ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] strange error message, please help with interpretation
On Mar 7 2013, Peter Bex wrote: On Thu, Mar 07, 2013 at 12:44:28PM +0100, Jörg F. Wittenberger wrote: So how would I interpret this message. As far as I can see, this tells me that somehow a typecheck failed on a (struct uri) testing it to be a (struct uri) -- which would have been supposed to succeed. Am I missing something Likely you've imported uri-generic or uri-common's definition of uri-path, which causes it to break. This expects an uri struct with a 'uri tag, not a 'uri tag. That's why the message is so Hardly. No, no. Those are not at my system at all. confusing: chicken adds #... around the name of the record type tag. HTH, Peter ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] strange error message, please help with interpretation
I'm afraid I have no idea how I could boil this down to a reproducible case. I've seen it once so far in a logfile of a process, which xreates approximately 200 threads a day when communicating over WAN with about ten peers plus all those public web access (approx. since chicken does not have unique thread numbers, but rscheme does; when I run the rscheme version instead I see thos 2^6 threads; though the chicken version has a different runtime behavior and does not share 100% of the code, hence it could use a different amount of threads.) sorry /Jörg On Mar 9 2013, Christian Kellermann wrote: * Jörg F. Wittenberger joerg.wittenber...@softeyes.net [130308 23:00]: (define (make-uri scheme authority path query fragment) (%make-uri (if (string? scheme) (string-symbol scheme) scheme) authority path query fragment)) Also: the code is pulled into the module via include and otherwise shared with rscheme. -- Any use of a # character in the code would make it unreadable to rscheme. Can you provide a minimal example which fails for you? This will make the bug hunt by magnitudes easier. Thanks! Christian ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] strange error message, please help with interpretation
No, Peter, you're right. It does not crash hard enough to get a core dup. But many thanks for your help on interpreting the message. In fact assuming a stack corruption would explain it at least. Especially since I'm observing various strange error messages since I updated to chicken 4.8.2. What worries me it that these are all within age old code. In fact nothing serious has changed since about a year. (By serious I mean, nothing, which would resort to FFI or similar. And a few changes to pure r5rs scheme code should not corrupt the stack.) I still can go back to 4.7.5. This runs just fine. But that's not of any help to testing the new chicken, is it? /Jörg On Mar 10 2013, Peter Bex wrote: On Sat, Mar 09, 2013 at 01:33:50PM +0100, Christian Kellermann wrote: * Jörg F. Wittenberger joerg.wittenber...@softeyes.net [130309 12:26]: I'm afraid I have no idea how I could boil this down to a reproducible case. I've seen it once so far in a logfile of a process, which xreates approximately 200 threads a day when communicating over WAN with about ten peers plus all those public web access (approx. since chicken does not have unique thread numbers, but rscheme does; when I run the rscheme version instead I see thos 2^6 threads; though the chicken version has a different runtime behavior and does not share 100% of the code, hence it could use a different amount of threads.) I am confused. I thought you were wondering about a compilation message you get during the flow analysis of the scrutinizer. As I understood it, it's an exception's message upon an error at runtime. I think this may be an off-by-one error somewhere resulting in data corruption. I've seen similar mystifying error messages when the stack wasn't quite right; alomst everything is fine except it tells you something is wrong. You're right; if this can't be easily reproduced, we're unlikely to find a root cause. A core dump and access to the machine might help, but I'm unsure it crashes this hard. Cheers, Peter ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH 1/2] tcp: disable interrupts
On Mar 17 2013, Florian Zumbiehl wrote: Somehow, this feels to me like you are applying test driven development techniques to concurrency correctness. Are you sure that your reasoning is correct and reliable and future-proof? Are the assumptions you make about the compiler true now for sure and can that be expected to stay that way even with new optimizations, say? ... As for the robustness of the code in the face of future changes, I would submit that the current code being defective could be considered evidence that the current approach is indeed too fragile. Not only could it break inadvertently, it _did_ already break. +1 the rcp code ougth to be re-written at least for use with srfi-18 this remindes me… at one point I fixed smth in srfi-18 will have to dig it out. mutex-lock! was somewhat broken. I can now only paste my current version. (define mutex-lock! (lambda (mutex . ms-and-t) (##sys#check-structure mutex 'mutex 'mutex-lock!) (let* ([limitsup (pair? ms-and-t)] [limit (and limitsup (compute-time-limit (car ms-and-t) 'mutex-lock!))] [threadsup (fx (length ms-and-t) 1)] [thread (and threadsup (cadr ms-and-t))] ) (when thread (##sys#check-structure thread 'thread 'mutex-lock!)) (##sys#call-with-current-continuation (lambda (return) (let ([ct ##sys#current-thread]) (define (switch) (##sys#setslot mutex 3 (##sys#append (##sys#slot mutex 3) (list ct))) (##sys#schedule) ) (define (check) (when (##sys#slot mutex 4) ; abandoned (return (##sys#signal (##sys#make-structure 'condition '(abandoned-mutex-exception) (list (##sys#slot mutex 1) ) ) (define (assign) (let ((abd (##sys#slot mutex 4))) (if (and threadsup (not thread)) (begin (##sys#setislot mutex 2 #f) (##sys#setislot mutex 5 #t) ) (let* ([t (or thread ct)] [ts (##sys#slot t 3)] ) (if (or (eq? 'terminated ts) (eq? 'dead ts)) (begin (##sys#setislot mutex 2 #f) (##sys#setislot mutex 5 #f) (##sys#setislot mutex 4 #t)) (begin (##sys#setslot mutex 2 t) (##sys#setislot mutex 5 #t) (##sys#setslot t 8 (cons mutex (##sys#slot t 8))) ) ) ) ) (return (if abd (##sys#signal (##sys#make-structure 'condition '(abandoned-mutex-exception) (list (##sys#slot mutex 1 #t (dbg ct : locking mutex) (cond [(not (##sys#slot mutex 5)) (assign) ] [limit (check) (##sys#setslot ct 1 (lambda () (if (##sys#slot ct 13) ; unblocked by timeout (return #f) (begin (##sys#remove-from-timeout-list ct) (assign))) )) (##sys#thread-block-for-timeout! ct limit) (switch) ] [else (##sys#setslot ct 3 'sleeping) (##sys#setslot ct 11 mutex) (##sys#setslot ct 1 assign) (switch) ] ) ) ) ) ) ) ) Are there any problems with disabling interrupts? The potential for starvation mentioned by Felix should be quite easy to fix by scheduling manually after every write() or read() call and not just on EAGAIN/EWOULDBLOCK/EINTR, I think?! ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH 3/4] Remove ##sys#expand-home-path.
On Mar 18 2013, Felix wrote: From: Peter Bex peter@xs4all.nl Subject: Re: [Chicken-hackers] [PATCH 3/4] Remove ##sys#expand-home-path. Date: Mon, 18 Mar 2013 21:22:12 +0100 On Mon, Mar 18, 2013 at 02:03:41PM -0400, Mario Domenech Goulart wrote: Maybe I'm too paranoid? Or missing something? No, you're spot on. I think given a choice, we should always err on the side of security and adhere to the Principle Of Least Astonishment. For convenient scripting, a dwim egg could be created that does all these dangerous but convenient things. Then this is the user's responsibility and if his system gets owned it wasn't due to a chicken fuck-up. Implicitly convenient behaviour is the root of all evil. We recently had the same discussion about substring; there is no easy way to build the sane features on top of an API with bells and whistles, except by adding lots of checks all over the place, as you pointed out in your example. Building those convenience layers on top of the core, stable functionality is easily done, and can be wrapped up as an egg. I disagree. We can still try to make the core system practical, instead of a mindless API server for low-level facilities wrapped in s-expression syntax. So lets for once try to find a solution without just being polemic (well, polemics is great fun, of course, but in this case it doesn't help). I suggest keeping the posix file-operations convenience-free (they duplicate a lot of the higher-level facilities anyway), while standard procedures and core-unit file-system operations could provide ... I barely dare to say it ... ~-expansion. Why not have both? The dwim egg (or whatever) could be loaded by the user. Maybe it would be better instead to have the functionality build into the core system and have some parameter to be set/reset at the start of the program to switch it off. IMHO for things like webservers, ugarit etc. it's just too dangerous to have any type of surprising expansions anywhere. (Since even some customization code loaded at startup could break.) I, for one, was just lucky that I haven't been trapped by this so far. Cheers /Jörg ... ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] Avoid context switch during TCP errno reporting
Hi all, I'm not yet convinced that this patch will fix everything screwed up by use of the tcp implementation. The past days I wrote a replacement for my use. (A bit incomplete wrt. API compatibility to the tcp unit and thrown into a module I'm using to drive the SSL/TLS implementation I'm using for the past couple of years; hence the module contains this code too plus some SOCK4a setup code for use with tor... If anybody is interested I'll forward the code or post it here, as you guys like it.) During the development I learned that Peter is *absolutely correct* about the strange error message I needed help to interpret a few days back. (When the logged error indicated that a type test - (struct uri) it happend to read - failed while the failed type was the same as the required one.) This is obviously due to some stack corruption. The same appears to apply to several other errors I observed all sudden. Most prominent among them heap full while resizing. With the tcp replacement code, all those frequent errors are suddenly gone. The code runs as stable as before now. However I found a way to reliably trigger the problem anyway. (Just run using PLT's dns resolver code against a bind server AND close the port underneath. That is, a slightly modified version, which will re-use the tcp connection.) I'll explore this in the next days. For the time I'm not re-using the tcp connection. The code I wrote however is a major deviation from the existing tcp code internally. 1.) No timeout parameters. (At least not at the lowest level.) Why? The Askemos/BALL code implements replication of sqlite3 databases and files in a way similar to bittorrent. This type of p2p network applications is subversive. You're deal with all sorts of failures in the network, plus hostile clients. a) In such a context it's little fun to maintain the timeout at a call-by-call basis. b) You want to have all sorts of different timeouts. E.g., wait for HTTP-keep-alive time for the next request line, wait a reasonable short amount of time for the next chunk in chunked encoding, even less for the next chunk header line. c) Almost all timeouts never kick in. Thus the overhead of inserting them into the timeout queue just to remove them a fraction of a second later turns out to be expensive and a huge slowdown for the overall i/o throughput. This is even true with the scheduler improvements I posted here (or at chicken-users ?) before, which would replace the linear list for timeouts with an LLRB tree. Therefore I'm using a different timeout handling, where thimeouts are inserted into a mailbox and the entry is kept at the callsite. Instead of removing the timeout from the full list, the entry is invalidated. Once a second the timeout queue/mailbox is replaced with a fresh one and in the next run, those timeouts, which where not yet invalidated are actually made active. Rather complicated to describe, but much, much faster to execute. 2.) The lowlevel code structure is kept more akin to the way it's handled in RScheme. Because this avoids those tricks to distinguish ports by their prot-data to eventually figure the tcp-adresses out. 3.) Avoid passing DNS names to tcp-connect. It depends the obsolete (as per Linux manual at least) gethostbyname, which could block the threading for too long time. Do a DNS hostlookup instead. 4.) Don't duplicate code from library.scm ##sys#thread-yield! to yield. Use srfi-18 thread-yield! instead. Best /Jörg PS/BTW: in extras read-lin there is a local definition fixup, which is unused. On Mar 18 2013, Jim Ursetto wrote: Here's a full patch to avoid context switches screwing up the error message reported to the user, and also consolidates much of the error handling. I think this patch is sufficient because the only actual issue, as I understand it, is that under high load you will occasionally get an incorrect error message (typically, operation in progress) instead of the real error message; an exception will still fire regardless. Disabling interrupts instead is probably overkill, unless you know that won't cause hangs. Also the patch doesn't do any harm and cleans up the code a bit, so you can still apply a different fix on top of it. Jim ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] wishlist entry: distinguished values for boolean constants in type declarations
Let's explain the idea by example. There are many standard (or srfi) procedures, which take either a distinguished type or (often) a boolean value (typically #f). For instance mutex-lock: the 1st optional argument is either a timeout or #f the second is either a thread or #f. Written as type declaration this looks like this (: mutex-lock! ((struct mutex) * (or boolean (struct thread)) - boolean) Now Chicken will catch (mutex-lock! mymux #f 27); nice. But it will fail to catch (mutex-lock! mymux 27 #t). To be consistent with the handling of other distinguished objects like () (expressed as null) and the eof-of-file-object (likewise eof) it would be great, if we had true and false to restrict these beyond the rather generic boolean in those cases. Feasible? Best /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] testcase -strict-types
On Mar 24 2013, John Cowan wrote: Jörg F. Wittenberger scripsit: Note: in local procedure `doloop9', in toplevel procedure `foo#bar': (strcttps.scm:10) in procedure call to `null?', the predicate is called with an argument of type `null' and will always return true That strikes me as Just Wrong. Even if a predicate is known to always succeed, it shouldn't be impossible to call it, any more than it should be impossible to call a predicate that always returns #t on any argument. Well, if it's a predicate (as in #:pure) know to return a fixed boolean value, it should reightfully be optimized out. That's actually important in cases like and-let* and also often handy when using macro expanded code. Thus the test being optimized away looks correct to me. The problem I see is that the foobar binding is mutated within the doloop but chicken does not see it and hence believes it's still for sure the initialization value when the doloop terminates. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] testcase -strict-types
On Mar 25 2013, Moritz Heidkamp wrote: Jörg F. Wittenberger joerg.wittenber...@softeyes.net writes: Though in a way the explanation is correct. -strict-types assumes '() to be null from the initialization. Short of a way to declare the type of foobar as (list-of whatever) this fails when it's used as the initial and correct value of type (list-of whatever) with zero length. You should be able to use `the' or `assume' for that purpose. Where would I find the respective assume to be documented? (Would the syntax break standard Scheme compatibility?) What the optimizer should do is to see into the doloop and notice the ambiguous type null being refined to a list. Given that the description of -strict-types is assume variable do not change their type I think the behavior is correct. Find attached two more variants. strcttps2.scm, which convinces chicken to do the right thing, ans strcttps2.scm, which fails the other way around. Right, here you initialize the variable's type with list (or pair, I don't know what's detected) and then possible change it to null while never checking with null? on the whole list but only on the rest. Looks fine from what I can tell! ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] testcase -strict-types
On Mar 30 2013, Felix wrote: Though in a way the explanation is correct. -strict-types assumes '() to be null from the initialization. Short of a way to declare the type of foobar as (list-of whatever) this fails when it's used as the initial and correct value of type (list-of whatever) with zero length. What the optimizer should do is to see into the doloop and notice the ambiguous type null being refined to a list. ... and this is exactly what the normal type-analysis is doing. But that refinement generalizes the type of a variable to cover all uses of it, and that makes it very difficult to figure out the most specific type, in particular when the type changes during the lifetime of a variable. List-types create more problems as this example shows. So I can only keep suggesting not to use strict-types but in special situations. After I understood that the type analysis would be troubled and there's the (the type init) alternative, I'm actually ok with the situation as it is. I've got around one warning per 2000 lines of code; easily tagged with the intended type. (That is, if there where not this problem to trick the compiler into accepting the init value.) Otherwise I love it for being stricter a check. Dunno so far how much effect it actually has as an optimization. /Jörg ... ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] testcase -strict-types
On Mar 31 2013, Felix wrote: From: Peter Bex peter@xs4all.nl Subject: Re: [Chicken-hackers] testcase -strict-types Date: Sat, 30 Mar 2013 23:44:01 +0100 On Sat, Mar 30, 2013 at 11:33:37PM +0100, Felix wrote: that refinement generalizes the type of a variable to cover all uses of it, and that makes it very difficult to figure out the most specific type, in particular when the type changes during the lifetime of a variable. List-types create more problems as this example shows. So I can only keep suggesting not to use strict-types but in special situations. Would you be so kind as to explain what exactly is the purported use of strict-types? It's all rather unclear to me right now. Strict-types means you declare that variables never change their type, once a type has been inferred (or explicitly declared). There are a few places in the scrutinizer where it can make stronger assumptions about variable types during flow-analysis, which leads to potentially much better code. In fact, the improvements where the reason that the option is there in the first place - it was just a straightforward thing to do. For normal Scheme code these stronger assumptions are often not valid. But if you use Scheme as a target language for other languages (compiler-generated or by using macros extensively), it may be possible to generate code that will still be valid in strict-types mode. If you think this isn't worth the trouble of users getting confused, we can remove the option and declaration (or simply undocument it). No, please keep it! A typesafe Scheme is not a Scheme in a way anymore. But it's a better Scheme in another way! /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] how to -block?
I'm trying to figure out howto use -block optimization. Whenever I use it with any module no matter how simple it is, the resulting binary will segfault immediately. Which restrictions should I watch? Thanks a lot /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] r7rs-tasks
On Apr 4 2013, John Cowan wrote: We can ignore environment immutability in practice. OTOH I'd *love* to have it; e.g., for sandboxing. .. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] how to -block?
On Apr 5 2013, Felix wrote: From: Jörg F. Wittenberger joerg.wittenber...@softeyes.net Subject: [Chicken-hackers] how to -block? Date: 31 Mar 2013 20:51:19 +0200 I'm trying to figure out howto use -block optimization. Whenever I use it with any module no matter how simple it is, the resulting binary will segfault immediately. Which restrictions should I watch? If you get a segfault, do you compile/run code in unsafe mode? Yes. Accidentally. The uses/import consusion as it turned out. … Took me 2 full days to work that out. Maybe we should recommend in the manual to NOT ever have uses and import in the same file. By doing so, I managed to make the compiler miss unbound variables. For whatever reason those where eventually bound when accessed - at least most of the time. By cutting out all uses from all modules and accumulating them into a single link-file (having no code except for a single import of the toplevel main procedure and call it I've been able to track several unboudn variables down. (Those came from code- rearrangements done under the safe-feeling that chicken points unbound variables out anyway. You should never trust…) Now I shall go back and retry with the -block. Thanks /Jörg Block-optimization should normally not result in crashes, but will produce runtime-errors when variables are referenced that are not bound. In block mode, defined toplevel variables will not be exposed and not be visible to code outside of the file compiled in that mode. This is really the only difference to compilation in normal mode. cheers, felix ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] -block not properly mixing with -strict-types
Hi, I just noticed, that I can't mix -block with -strict-types. That is, when I -emit-type-file, the resulting file contains the type definitions ONLY if -block is NOT given. With -block it's empty (except for comments), Thus I can either get warnings for call sites violating the required types OR apply -block. I don't thing this is how it should be, is it? Thanks /Jörg .. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] -block doesn't work for me
Sorry, I'd had sent this to chicken-users - but I'm kinda sure those who could answer don't read that list ;-) find attache two trivial module files foo.scm and bar.scm to be compiled with csc -c foo.scm bar.scm . Both declare (block) optimization. Plus several foobarX.scm for X in 1-6 versions, which show various attempts to make use of those exported foo and bar. To be compiled with csc foo.o bar.o foobarX.scm -o foobar. None of them let's me ever make use of those definitions compiled with block optimization. Am I doing something wrong? Thanks a lot /Jörg .. (declare (unit foo) (block) (emit-import-library foo)) (module foo (foo) (import scheme chicken) (: foo (fixnum -- fixnum)) (define (foo x) (* x 7)) ) (declare (unit bar) (block) (emit-import-library bar)) (module bar (bar) (import scheme chicken) (: bar (fixnum -- fixnum)) (define (bar x) (+ x 0)) ) (declare (uses foo bar)) (import (prefix foo foo:)) (import (prefix bar bar:)) (define aFoo foo:foo) (define aBar bar:bar) (display (aFoo (aBar 6))) (newline) (declare (uses foo bar)) (import (prefix foo foo:)) (import (prefix bar bar:)) (define (aFoo x) (foo:foo x)) (define (aBar x) (bar:bar x)) (display (aFoo (aBar 6))) (newline) (declare (uses foo bar)) (import (prefix foo foo:)) (import (prefix bar bar:)) (display (foo:foo (bar:bar 6))) (newline) (declare (uses foo bar)) (import foo) (import bar) (display (foo (bar 6))) (newline) (declare (uses foo bar)) (module foobar (reexport foo bar)) (import foobar) (display (foo (bar 6))) (newline) (declare (uses foo bar)) (module foobar (foo bar) (import scheme) (import (prefix foo foo:) (prefix bar bar:)) (define (bar x) (bar:bar x)) (define foo foo:foo)) (import foobar) (display (foo (bar 6))) (newline) ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] -block not properly mixing with -strict-types
On Apr 11 2013, Felix wrote: From: Jörg F. Wittenberger joerg.wittenber...@softeyes.net Subject: [Chicken-hackers] -block not properly mixing with -strict-types Date: 10 Apr 2013 19:40:32 +0200 Hi, I just noticed, that I can't mix -block with -strict-types. That is, when I -emit-type-file, the resulting file contains the type definitions ONLY if -block is NOT given. With -block it's empty (except for comments), Thus I can either get warnings for call sites violating the required types OR apply -block. I don't thing this is how it should be, is it? Well, actually it is. -emit-type-file emits types for all exported toplevel variables. But -block effectively hides all toplevel bindings, so no variables get exported. Block mode seals the compilation unit, nothing inside it is visible from the outside. Hm. This means that there is only one compilation unit ever? I've got 82 modules for a single program. Would I have to merge them all into a single file if I want -block to take effect? Well, this would still not cut it. I'd still need a way to (however complicated) to make some bindings visible (read only that is) from eval. So block it not for me? Thanks /Jörg -strict-types will work with -block, but -emit-type-file will not. We should probably document this properly to make it more obvious. cheers, fel ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Issue installing sxpath
On Apr 28 2013, Felix wrote: From: Thomas Hintz t...@thintz.com Subject: [Chicken-hackers] Issue installing sxpath Date: Sun, 28 Apr 2013 09:57:55 -0700 ... reading chunks ...[panic] out of memory - heap full while resizing - execution terminated This looks like a corrupt heap. The egg is quite large - perhaps the download triggers some corner case in the networking- or I/O system. I assume chicken-install -r sxpath fails as well? Is this crash always reproducible? heap full while resizing ??? May I suggest to try this patch? (Keyword `C_mutation_limit` ) http://lists.nongnu.org/archive/html/chicken-hackers/2013-03/msg00030.html Best Regards /Jörg .. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] r7rs, force/delay
r7rs section 4.2.6 (Dynamic bindings) mentions threads and the intro to section 4 documents a connection between ``parameter``s and ``delay``. Section 4.2.5 (Delayed evaluation) does not mention threads… Thus I'm not sure if the case described here would be better clarified in r7rs or fixed (whatever fixed could mean) in chicken. The case at hand: delayed expressions are somehow to be evaluated once. Find some code attached, which shows that the result of delay'ed expressions is indeed the same, while we find a rather random number of invocations of the single promise in the whole program. Best /Jörg #!/usr//bin/env csi (use srfi-18) ;; Be a construction worker (at least that's where I learned the ;; phrase). PURPOSE: make sure some other threads *will* do their ;; jobs. (define (seem-to-be-busy! income) (thread-sleep! (+ (/ income 1.0) (let ((p0 (/ (random 1000) 1000.0))) (+ p0 (* p0 p0 p0)) (define hand 2000.0) ; low income (define bigfish 1.0) (define-syntax hurry-up! (syntax-rules () ((_ sentence) (begin sentence sentence sentence ;; An artificial version of map trying to have some time off during ;; working hours. ;; ;; Returns AS IF the promise had been force just one time BUT returns ;; only the first result, ignoring cost. (define (map-force lst) (hurry-up! (for-each (lambda (element) (let ((t (make-thread (lambda () (force element) (thread-start! t))) lst)) (seem-to-be-busy! bigfish) (map force lst)) ;; Citing R7RS section 4.2.5: try to illustrate the property that ;; only one value is computed for a promise, no matter how many times ;; it is forced illustrate the property that only one value is ;; computed for a promise, no matter how many times it is forced (define count 0) (define workload-mux (make-mutex)) (define workload '()) ;; A single promise to test. (define p (delay (let ((n (random 1000))) (seem-to-be-busy! hand) (begin (mutex-lock! workload-mux) (set! workload (cons n workload)) (mutex-unlock! workload-mux)) n))) (define input (list p p p p p p)) (define output (map-force input)) ;; Now we find: (format #t \nAs expected - only one (the fastest) result ~a times:\n~a\n (length input) output) ;; However it's (format #t \nSuprising what was actually forced:\n from ~a promises we get ~a invocations\n (length input) (length workload)) (display workload) (display \n\nYou might need to rerun the test to find anything between 1 and MORE THAN the number of time the promise was forced here.\n) (exit 0) ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] clarification of multiple evaluation of promises
Peter Bex has pointed out that my example in http://lists.nongnu.org/archive/html/chicken-hackers/2013-05/msg00031.html was not very clear about the problem at hand; let my clarify. I wanted to show that we can proof the body of `(delay expr)` being evaluated any random number of times. Depending on circumstances. The code as posted includes only a single promise: ;; A single promise to test. (define p (delay (let ((n (random 1000))) (seem-to-be-busy! hand) (begin (mutex-lock! workload-mux) (set! workload (cons n workload)) (mutex-unlock! workload-mux)) n))) Reduced to the actual work it does, this would be (define p (delay (random 1000))) The `seem-to-be-busy!` is there only to simulate a real computation to require wall-clock time and incur thread switched in between, while the *workload*-related code underneath is there only for bookkeeping to do the accounting of what's actually going on. If there was no `seem-to-be-busy!`, the test would likely evaluate the promise just once, as RxRS suggests as the lord's intention. The value of `output` as reported will always *pretend* compliance with the single-evaluation property of promises by returning the *fastest* result seen - in each position of the list. Playing with the constant numbers and the number of repetitions caused by `hurry-up!` and `(define input (list p p p p p p))` it is easy to find any random amount of elements in the final value of `workload` between one (as in the promise p was evaluated just once and the result properly cached) and n+1 with n being m*o where m would be the number of repetitions in `hurry-up!` and o the repetitions of p in the definition of `input`. NB: Things would be even stranger, if we would cause the promise p to raise an exception for most results of (random 1000). Thus `delay` and `force` behave as my intuition would suggest *only* if `force` is called from the same thread, which produced the promise using `delay`. Best /Jörg BTW: To those concerned with r7rs: I really don't like that the draft still leaves it undefined what the result of (force promise) is, if the promise returns multiple values. force/delay are not that complicated to implement. The way the r7rs draft is now, one would still need to duplicate the definition in applications if one wanted to be sure that multiple values are ok. .. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] clarification of multiple evaluation of promises
On May 13 2013, Jörg F. Wittenberger wrote: Peter Bex has pointed out that my example in http://lists.nongnu.org/archive/html/chicken-hackers/2013-05/msg00031.html was not very clear about the problem at hand; let my clarify. ... NB: Things would be even stranger, if we would cause the promise p to raise an exception for most results of (random 1000). Thus `delay` and `force` behave as my intuition would suggest *only* if `force` is called from the same thread, which produced the promise using `delay`. Attached a test case, which shows that promises raising exceptions are also evaluated more than once, even when there is only the primordial thread involved. According to my understanding this clearly violates at least the r7rs draft. Best /Jörg ...(define p (delay (error (random 1000 (define (force/catch p) (handle-exceptions ex (with-output-to-string (lambda () (print-error-message ex))) (force p))) (define input (list p p p p)) (display (map force/catch input)) (newline) (exit 0) ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] plain lambdas as syntax transformers
On May 13 2013, Peter Bex wrote: On Mon, May 13, 2013 at 10:44:53AM +0200, Felix wrote: Hello! Hi! I know this is going to be controversial, but I'd like to un-deprecate the use of plain procedures as syntax-transformers. The way it is currently implemented, using a procedure can be seen as a simple default (er-transformer). I find the use of transformer-constructors clutters up the code, adds unnecessary typing and indentation, and is more or less meaningless for newbies. I'm a bit undecided about this. I think you're right in that the additional typing is a bit pointless, considering ER transformers are the native system. However, it's confusing to newbies, you can't really explain what exactly er-transformer *does*. Also, the extra typing isn't that bad and I'm not sure we should be subtly encouraging the use of ER transformers over IR transformers by making them less work to type (IR transformers are safer unless you know exactly what you're doing). While I'm currently struggling and failing (at seen on chicken-users) to replace something easily done in the unhygienic way with some portable and hygienic code... I'd like to second Peters argument here. One more identifier and indent level is not that bad. (It could even be avoided simply by a macro instead of the lambda keyword.) But the subtle damage it does to the newbies brain when reading existing code to learn… that's bad. Finally, I find that I don't tend to use that many macros at all, and if I use macros, they're usually syntax-rules unless I need programmatic generation of code, or if I need to break hygiene. And then I tend to prefer IR transformers because they require less (r 'foo) nonsense. So er-macro-transformer is not something I use often, therefore I don't see much benefit in making it less work to type them in. Cheers, Peter ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] plain lambdas as syntax transformers
On May 13 2013, John Cowan wrote: Felix scripsit: So wwhat would be the right default, if I may ask? When there is no clear preferred alternative, the right default is no default: prefer explicit over implicit in this case. May I 2nd that. Felix, I don't want to upset you with the arguments. But your'e probably too much of an expert already to see the point. The confusion stems from the differences of all those systems, which do take a lambda and then treat it different. Plus: some people - like Peter and me - believe that explicit renaming is a bit low-level and should be used only when really needed. Otherwise alt least the newbie should better choose another transformer. Clearly enforcing and thereby documenting which transformer is used is IMHO better than defaulting. Also consider: once a programmer got a good reputation, his code is seen not only by the compiler but by newbies. The latter take it as a teaching (at least iff they are smart enough to admit that there's more to learn). And it's a non-zero mental effort to keep track, which Scheme implementation has which default. Why not have (define-syntax er-lambda (syntax-rules () ((_ (e r c) body ...) (er-macro-transformer (lambda (e r c) body ...) And then (define-syntax my-er-based-syntax (er-lambda (ex re co) )) -- 1. no additional indent 2. just three more chars 3. explicit choice documented 4. when porting to other Schemes, it comes up. Best /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [Chicken-users] Solved: Re: need help with hygienic macros
On May 18 2013, Jörg F. Wittenberger wrote: Eventually I learned that the technique I've been looking for is known singe 2001 by the name Petrofsky Extraction. As I'm gaining experience with the Petrofsky extraction, I wonder how useful it might by to have a version of SRFI-8 `receive`, which avoids call-with-values (at least under circumstances). Possible gains: a) Some run-time errors are caught at compile-time. b) Returning multiple values is to my knowledge more run-time expensive than tail calls in chicken. Downside: `values` is only available as syntax here. It would be an interesting exercise (I'm beyond my current capabilities) to have it available as a first class procedure too. Find attached some test files: t1.scm: test cases with errors - non of which is caught at compiletime. t2.scm: a (limited) version of receive, which catches some errors and avoids call-with-values. However if the multi-value returning expression returns by calling some other procedure (instead of having a literal (values x ...) in tail position, the tail call needs to be wrapped in some crazy `values-from` form. t3.scm: same as t2.scm, but with magic to handle avoid the need for `values-from`. The trick in t3.scm however goes at the expense of always expanding into call-with-current-continuation and call-with-values. Just that it returns prior to going through C_values. But I'm afraid the additional setup costs will be worse than the gain. At the moment I'm contemplating if it would be possible to rewrite (preferable using syntax-rules for educational purpose) an expression such that tail calls are automatically wrapped by something like `values-from`. comments welcome Best /Jörg ;; This will become a compiletime error. (receive (a b c) (values 1 2) (display (cons a b))) (receive (a b) (let ((x 1) (a 2)) (values x a)) (cons a b)) (receive x (values 1 2 3) x) (define (foo) (values 1 2)) (receive (a b) (foo) (cons a b)) (define (bar a v) (values a v 1)) ;; This one will stay as a runtime error. (receive (a b) (bar 2 3) (cons a b)) (receive (a b) (receive (a b c) (values 1 2 3) (values c (cons a b))) (vector b a)) (receive (a b) (receive (a b c) (bar 5 6) (values c (cons a b))) (vector b a)) ;; `rcv`: A limited alternative for SRFI-8 `receive`. ;; ;; This avoids to use call-with-values - which incurs some overhead on ;; chicken. However, it works *only*, if `values` is used in tail ;; call position within the `expr` itself. Havoc if `values` is ;; referenced as a value inside the expression. ;; ;; At least it nests. ;; ;; Another advantage over the buildin receive: wrong number of return ;; values is caught at compile-time. ;; ;; There's some limited (and questionable) magic to support calling ;; procedures which produce multiple values in `expr` by wraping the ;; call into `values-from`. ;; # Petrofsky Extraction ;; ;; How to write dirty R5RS macros ;; http://groups.google.com/groups?selm=87oflzcdwt.fsf%40radish.petrofsky.org ;; How to write seemingly unhygienic macros using syntax-rules ;; Date: 2001-11-19 01:23:33 PST ;; ;; Extract several colored identifiers from a form ;;extract* SYMB-L BODY CONT ;; where SYMB-L is the list of symbols to extract, and BODY and CONT ;; has the same meaning as in extract, see below. ;; ;; The extract* macro expands into ;; (K-HEAD (extr-id-l . K-IDL) . K-ARGS) ;; where extr-id-l is the list of extracted colored identifiers. The ;; extraction itself is performed by the macro extract. (define-syntax extract* (syntax-rules () ;; Extract a colored identifier from a form ;;extract SYMB BODY CONT ;; BODY is a form that may contain an occurence of an identifier ;; that refers to the same binding occurrence as SYMB, perhaps ;; with a different color. ;; CONT is a form of the shape (K-HEAD K-IDL . K-ARGS) ;; where K-IDL are K-ARGS are S-expressions representing lists or ;; the empty list. ;; The extract macro expands into ;; (K-HEAD (extr-id . K-IDL) . K-ARGS) ;; where extr-id is the extracted colored identifier. If symbol ;; SYMB does not occur in BODY at all, extr-id is identical to ;; SYMB. ((_ extract symb body _cont) (letrec-syntax ((tr (syntax-rules (symb) ((_ x symb tail (cont-head symb-l . cont-args)) (cont-head (x . symb-l) . cont-args)) ; symb has occurred ((_ d (x . y) tail cont) ; if body is a composite form, (tr x x (y . tail) cont)) ; look inside ((_ d1 d2 () (cont-head symb-l . cont-args)) (cont-head (symb . symb-l) . cont-args)) ; symb does not occur ((_ d1 d2 (x . y) cont) (tr x x y cont) (tr body body () _cont))) ((_ (symb) body cont) ; only one symbol: use extract to do the job (extract* extract symb body cont)) ((_ _symbs _body _cont) (letrec-syntax ((ex-aux ; extract symbol-by-symbol (syntax-rules
[Chicken-hackers] Benchmark for values - interesting results
Hi, so far I lived in the believe that returning multiple values would be rather slow under Chicken. This goes back to a remark from Felix, which is several years old. (However I personally love using multiple values mostly for clarity of the code; but also because I know that some Scheme implementations have zero overhead, even less that explicitly passing a receiver continuation.) Turns out that Felix is correct even years later. Maybe it's worth to consider to change the way arguments and return values are passed in Chicken. I know this would be the usual hell to implement. But: I'm kinda familiar with RScheme's internals. It does not waste a fresh location for each parameter, thereby lowering the load in garbage collection. It just has a (list of) vector-like object holding the parameters and return values. Passing parameters (including the [hidden] current continuation) is a matter of storing them in the array and adjusting the current position and current top. Returns are handled likewise. Doing so in Chicken should be possible as well. (Though the FFI code would have to copy them onto the return stack to keep the interface as is.) This should save a lot of minor gc's since most procedures return no more values than they consume. And call/cc is rare enough. To the benchmark I did: I took partition from SRFI-1 to be a nice example and ran the following tests. The resulting times are in the profile.ods file attached. Code attached as partition.scm. A) In csi B) compiled with -O3 C) compiled with -O5 1) Labeled native/values in the spreadsheet. The code as copied from srfi-1.scm. 2) adhoc 1: using an ad-hoc alternative of values/call-with-values in portable Scheme 3) adhoc 2: a second ad-hoc alternative, minor modifications. 4) Novals/1: Converted into CPS avoiding values and receive a.k.a. call-with-values. 5) Novals/2: A slightly re-arranged version from (4) saving a let. 6) Iterative: a procedural version; this one drops the share common tail property. (IMHO not required by SRFI, just done by the code.) To summarize the results: * Version (1) is the looser here. * (2) and (3) are slightly (around 10%) faster. Under -O5 the difference disappear mostly. - That is, (2) and (3) run slightly slower under -O5 while (1) gains a little. - Notice that both ad-hoc implementations are *only two lines of code*, one for values and one for call-with-values. Since they are shorter code wise and still a little faster than (1), I wonder why we need the native C code version at all. * (4) is about twice as fast as (1). - Felix is correct. * (5) is about 2% faster than (4), but not always. * (6) beats them all hands down. Almost 20x as fast as (1) in csi. Almost 9x with -O3 and slightly more than 9x with -O5. As an immediate consequence I'd suggest to just drop this into srfi-1.scm in place of the partition code: ;; In constract to the SRFI-1 implementation, this one does not share ;; a common tail. (define (partition pred lst) (let ((t (cons #f '())) (f (cons #f '( (let ((tl t) (fl f)) (do ((lst lst (cdr lst))) ((null? lst) (values (cdr t) (cdr f))) (let ((elt (car lst))) (if (pred elt) (let ((p (cons elt (cdr tl (set-cdr! tl p) (set! tl p)) (let ((p (cons elt (cdr fl (set-cdr! fl p) (set! fl p Best regards /Jörg .. (use extras) (format (current-error-port) Some Test Cases And A Benchmark * partition/cps: an implementation taking a continuation to deliver it's result values * partition.1: an implementation of SRFI-1 \partition\ almost literal from SRFI-1 (here to make sure both are compiled with the same flags) * partition.2: an implementation of SRFI-1 \partition\ using partition/cps * partition.2.2: an implementation of SRFI-1 \partition\ using a slightly modified version * partition.1.b: SRFI-1 \partition\ using an ad-hoc alternative of values/call-with-values in portable Scheme * partition.1.c: SRFI-1 \partition\ using a second ad-hoc alternative of values/call-with-values in portable Scheme * partition-iterative: another alternative in procedural style ) (define (partition/cps return pred lis) (let recur ((lis lis) (return return)) (if (null? lis) (return lis lis) (let ((elt (car lis)) (tail (cdr lis))) (let ((cont (lambda (in out) (if (pred elt) (return (if (pair? out) (cons elt in) lis) out) (return in (if (pair? in) (cons elt out) lis)) (recur tail cont)) (define (partition/cps.2 return pred lis) (let recur ((lis lis) (return return)) (if (null? lis) (return lis lis) (recur (cdr lis) (lambda (in out) (let ((elt (car lis))) (if (pred elt) (return (if (pair? out) (cons elt in) lis) out) (return in (if (pair? in) (cons elt
Re: [Chicken-hackers] Benchmark for values - interesting results
On May 20 2013, Arthur Maciel wrote: Dear Joerg, certainly I'm not the best person to delve into details about whether this should be done or not. But recalling some requests I've made to this list in order to change chicken's core for the sake of 'optimizations' I realized that it is better to change it only if we touch a critical point that can't be bypassed in other ways. Maybe I missed your point. IHMO Chicken is a compiler. And a compilers point is run code faster than possible in interpreters, isn' it? If we would not develop Chicken further, why not just freeze it as it is? The whole chicken-hackers mailing list had no purpose then. Furthermore according to my understanding it's better to optimize the compiler than rewriting all existing code to work around weaknesses. Sure, incorporating lessons learned from other Scheme implementations is hard work and certainly nothing done between noon and lunch. At the other hand, replacing a library procedure with compatible code found to be 20x faster is certainly no big deal. Except for the effect. The chicken compiler itself won't profit much in this particular case, since it only used partition to analyse lambdas during optimization. Applications however might profit. Is it really critical to you and other users? For me: definitely yes. No matter how much I dislike XML as such (as most programmers do), one of the most frequent things my code does for me in daily life involves parsing and serializing XML. Furthermore this code runs on multiple Scheme implementations for better/compatibility testing. Removing uses of multiple value from the code to speed up one platform would be a serious drawback for another. No point in this. Let alone that - according to my taste - using multiple values results in much more readable code than resorting to all sorts of rewrites. For others: I rewrote the SSAX parser to avoid actually going through multiple value returns all the time. The speedup is nowhere near those 20x as in the partition-case. But still enough to actually feel it (at least when at machines like the Segate DockStar - as in my case) in daily use. (I'll release the code once time permitted me to converted the test cases too. By now I just deployed it, which means it's running several thousand times a day on all sorts of XML.) Nevertheless thanks for your comment. Better than feeling ignored. ;-) /Jörg ... ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] expand seems not to keep the promise made in the manual
Citing the manual
procedure(expand X)/procedure
If {{X}} is a macro-form, expand the macro (and repeat expansion
until expression is a non-macro form). Returns the resulting expression.
Let's try:
(expand '(let ((foo 1))
(let-syntax
((foo (syntax-rules () ((_) 2
(list 'bar foo (foo)
(##core#let ((foo 1)) (let-syntax ((foo (syntax-rules () ((_) 2 (list
(quote bar) foo (foo
Shouldn't this result in
(##core#let ((foo 1)) (list (quote bar) 1 2)))
??? (foo) is still an application of a macro and `let-syntax` should
IMHO not appear in any expansion at all.
Or am I mistaking something?
/Jörg
...
___
Chicken-hackers mailing list
Chicken-hackers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] Would it be hard to make csc catch this equivalence?
Try to compile this code, csc will complain about a known procedure called with the wrong number of arguments: (let ((x (lambda (y) (+ y 1 (x 2 3)) Now try this equivalent code, csc will happily accept it. (Thus converting the compile-time error into a run-time error.) ((lambda (x) (x 2 3)) (lambda (y) (+ y 1))) Practical consequence: when writing macros, one will have to keep in mind, that chicken will not catch the error in the second case. I dunno how many possible optimizations become a causality that way. But I'd love to off-load my brain from tracking that one. Would it be hard to make csc catch this one. At least for the case of immediate application in known context? Best /Jörg ... ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] Benchmark for values - interesting results
On May 24 2013, John Cowan wrote: I've always thought that case-lambda should be built into the core, so that simple tail recursions, like this: (define foo (case-lambda ((bar baz) ...) ((baz) (foo #f baz can be rewritten as a varargs function with appropriate internal jumps, without the need to construct and then deconstruct a list. Hm. At the moment I'm short of imagination how much overhead these internal jumps would incur. I can't yet imagine that those would be for free. Could they? /Jörg .. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] fix incorrect type of jmp_buf
On Jun 18 2013, Peter Bex wrote:
On Tue, Jun 18, 2013 at 02:04:32PM +0200, Jörg F. Wittenberger wrote:
On Jun 18 2013, Jim Ursetto wrote:
On Jun 17, 2013, at 2:06 AM, Felix
fe...@call-with-current-continuation.org wrote:
This is quite a serious bug, and I recommend putting the patch into
the stability branch.
Congratulations to this catch!
While I'm not sure that this was really the reason, I can't see any
other. I had long standing problems with threads dropped from the
waiting queue at random. Couldn't find the reason. Since yesterdays
update the problem appears gone with no other code changes.
Great to hear!
I know you've used Valgrind on CHICKEN in the past. Does it not detect
this problem, or have you not tried it recently?
At least I tried not recently, certainly not since the sig*jmp stuff
came in.
Also running my code under valgrind doesn't *really* work. It did
catch some uninitalized variables etc (one count is still not fixed in
C_reclaim:
if(gc_mode == GC_REALLOC) {
C_rereclaim2(percentage(heap_size, C_heap_growth), 0);
gc_mode = GC_MAJOR;
count = (C_uword)tospace_top - (C_uword)tospace_start;
goto i_like_spaghetti;
}
The count above.
But the result runs way too slow to work in concert within the
timeout limits of the whole network.
Also those thread drops where really random to be. Rarely within
minutes after program start, sometimes only after hours. I would
not even have the disk space to log this, let alone patience.
Maybe we could start using Valgrind in the Salmonella runs.. Not sure if
that would skew the test results, though...
Cheers,
Peter
___
Chicken-hackers mailing list
Chicken-hackers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] fix incorrect type of jmp_buf
On Jun 18 2013, Peter Bex wrote:
On Tue, Jun 18, 2013 at 02:50:21PM +0200, Jörg F. Wittenberger wrote:
At least I tried not recently, certainly not since the sig*jmp stuff
came in.
OK, makes sense.
Also running my code under valgrind doesn't *really* work. It did
catch some uninitalized variables etc (one count is still not fixed in
C_reclaim:
if(gc_mode == GC_REALLOC) {
C_rereclaim2(percentage(heap_size, C_heap_growth), 0);
gc_mode = GC_MAJOR;
count = (C_uword)tospace_top - (C_uword)tospace_start;
goto i_like_spaghetti;
}
The count above.
What, specifically, is wrong with it?
Rigth before goto i_like_italian_food there is no assignment to count
in the chicken source. Valgrind therefore complains about the possible use
of an uninitialized variable.
Cheers,
Peter
___
Chicken-hackers mailing list
Chicken-hackers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] fix incorrect type of jmp_buf
On Jun 21 2013, Peter Bex wrote:
On Wed, Jun 19, 2013 at 02:32:01PM +0200, Jörg F. Wittenberger wrote:
On Jun 18 2013, Peter Bex wrote:
On Tue, Jun 18, 2013 at 02:50:21PM +0200, Jörg F. Wittenberger wrote:
Also running my code under valgrind doesn't *really* work. It did
catch some uninitalized variables etc (one count is still not fixed
in C_reclaim:
if(gc_mode == GC_REALLOC) {
C_rereclaim2(percentage(heap_size, C_heap_growth), 0);
gc_mode = GC_MAJOR;
count = (C_uword)tospace_top - (C_uword)tospace_start;
goto i_like_spaghetti;
}
The count above.
What, specifically, is wrong with it?
Rigth before goto i_like_italian_food there is no assignment to
count in the chicken source. Valgrind therefore complains about the
possible use of an uninitialized variable.
I don't see this assignment in a clean copy of the master branch, and
count doesn't appear to be used when gc_mode is GC_REALLOC.
Sure, you can't. It's missing.
Sorry, I'm not native English: I don't know how to express better that
anything ought to be assigned to count before used (further down).
If you still think this is wrong,
It's not me, it's valgrind who complains. I don't care. ;-)
could you prepare a patch and elaborate
a little on what exactly is going wrong?
No, I'm not even sure the value l assigned is correct.
I just guessed from context… it's going to be printed only anyway.
There are more important things to do about chicken than killing
age old none-bugs.
Who's running valgrind on chicken besides me?
Cheers,
Peter
___
Chicken-hackers mailing list
Chicken-hackers@nongnu.org
https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] R7RS delay/force/delay-force
Am 02.11.2013 06:21, schrieb John Cowan: Jörg F. Wittenberger scripsit: Rather uncontroversial: r7rs says the result of a delayed expression returning multiple values is unspecified. Hence it's OK to actually return multiple values (as I'd prefer r7rs should have explicitely required). The downside is that a portable program must not depend on this to work. The reason we didn't require this is that in order to cache the multiple values, in practice the implementation would have to put them into a data structure, Sure it's easy to convert multiple value returns into a single value (e.g. a list) and back. Scheme _could_ be done without multiple values at all (the way you describe). Just: multiple values are useful as an alternative to (one shot) objects and _may_ be implemented more efficient. The downside of the r7rs decision to leave things open this way is what we observe right now with chicken: users desire the consistency between direct calls and delay/force and hence implement the complicated way using an intermediate list structure (even deconstructing the zero-value case as a special case but handling the one-valued case using the generic multiple value procedure C_values, which is according to my understanding less efficient - hence it would have been better to handle the one-value case special... but even talking about these details is more complicated than the source code ever would be) while at the same time portable programs must avoid dependencies on this behavior and implement the very same thing otop once again. :-/ It specifies if it is forced a second time, the previously computed value is returned. This _could_ (and IMHO _should_) be understood as if the delayed expression must only ever be evaluated once. That's the most plausible interpretation, yes. While I dunno how the wording could be: I wish it was possible to at least suggest that in rXrs somehow. But there are at least two issues: re-entrancy via call/cc and via threads (which in some implementations may be the same). We left that open, because it's not clear what the Right Thing is. I'm short on wording too. Wouldn't it be possible to require all sorts of reentrance to return the result of the first invocation? Maybe even declare at least threads should do the right thing and block (if nobody can come up with a trick to handle call/cc too. As things are, the situation is so inconsistent that, well it's not Schemish at all. :-/ It's easier to provide test cases for all sorts of failures than an idea what force/delay are supposed to do. In my code I try at least to cope with the multi-threaded case. ... and to avoid dancing all time around export/toplevel issues ... I'm using a weird delay* instead of the intended delay everywhere. Ugly a work around. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] R7RS delay/force/delay-force
Am 02.11.2013 23:49, schrieb Evan Hanson: On 2013-11-02 21:10, Jörg F. Wittenberger wrote: deconstructing the zero-value case as a special case but handling the one-valued case using the generic multiple value procedure C_values, which is according to my understanding less efficient - hence it would have been better to handle the one-value case special... FWIW, that was purely to allow checking for forced values without incurring the cost of `list?`. It might be good to add a special case for single values, too; that was just more code, so I didn't. I don't have an opinion on the question of threads, but do think supporting MVs properly (i.e. without pushing it onto the user) is a Good Thing. Me too thinks so. Evan ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] R7RS delay/force/delay-force
Am 04.11.2013 06:17, schrieb John Cowan: Jörg F. Wittenberger scripsit: Sure it's easy to convert multiple value returns into a single value (e.g. a list) and back. Scheme _could_ be done without multiple values at all (the way you describe). Just: multiple values are useful as an alternative to (one shot) objects and _may_ be implemented more efficient. Yes, they may be when returned directly, but not when cached for later return. In that case, allocating a data structure (even if it's never used again) is unavoidable. Sure, but that's not the point I'm after. For the sake of consistency I feel it's better to define `force` to return just what the delayed expression returned. Thus multiple values if that's been the thing. Otherwise chicken users - and for sure other Scheme's too - are in the uncomfortable situation that writing portable code requires them to know that their implementation of choice will do the right thing but they still have to go to the trouble and double the implementation because of their intention to end up with portable code. I feel that's bad. Especially in the situation at hand, where we know: a) because of this possibly re-entry a modern implementation needs to unroll in constant space (additional code over the trivial case) and b) in some (hopefully near) future something should be defined how to handle concurrent use of `force`, which will likely require a similar trick. Maybe even declare at least threads should do the right thing and block (if nobody can come up with a trick to handle call/cc too. Threads are not defined in R7RS-small, and the only place we discuss them is in `parameterize`, where we explain that a value dynamically bound in one thread does not affect any other threads (a bare minimum requirement). Note that R7RS parameters can be rebound but can't be mutated, which was done specifically because some implementations copy the mutated value when forking a new thread and others do not. Nevertheless they are a) mentioned and thus in scope somehow b) may be just refined call/cc, which takes us back to the question I can't answer: can something be done to make sure delayed expressions are evaluated just once. Note that there is a concept set variables (I came across it in paxos at war http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-593.pdf but I'm not sure that this is the origin) which is quite helpful when reasoning about parallel processing in general. This concept would perfectly map to Scheme's delayed expressions … if those where evaluated just once in any case. Best Regards /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] Fix for #989 and hopefully #877 too
Hi, despite the fact that salmonella currently seems to be happy with the situation, let me add my €0,02. Sometimes the solution is not more code but less. Chicken's interrupt handling was broken for quite a long time. There is by now still a problem lurking and waiting to be triggered: there are only so many pending interrupts possible. Let me assure you: I'm NOT making this up. This has been a problem even before it was introduced into chicken. RScheme happens to be very similar to chicken in that respect. And I've been burned by those droped signals before chicken received the limitation too. Please refer to the archives of this mailing list. In fact I've been taught the solution to this problem when I attended the university. What chicken is solving when scheduling signals is pretty much the same problem operating system kernels solve when handling interrupts. We've been taught at that time that the best (in terms of simplicity and safety) way to handle this in a multi threaded environment would be to dispatch the signal/interrupt to a user level thread on short path. Optionally schedule this thread next for execution. Never handle the interrupt in the kernel itself. If you don't have multiple thread running (but have the machinery already, otherwise the point is moot) create an extra thread just for this case. When the issue was introduced into chicken, I forked my own environment and just tried following the text books. Safe for some dead code I never had to deal with these problems again. (Let me repeat: the limited queue of pending interrupts was a problem in practice for me. All I have to do is pull the network plug when running certain code to make it visible.) I'd love to help bringing these fixes into the mainline chicken. (After all it's a horrible burden to always maintain the diff). @Peter: I'm still waiting for your reply on that topic. Best Regards /Jörg Am 05.11.2013 22:41, schrieb Mario Domenech Goulart: On Tue, 5 Nov 2013 16:52:16 +0100 Peter Bex peter@xs4all.nl wrote: The tests work, the sleep test from #989 works and the synch egg's tests no longer hang, so I think this patch should probably go in ASAP. ... Thanks, Peter. I could successfully run salmonella on the whole set of linux eggs. I pushed your patch. Best wishes. Mario ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
[Chicken-hackers] noreturn clang
I used to compile with clang to avoid the endless list of warning: array subscript is above array bounds [-Warray-bounds] from gcc (4.6.1) Now it's replaced with an endless list of warning: function declared 'noreturn' should not return [-Winvalid-noreturn] from clang. (Besides that I often value compilation time.) Does anybody know how to make clang shut up on this one? Thanks a lot /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] noreturn clang
Am 06.11.2013 11:14, schrieb Peter Bex: On Wed, Nov 06, 2013 at 11:10:39AM +0100, Jörg F. Wittenberger wrote: Also this commit is obviously applied and the code is in chicken.h What version of clang are you using? Looks like version 2.9. I wrote this code testing with clang 3.3, so maybe the 3 major version only has support for it. If it's important to you, try upgrading to that version. Appears that 3.0 does already honor the noreturn declaration. But now it complains about the array subscriptions as gcc did before... Cheers, Peter ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] noreturn clang
Am 06.11.2013 14:47, schrieb Peter Bex: On Wed, Nov 06, 2013 at 02:35:20PM +0100, Jörg F. Wittenberger wrote: Appears that 3.0 does already honor the noreturn declaration. But now it complains about the array subscriptions as gcc did before... Can you provide a minimum set of steps which one needs to take in order to see this happening, using a clean clone of git master? AFAIK it's fixed and I don't see the messages here, so there must be something different between what you're doing and what I'm doing. Please remember to go through a bootstrap CHICKEN, as well, in order to absolutely eliminate most environmental differences. I'm afraid I *have to* regret. Save yourself the details. It's about my machine, my time and financial budget and those real world duties, which I'm supposed to eventually complete. I'm going to get real, bad trouble if I invest more into chicken. Loosing today at the failed experiment to incorporate the loss of interrupt_reason was already too much at the moment. I'll try to figure these things out, once I'm under less pressure. Sorry. /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] Fix for #989 and hopefully #877 too
Am 06.11.2013 15:29, schrieb Peter Bex: On Wed, Nov 06, 2013 at 03:21:18PM +0100, Jörg F. Wittenberger wrote: My solution at that time was to coalesce signals by type and handle all reasons of a particular signal in a single signal handler (anyway) since - as you're pointing out - this must be done in any case. That's why it's actually enough to have a single int signal_pending accumulate the signals in the global_signal_handler like this: signal_pending |= 1 signum; Maybe it's just me, but there are 256 signal handler slots. Now I cam confused. True, there are 256 possible signal handlers in ##sys#signal-vector How could those be addressed/used at all? The manpage for signal(7) on my NetBSD box shows 32 signals. That means the pending signals won't ever fit in a fixnum on a 32-bit machine. You're right: somehow it seems to have escaped me that attaching the type tag on the way from C to Scheme would reduce the number of bits avail. (Otherwise I've been working from my sure knowledge that there are only 32 signals, which in turn would fit into a single word.) /Jörg ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] Fix for #989 and hopefully #877 too
Am 06.11.2013 19:11, schrieb Peter Bex: On Wed, Nov 06, 2013 at 06:00:34PM +0100, Jörg F. Wittenberger wrote: Now I cam confused. True, there are 256 possible signal handlers in ##sys#signal-vector How could those be addressed/used at all? The timer interrupt signal has value 255. I'm sure there must be UNIXen out there which use nonstandard signals of higher numbers than 32. For example, the Linux signal(7) manual on the box which serves call-cc.org mentions that there are real-time signals which are numbered 33 through 64. Yeah, maybe. Dunno. Maybe they can be used. That never happened in my code. timer is handled special everywhere anyway. The manpage for signal(7) on my NetBSD box shows 32 signals. That means the pending signals won't ever fit in a fixnum on a 32-bit machine. You're right: somehow it seems to have escaped me that attaching the type tag on the way from C to Scheme would reduce the number of bits avail. (Otherwise I've been working from my sure knowledge that there are only 32 signals, which in turn would fit into a single word.) It could be worse, you could be losing one more bit, depending on what happens to the sign bit when you're shifting unsigned signal numbers into a signed signal bitmask (how this stuff is defined in the C spec confuses me no end). Still probably. Things could hardly be worse: it fact those don't matter for me in practice at all at the moment anymore. Because: till now I just removed my .o files and tested how those modifications which came in from the git pull to master effected the code. Now I ran a full recompile. To the effect that some macro expansion has been changed to the worse. It's somehow related to code, which uses Petrofsky extraction (replacing some hon-hygienic macros I've been using before). This used to compile perfectly for several month. Now it returns some #unspecified which means that I have no working code at all anymore using chicken more recent than July 23rd (my last follow git). Dunno what changed. Its all so bad I could cry endlessly. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] Fix for #989 and hopefully #877 too
Am 06.11.2013 19:33, schrieb Mario Domenech Goulart: Hi Jörg, On Wed, 06 Nov 2013 19:20:41 +0100 Jörg F. Wittenberger joerg.wittenber...@softeyes.net wrote: It's somehow related to code, which uses Petrofsky extraction (replacing some hon-hygienic macros I've been using before). This used to compile perfectly for several month. Now it returns some #unspecified I'd bet it is a consequence of the introduction of `letrec*'. The behavior of `letrec' has been fixed with the introduction of `letrec*' (see http://lists.nongnu.org/archive/html/chicken-hackers/2013-08/msg00017.html). If you use `letrec' in your code, check if it is ok. I've just been checking for that one. At least the petrofsky extraction and Kiseljov's keyword syntax macros do not use letrec directly. I'm afraid it's something within the macro expander itself... which means that it might effect more code than just the one here. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] Fix for #989 and hopefully #877 too
Am 06.11.2013 19:35, schrieb Jörg F. Wittenberger: Am 06.11.2013 19:33, schrieb Mario Domenech Goulart: I'd bet it is a consequence of the introduction of `letrec*'. The behavior of `letrec' has been fixed with the introduction of `letrec*' (see http://lists.nongnu.org/archive/html/chicken-hackers/2013-08/msg00017.html). If you use `letrec' in your code, check if it is ok. I've just been checking for that one. At least the petrofsky extraction and Kiseljov's keyword syntax macros do not use letrec directly. I'm afraid it's something within the macro expander itself... which means that it might effect more code than just the one here. To that end: my code uses this syntax eventually in the context of a lambda. Maybe that's what's effected? ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] Fix for #989 and hopefully #877 too
Am 06.11.2013 19:35, schrieb Peter Bex: On Wed, Nov 06, 2013 at 07:20:41PM +0100, Jörg F. Wittenberger wrote: Now I ran a full recompile. To the effect that some macro expansion has been changed to the worse. It's somehow related to code, which uses Petrofsky extraction (replacing some hon-hygienic macros I've been using before). This used to compile perfectly for several month. Now it returns some #unspecified Have a look at whether your code uses letrec. Its semantics changed recently to be more standards-correct. The quick and dirty solution is to change all letrec uses to letrec*. This is changeset a647d9ed65f44df527e513464093447f56e24ead, I think. which means that I have no working code at all anymore using chicken more recent than July 23rd (my last follow git). The change I mentioned is from August, so that makes sense. Dunno what changed. Its all so bad I could cry endlessly. Yeah, breaking changes like this suck :( At the moment this is my favorite, where I'd bet things break: (define-syntax define-transformer (syntax-rules () ((_ transformer errloc body) (begin ;; (: name :render-renderer:) (define transformer (lambda-transformer errloc body)) ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] Fix for #989 and hopefully #877 too
Am 06.11.2013 19:39, schrieb Jörg F. Wittenberger: Am 06.11.2013 19:35, schrieb Peter Bex: On Wed, Nov 06, 2013 at 07:20:41PM +0100, Jörg F. Wittenberger wrote: Now I ran a full recompile. To the effect that some macro expansion has been changed to the worse. It's somehow related to code, which uses Petrofsky extraction (replacing some hon-hygienic macros I've been using before). This used to compile perfectly for several month. Now it returns some #unspecified Have a look at whether your code uses letrec. Its semantics changed recently to be more standards-correct. The quick and dirty solution is to change all letrec uses to letrec*. This is changeset a647d9ed65f44df527e513464093447f56e24ead, I think. which means that I have no working code at all anymore using chicken more recent than July 23rd (my last follow git). The change I mentioned is from August, so that makes sense. Dunno what changed. Its all so bad I could cry endlessly. Yeah, breaking changes like this suck :( At the moment this is my favorite, where I'd bet things break: Things seem to be much worse than expected. Without going into all the nasty details, this is roughly where the problem is: The parameter named 'static-variables' here turned out to be #unspecified when referenced: (define (rules-process recurse static-variables rules) (lambda-process rules-process(rules) ;; FIXME: this needs to be rewritten using the new macros! (xml-walk*1 (current-place) (current-message) (root-node) (xsl-envt-union (environment) static-variables) (namespaces) (ancestors) (current-node) (current-node) recurse rules))) Forget about the lambda-process macro: it's complicated like hell (Petrofsky's extraction + Kiseljov's keywords) it creates a lambda of well known signature and those (current-place), (environment), (namespaces) etc. things will extract the parameters to be fed to this xml-walk*1, which will just continue the xslt transformation. But that's it. But all those details that can't really matter (IMHO) because if things live up to hygienic let me promise one thing: the character sequence static-variables is only used twice and both are on display here. Not captured no matter of hygienic or not. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
Re: [Chicken-hackers] [PATCH] Fix for #989 and hopefully #877 too
Am 06.11.2013 20:27, schrieb Peter Bex: On Wed, Nov 06, 2013 at 08:18:20PM +0100, Jörg F. Wittenberger wrote: Things seem to be much worse than expected. Without going into all the nasty details, this is roughly where the problem is: The parameter named 'static-variables' here turned out to be #unspecified when referenced: (define (rules-process recurse static-variables rules) (lambda-process rules-process(rules) ;; FIXME: this needs to be rewritten using the new macros! (xml-walk*1 (current-place) (current-message) (root-node) (xsl-envt-union (environment) static-variables) (namespaces) (ancestors) (current-node) (current-node) recurse rules))) This is all extremely contextual. I have no idea what half of this stuff does, and you can't expect us to dig into a big pile of custom code just to distill a bug. Can you not reduce it to something that breaks, but has no external dependencies? NAH; I don't expect you to dig into it. Actually it looks as if my assessment was wrong anyway: in the meantime I traced the static-variables value *before* the lambda-process form and it too is #unspecified hence I rather look elsewhere. Sorry for the noise. I'll be back when I know more of have better questions. But it could well be not today. I have no energy left. ___ Chicken-hackers mailing list Chicken-hackers@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-hackers
