Re: Generalized-Arrays egg v1.0.1

[Jeremy Steward]

On 2/3/24 16:03, John Cowan wrote:



On Sat, Feb 3, 2024 at 5:39 PM Jeremy Steward > wrote:


  Likewise, I opt to use (chicken fixnum) but this is
because I discovered that SRFI-143 is so egregiously slower than
(chicken fixnum) that it was causing serious performance regressions in
many of the array APIs.


That's very odd. The Chicken-specific library wrapper for SRFI 143 at 
GitHub (srfi-143.scm, which needs to be updated to Chicken 5) makes use 
of Chicken's native fx* procedures.  I don't know how the Chicken (srfi 
143) egg was built, though; it may be using rubber-chicken.scm and 
fxcore.scm, which are slow emulations meant for complete portability.


From what I understand on the SRFI page it is in fact re-exporting 
these procedures and not handling the overflow / underflow of some of 
the operations.


There's this note on overflow on the docs page:



Differences from the intended implementation details aside, I think the 
core of what was a performance hazard (at least as far as I looked into 
it) could have been one of two things (maybe both?):


1. The egg itself is not compiled with -O3, whereas if I link to 
(chicken fixnum) I believe that these procedures will be inlined by the 
CHICKEN compiler when the arrays egg is compiled with -O3 or higher.


2. There may be some rewriting rules that the compiler uses for 
procedures in the (chicken ...) namespace that optimizes these directly 
into their equivalent optimized C procedures. I'm not sure the compiler 
has the same visibility if you re-export these from behind a module, and 
especially not if you link dynamically (any hope of inlining those is 
thus gone forever).


In fact on the latter point above, this raises an interesting question I 
had for the mailing list: if I re-export a CHICKEN-specific procedure, 
does the way the compiler handles translation units prevent certain 
optimizations from applying? I've somewhat noticed that to be the case 
but I haven't seen any writing about it nor have I understood the extent 
to which that might affect certain SRFIs or eggs.


If you have time/energy, it would be useful to make Chicken's 
implementation do what it was originally intended.  See the 
"Implementation" section in the SRFI, or feel free to ask me for 
explanations of details.


I might have the time/energy for this, so I'll let you know if I stumble 
into anything soon. I think a CHICKEN-specific implementation probably 
wants to use the FFI since fixnums are guaranteed in their representation:




It might also be a good idea to enforce SRFI-143's behaviour around 
overflow / underflow as part of CHICKEN's default behaviour in CHICKEN 
6, but I don't really know how I would get started having that discussion.


In any case, I've cc'd Diego (SRFI-143 maintainer), maybe together we 
can find a way to improve the situation :)


Regards,
--
Jeremy Steward

Re: Generalized-Arrays egg v1.0.1

[John Cowan]

On Sat, Feb 3, 2024 at 5:39 PM Jeremy Steward  wrote:

 Likewise, I opt to use (chicken fixnum) but this is
> because I discovered that SRFI-143 is so egregiously slower than
> (chicken fixnum) that it was causing serious performance regressions in
> many of the array APIs.
>

That's very odd. The Chicken-specific library wrapper for SRFI 143 at
GitHub (srfi-143.scm, which needs to be updated to Chicken 5) makes use of
Chicken's native fx* procedures.  I don't know how the Chicken (srfi 143)
egg was built, though; it may be using rubber-chicken.scm and fxcore.scm,
which are slow emulations meant for complete portability.

If you have time/energy, it would be useful to make Chicken's
implementation do what it was originally intended.  See the
"Implementation" section in the SRFI, or feel free to ask me for
explanations of details.

Generalized-Arrays egg v1.0.1

[Jeremy Steward]

Hi all!

After about ~6 months of work, I've finally rewritten the 
generalized-arrays egg and am today releasing v1.0.1. You can find the 
repo at:




I learned a lot from this rewrite, but here's a few highlights:

1. The egg is MUCH faster now, thanks to generous use of chicken-profile 
and the statistical profiler. I've left -d2 on the egg so that others 
can profile as well.


2. The egg now heavily relies on my transducers egg. This is 
intentional, and is part of why the egg was able to be made so much 
faster than it previously was. While one could use generalized-arrays 
without transducers, it is certainly easier to leverage transducers as 
much as possible.


3. The egg now has tests and even runs these tests in GitLab CI. 
Previously, this was not true, and many APIs were just broken or 
incoherent in ways.


4. Every API is documented and has examples to demonstrate how to use it.

5. The library is now an R7RS library. The main dependencies that aren't 
R7RS (that are used) are read-syntax extensions, as well as the 
check-errors egg. Likewise, I opt to use (chicken fixnum) but this is 
because I discovered that SRFI-143 is so egregiously slower than 
(chicken fixnum) that it was causing serious performance regressions in 
many of the array APIs.


There were a ton of smaller lessons about CHICKEN, profiling, and 
thinking about array APIs in general in there as well; however, I might 
save those for a future blog post.


In any case, if you've used this egg at any point in the past and been 
disappointed, I hope you'll give it a second chance. I've even added 
some cool examples of how to use this with BLAS in the docs.


~~~

A note to Mario: since this was a rewrite, I reran test-new-egg and 
noticed that my URI in the release-info file was wrong. This was 
probably some artefact of GitLab changing their URI scheme, but it 
should be working now (I have run this with test-new-egg).


Beyond that, the egg is now CHICKEN-5 only. If it's possible, can we cut 
CHICKEN-4 support from 1.0.0 onwards? If not, let's cut out CHICKEN-4 
support entirely, since pre-1.0 versions of this egg should probably not 
be used, ever.


~~~

Thanks, and happy hacking! I hope this provides a better arrays API than 
I had previously shipped. I am definitely more proud of my egg now.

--
Jeremy Steward

Re: chicken (un)install, deployment

[Mario Domenech Goulart]

Hi Al,

On Sat, 3 Feb 2024 12:22:46 +0200 Al  wrote:

> I'd like to distribute a project that uses chicken and a number of
> eggs. In the Makefile, I'm trying to add a target that ensures the
> user has those eggs installed. However,
>
> * chicken-install seems to proceed unconditionally (even if egg is
>   already installed).

Right.  If you ask chicken-install to install an egg, it'll do so.  It
might skip its dependencies in case they are already installed.
chicken-install might skip fetching/compiling eggs in case they can be
found in the chicken-install cache (typically ~/.cache/chicken-install).

> * I see no chicken-uninstall, or at least chicken-remove-all-eggs, so
>   that at least I can test things in a clean-room installation,
>   without rebuilding

chicken-uninstall is part of the installation of chicken.  It should be
available in the same directory where chicken-install is located.

> I came up with a way sed hack to extract all imports, transform those
> to arguments for "chicken-status -list", and generate a
> chicken-status.scm file (which can then be processed by
> "chicken-install -from-list"). But still it installs
> unconditionally. I could add an extra step to check if "csi import
> egg... 
>
> What to do? Maybe I should convert my project to egg format?

That probably would be the easiest way.

All the best.
Mario
-- 
http://parenteses.org/mario

chicken (un)install, deployment

[Al]

 Hi,


I'd like to distribute a project that uses chicken and a number of eggs. 
In the Makefile, I'm trying to add a target that ensures the user has 
those eggs installed. However,


* chicken-install seems to proceed unconditionally (even if egg is 
already installed).


* I see no chicken-uninstall, or at least chicken-remove-all-eggs, so 
that at least I can test things in a clean-room installation, without 
rebuilding



I came up with a way sed hack to extract all imports, transform those to 
arguments for "chicken-status -list", and generate a chicken-status.scm 
file (which can then be processed by "chicken-install -from-list"). But 
still it installs unconditionally. I could add an extra step to check if 
"csi import egg... 


What to do? Maybe I should convert my project to egg format?


-- Al